Three Laws of Distributed Memory Caching

Alrighty!! I am approaching end of the project I been working on since past few months and I am damn excited about it. Now, this is the time for optimizations, refactoring and testing. Following are the generic guidelines for memory caching which I think should be followed as design guidelines. I be following them from start from next project onwards. This time I will have to throw and add lots of code in optimization step.

1. Cache should act as an optional layer between application and database i.e. in case caching servers goes down application should work without any problem. This of course will result in degradation in performance but it should not affect application logic and working. It should be noted that if one server or set of servers goes down, client library can opt to remove them from the cache server pool and subsequently should start caching data in rest of the servers.
2. Cache should have most recent copy of the data. Either cache should not get hit for a read or it should have the most recent value. In case caching servers are restarting and values are getting updated to database, application should have scheme to reload updated data back into the cache.
   
3. Never use indefinite locks while writing to cache. Synchronizing writes to a cache can be tricky but unlike applications, lock is not on the server which is executing the code. It can happen the server which requested a lock goes down after the lock is granted. In that scenario rest of the servers will never be able to acquire the lock because in cache data is still locked. This can result in whole application getting deadlocked if there is indefinite wait for the lock to be acquired. Timed locks expires the lock after a certain time interval is elapsed.

I was asked about finishing (the mammoth task of writing :)) rest of the parts of previous post. Will be doing that on weekend.