4 ms·
Can someone explain database scaling issues for web?
I don't really understand the technical challenges behind scaling, when people talk about using NoSQL or Redis.
I mean at the end of the day aren't you still trying to lookup (or insert or delete) into a very, very, very large list? Is one technique to "flatten" out your entire user database into a tree, say by location and then further subdivide so you just go down the tree to choose which smaller DB to access?
Those are my initial naive thoughts on the matter; could someone help me understand some of the non-trivialities to database scaling?
Thanks
- andrejewski 14y agoNowadays instead of having the sub-systems you are referring to for storing data, many scaled databases use NoSQL and Redis for databasing and caching. The caching is probably the more scalable of the two because caching at any level has to do with usage and not all database data is used. Caching has allowed the most used data to be readily available. The limits of scalability may only require the system you described, but bigger and more complex systems like Twitter and Pinterest have been documented using database scaling via caching with things similar to Redis and Memcache. Keywords for finding more information are "cache/caching". Problems with the sub-system of databasing are too many transactions (I/O) and the speed of getting the data and acting on it is very inefficient. If you want to talk to me about it in more detail, message me on Twitter: @compooter or get my email at mayorpear.com
- zxcvvcxz 14y agoHere's a question: where is all this caching taking place? I understand caching in the context of processors but not so much what you're describing.
- jeffbcross 14y agoWith MongoDB, there's a concept called Sharding, that divides a collection based on a property in the collection the developer selects. http://www.mongodb.org/display/DOCS/Sharding+Introduction http://www.mongodb.org/display/DOCS/Sharding+Introduction Each shard can be either a single node, or can be divided among nodes in a Replica Set for distributed reads. http://www.mongodb.org/display/DOCS/Replica+Sets http://www.mongodb.org/display/DOCS/Replica+Sets