3 ms·

You threw me off with asking about a "persistent in-memory" database... kinda like an "invisible multi-color" car... I get what you are saying, seems like you

You threw me off with asking about a "persistent in-memory" database... kinda like an "invisible multi-color" car... I get what you are saying, seems like you want a write-through cache. Sure seems that you are in the right ball park for a solution, but may have to write a bit of application code layered to do what you want with those parts. It would probably be helpful if you broke up the read/writes estimates. Also, are those writes mostly inserts or updates? There are different types of solutions depending one what you expect your profile to be like. Also are you reads just a key look up or are you doing some sort of aggregation/searching across all records? Seems like you may be prematurely optimizing your solution and underestimating the capabilities of database engines. Most of them can probably handle the load you are thinking with correct usage and sufficient tuning and hardware. For example, I believe twitter uses MySQL: http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster http://highscalability.com/scaling-twitter-making-twitter-10...

iigs 17y ago
It would probably be helpful if you broke up the read/writes estimates. Also, are those writes mostly inserts or updates? There are different types of solutions depending one what you expect your profile to be like. Also are you reads just a key look up or are you doing some sort of aggregation/searching across all records? Please also estimate your overall size demands. Manipulating 2GB of data is different than 200GB. :)
paraschopra 17y ago
Hey, Thanks for the pointers. I estimate Reads to be 10x more frequent than writes and Reads would mostly be a key lookup, with occassional aggregation. Yes, I was also wondering if anybody has demonstrated sclability with MySQl for this level, as I won't have to fiddle around with code too much.
- bsaunder 17y ago
  That sounds rather normal (reads 10x writes and 100s/1000s per second). As far as I understand MySQL can definitely scale to that. It does take considerable effort to configure and tune it (but any database solution will perform horribly out-of-the-box). Twitter uses MySQL. I'd doubt that you will hit their level of performance (though perhaps you are hoping to ;). I think you should read up on using Memcached with MySQL. Here's a good starting point: http://highscalability.com/bunch-great-strategies-using-memcached-and-mysql-better-together http://highscalability.com/bunch-great-strategies-using-memc... At this point in the game (it sounds early in your development process), you aren't painting yourself in a corner (the only thing you should be worried about here IMHO) if you go with these two, but you will probably have to refine things later on to get your scalability (again, that's true with any solution you pick). Also realize that Twitter probably has a larger hardware budget than you do and real scalability costs real money in both hardware and effort. It's been widely reported that many of their engineers have been spending considerable effort keeping things stable and improving scalability. Another performance related mantra is that "to go faster, do less". This is definitely true here... there may be things you think you need to store, but don't. Removing that effort could have a profound impact on performance. I suppose its worth mentioning that you may want to check out AWS and their storage solutions (and push the performance problems off to them). This decision also has other business factors (like can you deal with potential outages, potential security problems, cost, etc...).

iigs 17y ago

It would probably be helpful if you broke up the read/writes estimates. Also, are those writes mostly inserts or updates? There are different types of solutions depending one what you expect your profile to be like. Also are you reads just a key look up or are you doing some sort of aggregation/searching across all records? Please also estimate your overall size demands. Manipulating 2GB of data is different than 200GB. :)

paraschopra 17y ago

Hey, Thanks for the pointers. I estimate Reads to be 10x more frequent than writes and Reads would mostly be a key lookup, with occassional aggregation. Yes, I was also wondering if anybody has demonstrated sclability with MySQl for this level, as I won't have to fiddle around with code too much.

bsaunder 17y ago
That sounds rather normal (reads 10x writes and 100s/1000s per second). As far as I understand MySQL can definitely scale to that. It does take considerable effort to configure and tune it (but any database solution will perform horribly out-of-the-box). Twitter uses MySQL. I'd doubt that you will hit their level of performance (though perhaps you are hoping to ;). I think you should read up on using Memcached with MySQL. Here's a good starting point: http://highscalability.com/bunch-great-strategies-using-memcached-and-mysql-better-together http://highscalability.com/bunch-great-strategies-using-memc... At this point in the game (it sounds early in your development process), you aren't painting yourself in a corner (the only thing you should be worried about here IMHO) if you go with these two, but you will probably have to refine things later on to get your scalability (again, that's true with any solution you pick). Also realize that Twitter probably has a larger hardware budget than you do and real scalability costs real money in both hardware and effort. It's been widely reported that many of their engineers have been spending considerable effort keeping things stable and improving scalability. Another performance related mantra is that "to go faster, do less". This is definitely true here... there may be things you think you need to store, but don't. Removing that effort could have a profound impact on performance. I suppose its worth mentioning that you may want to check out AWS and their storage solutions (and push the performance problems off to them). This decision also has other business factors (like can you deal with potential outages, potential security problems, cost, etc...).