4 ms·
Sorry for replying to the root, this thread is kind of a mess, with stuff scattered everywhere. Here's what I think: What you're asking to do is very hard for
by iigs 18y ago
Sorry for replying to the root, this thread is kind of a mess, with stuff scattered everywhere. Here's what I think:
What you're asking to do is very hard for anything beyond simple static page retrievals. Tiny mistakes in schema design (let alone architecture) could make or break your site.
Get expert architecture help:
If there's no way for you to start small and make mistakes early, your best bet is to find some knowledgeable people, say 3-5 that have professional or large open source scaling experience. Ask them to NDA if you must, but they won't be able to help you unless you can give them all of the details. Solicit their opinions for architecture, and try to reconcile the dissenting opinions about any given technology to see if anyone is off base. Proper architecture could take this from a seven figure hardware expenditure down to a five figure one.
Get commercial support for your hardware and software:
Once you have a rough idea how you'll do it, you'll want to engage good support from any vendors possible. When you're starting small you can just install Ubuntu, apt-get install mysql and get to work, but at millions of hits per hour you may well start to uncover unusual things involving bad hardware, hardware/kernel, or kernel/userland interaction -- likewise with any DB or other userland software. I would say expect to buy tier 1 hardware, an OS with a support contract, and support for any major software component as well. This isn't a very web 2.0 formula, but you're buying expertise that is focused on your solution.
Other things:
1) Sharding isn't a magic bullet but depending on how your data is laid out (specifically what data is not local to other data) it could totally save your bacon here. One box (or a standard three tier architecture) probably wouldn't even have a chance.
2) SSDs aren't ready yet, IMO. Due to the fact that failure is write cycle count related and not heat/time related, they're not suitable (at this time) for use in RAID systems. I've read reports of SSDs in RAIDs that die in such a tight timespan that you can't get the first one rebuilt before the second one dies.
Good luck. It sounds like a really cool project.