4 ms·

What your not taking into account is that thousands of small servers will likely fail faster than dozens of big servers. I think you'll find that this and the f

What your not taking into account is that thousands of small servers will likely fail faster than dozens of big servers. I think you'll find that this and the fact that you'd have to maintain much more hardware (the servers themselves and switches) tips the balance.

lsc 15y ago
I've managed clusters of 60K plus large servers. I mean, I didn't manage them by myself; my primary job was automating hardware failure detection, but I took my turn on the pager. My personal focus was to see to it that bad hardware didn't go back into production; as someone who shared pager, that was... offensive. But really, hardware failure is less of an issue than people seem to say. I mean, sure, it happens. But not nearly as often as software issues that bring down the box, unless you count bad hard drives; and those happen all the time. Even with 60K servers, I quickly learned that if I found a lot of bad servers that were bad in similar ways, I needed to be careful before having them sent back to the vendor. More than once I embarrassed myself blaming software problems on the hardware. For the first 3-5 years of it's life, if the computer was properly assembled, very likely the only things that will need replacement are the disks. My thought would be to treat the things as hard drives. two options: For the lowest tier of support, if your system fails, I leave it and provision a new one. Leave the dead ones in the rack until it's time to upgrade. The second option is to put the boards on some sort of hot-swap power backplane, and when the thing fails, pop it out and replace it like a hard drive. The question would be "how expensive is it to make the hot-swap system, and does the manufacturer give me a warranty that makes it worth my time to swap 'em out and send 'em back." I suspect that if I can get 'em to give me a discount to not have a warranty, my best bet would be to set it up so I could hot swap whatever i used for storage, but if anything but storage failed, I'd just leave the system in place. sure, maybe I'd want to offer 'premium' support with mirrored storage where I'd swap your storage if it fails, and yeah, in that case, expenses would be higher than a giant server with fewer disks. (by fewer disks, I mean fewer disks per 256mb guest. Obviously, it has more disks total) I/O performance is probably also going to be worse on the giant server with fewer disks, so I can charge more for the 'premium' dedicated hardware with dedicated I/O. I mean, you get into the 'all troubleshooting must be automated' range much faster with those than with physical hardware, but somewhere between a couple hundred and a couple thousand hardware nodes, you are going to get there either way.
- DanWaterworth 15y ago
  OK, well I'm certainly more convinced now by your idea, but I feel that there are too many unknowns to be able to say confidently that it would actually be a viable alternative.
  - lsc 15y ago
    oh yeah, it's definitely still in the "maybe" zone; and right now, things still lean towards VPSs being enough more efficient that it doesn't make sense to try little ARM servers seriously. I'm just saying, things are moving towards the point, as far as I can tell, where tiny arm servers could be competitive, maybe, with VPSs. The big problem that is stopping me from betting my money on little ARM servers is that nobody makes commodity-priced ARM boards that use the latest ARM CPUs and have reasonable amounts of ram and don't come with a bunch of stuff like video that I don't want to pay for. I mean, even if I had that, it's still pretty borderline and quite possible that I'm wrong and it still wouldn't make sense; but it's close enough that I'd be willing to bet a large (by my standards) chunk of my money on it.
    - DanWaterworth 15y ago
      A few thoughts: * It seems likely that there exists an ARM IC that would suit your purposes, you'd just need to commission a PCB and if you'd planning on having 10s of thousands of them, then that would be no great cost. * Having more memory will increase the board area and cost significantly. * You might be better off PXE booting and using a SAN rather than having local disks. * If you wanted to test the waters, you could try stuffing dozens of raspberry pis into a 4U case with a switch and co-locating. If this took off, you could develop your own custom hardware.
      - lsc 15y ago
        Well, a SAN has other problems; the primary advantage of a dedicated over a VPS is I/O, and SANs, well, you know with software, you say "good, fast, cheap; pick two" - with a SAN, really, the hard part is software. In this case, "good" means "reliable" and you have to be really careful to get two; usually you only get one. I have a lot of admiration for Amazon's engineering; but even they can't come up with a reliable, performant, and cost-effective SAN solution; and they are able to charge a lot more per gigabyte and per iop than I can. Now, it's a tradeoff, I mean, local hard drives will /massively/ increase the failure rates, and moderately increase the cost of the individual servers. I've got someone close to me who is an ARM person who has done PCB layouts and has participated in making custom SBCs; according to this person, the nice, fast dual-core chips you see in cellphones that we'd like to use are often unavailable in quantities of around 1000, which is where I want to start out (even this, 1000 at a target cost of $100 each, is going to require investment. I can drop maybe 1/5th to 1/10th of that of my money, but there is zero chance of custom hardware at the volume I can pay for being economical. My understanding is that you start to see reasonable pricing at around 1000 units.) Memory, yeah, will be a big cost. But it's also really, really important; Until you have enough ram, nothing else matters.

lsc 15y ago

I've managed clusters of 60K plus large servers. I mean, I didn't manage them by myself; my primary job was automating hardware failure detection, but I took my turn on the pager. My personal focus was to see to it that bad hardware didn't go back into production; as someone who shared pager, that was... offensive. But really, hardware failure is less of an issue than people seem to say. I mean, sure, it happens. But not nearly as often as software issues that bring down the box, unless you count bad hard drives; and those happen all the time. Even with 60K servers, I quickly learned that if I found a lot of bad servers that were bad in similar ways, I needed to be careful before having them sent back to the vendor. More than once I embarrassed myself blaming software problems on the hardware. For the first 3-5 years of it's life, if the computer was properly assembled, very likely the only things that will need replacement are the disks. My thought would be to treat the things as hard drives. two options: For the lowest tier of support, if your system fails, I leave it and provision a new one. Leave the dead ones in the rack until it's time to upgrade. The second option is to put the boards on some sort of hot-swap power backplane, and when the thing fails, pop it out and replace it like a hard drive. The question would be "how expensive is it to make the hot-swap system, and does the manufacturer give me a warranty that makes it worth my time to swap 'em out and send 'em back." I suspect that if I can get 'em to give me a discount to not have a warranty, my best bet would be to set it up so I could hot swap whatever i used for storage, but if anything but storage failed, I'd just leave the system in place. sure, maybe I'd want to offer 'premium' support with mirrored storage where I'd swap your storage if it fails, and yeah, in that case, expenses would be higher than a giant server with fewer disks. (by fewer disks, I mean fewer disks per 256mb guest. Obviously, it has more disks total) I/O performance is probably also going to be worse on the giant server with fewer disks, so I can charge more for the 'premium' dedicated hardware with dedicated I/O. I mean, you get into the 'all troubleshooting must be automated' range much faster with those than with physical hardware, but somewhere between a couple hundred and a couple thousand hardware nodes, you are going to get there either way.

DanWaterworth 15y ago
OK, well I'm certainly more convinced now by your idea, but I feel that there are too many unknowns to be able to say confidently that it would actually be a viable alternative.
- lsc 15y ago
  oh yeah, it's definitely still in the "maybe" zone; and right now, things still lean towards VPSs being enough more efficient that it doesn't make sense to try little ARM servers seriously. I'm just saying, things are moving towards the point, as far as I can tell, where tiny arm servers could be competitive, maybe, with VPSs. The big problem that is stopping me from betting my money on little ARM servers is that nobody makes commodity-priced ARM boards that use the latest ARM CPUs and have reasonable amounts of ram and don't come with a bunch of stuff like video that I don't want to pay for. I mean, even if I had that, it's still pretty borderline and quite possible that I'm wrong and it still wouldn't make sense; but it's close enough that I'd be willing to bet a large (by my standards) chunk of my money on it.
  - DanWaterworth 15y ago
    A few thoughts: * It seems likely that there exists an ARM IC that would suit your purposes, you'd just need to commission a PCB and if you'd planning on having 10s of thousands of them, then that would be no great cost. * Having more memory will increase the board area and cost significantly. * You might be better off PXE booting and using a SAN rather than having local disks. * If you wanted to test the waters, you could try stuffing dozens of raspberry pis into a 4U case with a switch and co-locating. If this took off, you could develop your own custom hardware.
    - lsc 15y ago
      Well, a SAN has other problems; the primary advantage of a dedicated over a VPS is I/O, and SANs, well, you know with software, you say "good, fast, cheap; pick two" - with a SAN, really, the hard part is software. In this case, "good" means "reliable" and you have to be really careful to get two; usually you only get one. I have a lot of admiration for Amazon's engineering; but even they can't come up with a reliable, performant, and cost-effective SAN solution; and they are able to charge a lot more per gigabyte and per iop than I can. Now, it's a tradeoff, I mean, local hard drives will /massively/ increase the failure rates, and moderately increase the cost of the individual servers. I've got someone close to me who is an ARM person who has done PCB layouts and has participated in making custom SBCs; according to this person, the nice, fast dual-core chips you see in cellphones that we'd like to use are often unavailable in quantities of around 1000, which is where I want to start out (even this, 1000 at a target cost of $100 each, is going to require investment. I can drop maybe 1/5th to 1/10th of that of my money, but there is zero chance of custom hardware at the volume I can pay for being economical. My understanding is that you start to see reasonable pricing at around 1000 units.) Memory, yeah, will be a big cost. But it's also really, really important; Until you have enough ram, nothing else matters.