4 ms·

Along the same lines, if you can avoid a distributed architecture, things get a lot more reliable. You can get a crazy amount of RAM, SSD, CPU cores on a singl

Along the same lines, if you can avoid a distributed architecture, things get a lot more reliable. You can get a crazy amount of RAM, SSD, CPU cores on a single machine. If you run your system on a powerful machine with some other ones on hot standby, a lot of complexity goes away.

nine_k 6y ago
Your reservation rate goes to 200% though (a full hot standby) instead of, say, 120% to accommodate for some nodes becoming unavailable. If your hot standby is a$100/mo VM, it's not noticeable. If it's $5000/mo, less so. To say nothing of scaling up and down with the load — which, of course, you only need if you are a pretty large operation.
- weehoo 6y ago
  If you’re paying Bay Area prices for engineers, $5K a month is a steal to not have to pay people to deal with sharding.
  - cobookman 6y ago
    Not even Bay Area prices. A Jr. SWE after overhead (benefits, HR, laptop, office-space,...etc) is easily costing the company 150k+/year in most markets.
  - ocdtrekkie 6y ago
    Indeed, but take it a step further. Two ten thousand dollar servers in your basement with UPS and some rudimentary failover configuration is basically fire and forget. Remote in monthly and install updates. Done. It'll run for ten years for next to nothing.
    - nine_k 6y ago
      Depends on your bandwidth requirements. Also, if you want even higher reliability, you might consider getting two independent internet links into your basement, which is pretty doable in an urban setting.
      - snovv_crash 6y ago
        Run the backup from your friend's basement in the next town over using a different ISP. You can run the backup for them.
        ocdtrekkie 6y ago
        For a long time, my off-site backup was at my grandmother's house because it was the furthest geographic location I could give someone a box who would leave it plugged into their Internet. ;)
      - Mauricebranagh 6y ago
        But they wont have diverse routing :-) all it takes is a navvy with a back hoe digging in the wrong place. And you also need to have diverse routing for power coming in and generator / battery room set up.
    - Cthulhu_ 6y ago
      Until there's a power outage, flooding, malice, etc. I think the main issue is that the cloud providers don't publish much about outages that don't affect the end-user. I mean a failed hard drive happens all the time, but S3 is never affected by that.
epic9x 6y ago
If you can run your system on a single machine, you don't need an SRE. If you have hundreds or thousands of machines, that's an indicator that you /may/ have the complexity that requires the disciplines that can come from dedicated SRE. The tough thing is conflating filling operations problems with a role named SRE, versus actually using the best practices that will help you scale and improve reliability.

nine_k 6y ago

Your reservation rate goes to 200% though (a full hot standby) instead of, say, 120% to accommodate for some nodes becoming unavailable. If your hot standby is a$100/mo VM, it's not noticeable. If it's $5000/mo, less so. To say nothing of scaling up and down with the load — which, of course, you only need if you are a pretty large operation.

weehoo 6y ago
If you’re paying Bay Area prices for engineers, $5K a month is a steal to not have to pay people to deal with sharding.
- cobookman 6y ago
  Not even Bay Area prices. A Jr. SWE after overhead (benefits, HR, laptop, office-space,...etc) is easily costing the company 150k+/year in most markets.
- ocdtrekkie 6y ago
  Indeed, but take it a step further. Two ten thousand dollar servers in your basement with UPS and some rudimentary failover configuration is basically fire and forget. Remote in monthly and install updates. Done. It'll run for ten years for next to nothing.
  - nine_k 6y ago
    Depends on your bandwidth requirements. Also, if you want even higher reliability, you might consider getting two independent internet links into your basement, which is pretty doable in an urban setting.
    - snovv_crash 6y ago
      Run the backup from your friend's basement in the next town over using a different ISP. You can run the backup for them.
      - ocdtrekkie 6y ago
        For a long time, my off-site backup was at my grandmother's house because it was the furthest geographic location I could give someone a box who would leave it plugged into their Internet. ;)
    - Mauricebranagh 6y ago
      But they wont have diverse routing :-) all it takes is a navvy with a back hoe digging in the wrong place. And you also need to have diverse routing for power coming in and generator / battery room set up.
  - Cthulhu_ 6y ago
    Until there's a power outage, flooding, malice, etc. I think the main issue is that the cloud providers don't publish much about outages that don't affect the end-user. I mean a failed hard drive happens all the time, but S3 is never affected by that.

epic9x 6y ago

If you can run your system on a single machine, you don't need an SRE. If you have hundreds or thousands of machines, that's an indicator that you /may/ have the complexity that requires the disciplines that can come from dedicated SRE. The tough thing is conflating filling operations problems with a role named SRE, versus actually using the best practices that will help you scale and improve reliability.