6 ms·

Such an awesome article. I like how they didn't just go with the Cloud wave but kept sysadmin'ing, like ol' Unix graybeards. Two interesting things they wrote a

• tgv 2y ago
  To avoid sysadmin tasks, and keep costs down, you've got to go so deep in the cloud, that it becomes just another arcane skill set. I run most of my stuff on virtual Linux servers, but some on AWS, and that's hard to learn, and doesn't transfer to GCP or Azure. Unless your needs are extreme, I think sysadmin'ing is the easier route in most cases.
  • baxtr 2y ago
    I predict a slow but unstoppable comeback of the sysadmin job over the next 5-10 years.
    • homebrewer 2y ago
      It never disappeared in some places. In my region there's been zero interest in "the cloud" because of physical remoteness from all major GCP/AWS/Azure datacenters (resulting in high latency), for compliance reasons, and because it's easier and faster to solve problems by dealing with a local company than pleading with a global giant that gives zero shits about you because you're less than a rounding error in its books.
  • wongarsu 2y ago
    For so many things the cloud isn't really easier or cheaper, and most cloud providers stopped advertising it as such. My assumption is that cloud adoption is mainly driven by 3 forces: - for small companies: free credits - for large companies: moving prices as far away as possible from the deploy button, allowing dev and it to just deploy stuff without purchase orders - self-perpetuating due to hype, cv-driven development, and ease of hiring All of these are decent reasons, but none of them may apply to a company like fastmail
    • graemep 2y ago
      Also CYA. If you run your own servers and something goes wrong its your fault. if its an outage at AWS its their fault. Also a huge element of follow the crowd, branding non-technical management are familiar with, and so on. I have also found some developers (front end devs, or back end devs who do not have sysadmin skills) feel cloud is the safe choice. This is very common for small companies as they may have limited sysadmin skills (people who know how to keep windows desktops running are not likely to be who you want to deploy servers) and a web GUI looks a lot easier to learn.
      • dietr1ch 2y ago
        > If its an outage at AWS its their fault. Well, still your fault, but easy to judo the risk into clients saying supporting multi-cloud is expensive and not a priority.
        • graemep 2y ago
          Management in many places will not even know what multi-cloud is (or even multi-region). As Cloudstrike showed, if you follow the crowd and tick the right boxes you will not be blamed.
          • bobnamob 2y ago
            nit: Crowdstrike Unless the incident is now being referred to as “Cloudstrike”, in which case, eww
            • dietr1ch 2y ago
              Yeah, he meant Crowdstrike. Cloudstrike is the name of a future security incident affecting multiple cloud provides. I can't disclose more details.
    • ghaff 2y ago
      There are other, if often at least tangentially related, reasons but more than I can give justice to in a comment. Many people largely got a lot of things wrong about cloud that I've been meaning to write about for a while. I'll get to it after the holidays. But probably none more than the idea that massive centralized computing (which was wrongly characterized as a utility like the electric grid) would have economics with which more local computing options could never compete.
  • graemep 2y ago
    > it becomes just another arcane skill set Its an arcane skill set with a GUI. It makes it look much easier to learn.
• edward28 2y ago
  The power of Moore's law.
• jeffbee 2y ago
  I don't see how point 2 could have come as a surprise to anyone.
• kwillets 2y ago
  SSD's are also a bit of an achilles heel for AWS -- they have their own Nitro firmware for wear levelling and key rotations, due to the hazards of multitenant. It's possible for one EC2 tenant to use up all the write cycles and then pass it to another, and encryption with key rotation is required to keep data from leaking across tenant changes. It's also slower. We had one outage where key rotation had been enabled on reboot, so data partitions were lost after what should have been a routine crash. Overall, for data warehousing, our failure rate on on-prem (DC-hosted) hardware was lower IME.
• brongondwana 2y ago
  My beard isn't entirely grey yet! The new NVMe drives we've only had for a few years, but so far there's only been a single failure across the whole fleet, and we keep spares in stock. It's been very reliable, not like the weeks back in (hmm, 2006? 2007?) the ancient past, when we were losing 15kRPM velociraptors every other day. They had a firmware fault and we eventually got an update which made them reliable, but it was a wild few months.
  • AndrewDavis 2y ago
    A few more than one, but it has been a lot less than when we were dealing with spinner. I think I requested about one or two replacements a year, a far cry from the one a week I was doing before.
    • brongondwana 2y ago
      If I didn't see it, it didn't happen... OK, I stand corrected. We lose one or two NVMe drives per year :)
    • karolzlot 2y ago
      Can I ask which brands / models of SSD are you using?
      • AndrewDavis 2y ago
        We didn’t replace all servers at once, it was progressive, therefore due to availability the models we used changed over time. Our first batch of all nvme machines had SSDPE2KX080T8, they became harder to source and we moved to SSDPF2KX076T1. With Intel no longer in the ssd business I believe we have some Micron MTFDKCC7T6TGH and MTFDKCC30T7TGR. And as mentioned in the blog post we've recently purchased some Solidigm D5-P5336 which are 61TB monsters. Here's a fun related story. Our supplier had so much trouble finding SSDPE2KX080T8 that when we had exhausted our spares, I had to sync everything off a machine, tear it down and pull its drives for spares and rebuild it with the smaller SSDPF2KX076T1. Then we had lots of spares