11 ms·

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely sc

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.

saiya-jin 4y ago
Not necessarily. Probably they already have pool of admins/devops that ran their systems for past 30 years, but have 0 AWS experience. Also, most companies don't need to scale infinitely, not everybody is building next google (in fact, almost nobody outside SV is).
Kon-Peki 4y ago
You can buy on-prem managed storage solutions. Like PureStorage type things, where they call you up if something goes wrong, and are FedEx’ing you replacement parts before you even noticed that it had imperceptibly failed over to the standby power supply or whatever.
dijit 4y ago
There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
- icedchai 4y ago
  One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
  - dijit 4y ago
    You’re quite mistaken on the first point. Most automation that you know of as “devops tools” are borne from sysadmins. Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too. It is revisionist to think sysadmins were not automating their jobs. Old school sysadmins used to know C, bash and Perl. New school devops just traded that for a handful of DSLs and Python. I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)
    - icedchai 4y ago
      I'm sure we all had different experiences with old school sysadmins. The ones I'm familiar with (early 90's, ISP industry) could do shell scripting and some perl. C was way out of their skill sets, except for a bit of copy-and-paste. The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.
      - dijit 4y ago
        I’m biased. I worked in teams with such people basically my whole career. The pay increased, the titles and tools changed but the mentality didn’t. People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are. Sounds like they were automating though. “Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.
        icedchai 4y ago
        There was certainly some automation being done by the sysadmins I was familiar with (I did say "relatively little" in my original post, meaning compared to today!) It was definitely more pets than cattle at the places I was familiar with. These were small-ish companies, and the business didn't care as long as the systems worked.
lakomen 4y ago
Minio is FOSS and a lot cheaper. I don't see your point
- dijit 4y ago
  I think that's a misrepresentation of the argument though. MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive. On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent. On the other, that's expensive but running your own is complex if you're not setup to do it already. MinIO is but one part of a data storage puzzle. Though a very important one.
late2part 4y ago
except that s3 doesn't infinitely scale. Try uploading 5PB of content and downloading it that day at 1M RPS.
agentultra 4y ago
How many applications need infinitely scalable distributed object storage? I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data. But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.
- treffer 4y ago
  This is my favorite joke. Is the issue big data or small machine? Memory size of the largest AWS/GCP/... instances is a good indicator for small machine. It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X. And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.
- boltzmann-brain 4y ago
  You can hardly make quantitative statements about how many applications need this or that. It's not like we're in the 60s and IBM can tell that they'll need to make exactly seven of a certain computer. I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.
  - agentultra 4y ago
    That does seem to be the common trend. If you generate that much data you tend to host on premises because the network becomes the bottleneck. It becomes important to place compute as close to the data as possible.

saiya-jin 4y ago

Not necessarily. Probably they already have pool of admins/devops that ran their systems for past 30 years, but have 0 AWS experience. Also, most companies don't need to scale infinitely, not everybody is building next google (in fact, almost nobody outside SV is).

Kon-Peki 4y ago

You can buy on-prem managed storage solutions. Like PureStorage type things, where they call you up if something goes wrong, and are FedEx’ing you replacement parts before you even noticed that it had imperceptibly failed over to the standby power supply or whatever.

dijit 4y ago

There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.

icedchai 4y ago
One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
- dijit 4y ago
  You’re quite mistaken on the first point. Most automation that you know of as “devops tools” are borne from sysadmins. Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too. It is revisionist to think sysadmins were not automating their jobs. Old school sysadmins used to know C, bash and Perl. New school devops just traded that for a handful of DSLs and Python. I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)
  - icedchai 4y ago
    I'm sure we all had different experiences with old school sysadmins. The ones I'm familiar with (early 90's, ISP industry) could do shell scripting and some perl. C was way out of their skill sets, except for a bit of copy-and-paste. The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.
    - dijit 4y ago
      I’m biased. I worked in teams with such people basically my whole career. The pay increased, the titles and tools changed but the mentality didn’t. People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are. Sounds like they were automating though. “Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.
      - icedchai 4y ago
        There was certainly some automation being done by the sysadmins I was familiar with (I did say "relatively little" in my original post, meaning compared to today!) It was definitely more pets than cattle at the places I was familiar with. These were small-ish companies, and the business didn't care as long as the systems worked.

lakomen 4y ago

Minio is FOSS and a lot cheaper. I don't see your point

dijit 4y ago
I think that's a misrepresentation of the argument though. MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive. On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent. On the other, that's expensive but running your own is complex if you're not setup to do it already. MinIO is but one part of a data storage puzzle. Though a very important one.

late2part 4y ago

except that s3 doesn't infinitely scale. Try uploading 5PB of content and downloading it that day at 1M RPS.

agentultra 4y ago

How many applications need infinitely scalable distributed object storage? I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data. But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.

treffer 4y ago
This is my favorite joke. Is the issue big data or small machine? Memory size of the largest AWS/GCP/... instances is a good indicator for small machine. It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X. And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.
boltzmann-brain 4y ago
You can hardly make quantitative statements about how many applications need this or that. It's not like we're in the 60s and IBM can tell that they'll need to make exactly seven of a certain computer. I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.
- agentultra 4y ago
  That does seem to be the common trend. If you generate that much data you tend to host on premises because the network becomes the bottleneck. It becomes important to place compute as close to the data as possible.