4 ms·

Sneakernets! https://en.m.wikipedia.org/wiki/Sneakernet https://en.m.wikipedia.org/wiki/Sneakernet But I also never fully understood why an enterprise would

Sneakernets! https://en.m.wikipedia.org/wiki/Sneakernet https://en.m.wikipedia.org/wiki/Sneakernet But I also never fully understood why an enterprise would pay so much money to give their data of this size to AWS to manage. It’s so much more expensive. If you’re at the capacity of needing a semi sneakernet, it would be far more cost effective and competitively advantageous to manage it yourself.

coldtea 2y ago
Good luck achieving the realibility and security of AWS by managing it yourself at anything "competitive". For some companies this would mean creating a whole department to manage that.
- spamizbad 2y ago
  With that much data you're paying a fortune for AWS Operations people either way. Every company I've seen that tries to large-scale AWS thing lean ends up driving themselves into a ditch where they spend millions on contractors, consultants and 3rd party vendors to dig themselves out of. And when they're in that ditch they are neither secure nor reliable.
  - nostrebored 2y ago
    I can set up an S3 bucket, efs, access, and backups, all reliable and geographically distributed in a single working day.
    - spicyusername 2y ago
      Yes, indeed. This is exactly the sneaky thing about the cloud. It seems so "simple". Now try doing that for multiple multi-thousand person orgs with competing processes, architectures, product cycles, etc. You'll find very quickly that costs spiral out of control in a very unpredictable and irreversible way. Self-hosting is something a company has much tighter control of the costs for. Most of those costs are known up front as capex purchases and predictable opex expenditures. They tend to seem expensive compared to the PoC S3 demo you just did, but often aren't compared to the actual realized expenses after migrating.
      - nostrebored 2y ago
        … I have owned core product teams in multi thousand person companies and can confidently say that your spiraling costs sound like a lack of planning and product ownership. Platform teams are your friend. Costs being well known for self hosting is just wrong. It completely ignores cost of ownership and tail risk for your infra
  - scarface_74 2y ago
    You’re right. Because every company you have seen with a large scale implementation on AWS must mean that all large companies - including Netflix - are dumb.
    - coldtea 2y ago
      Is there an argument in your comment? For my argument not "all large" companies need to be dumb, as in the strawman above. It's enough to make more sense for the average company to do so. And there are tons of companies, large, SME and others, with big data to store, that have no ops or IT department as such, or just a couple of people. They're not going to build datacenters of their own, distribute them globally, research into reliability and failover solutions, add some custom or adapt some third party stack, and so on. The salary for just an extra person could tower their AWS costs.
      - scarface_74 2y ago
        I can’t quite grok from your original comment are you saying that every company who tried to duplicate what AWS does on prem themselves end up spending too much money on consultants etc or are you saying that companies that use AWS still spend too much money on consultants and they don’t have any cost savings?
        coldtea 2y ago
        Am saying that: - most companies will do better to not try to replicate AWS (or the share of AWS functionality that they need) - most companies aren't tech companies (not talking about the Netflixes and co of the world here) and would botch it, and at best get a costly + high maintenance bad copy
      - vidarh 2y ago
        When I did contracting in this space, my AWS based customers invariably needed far more of my time for similar sized setups than the non-cloud customers. The assumption that the non-cloud users will need to pay for any extra people is the opposite of my experience actually running both types of setup. I used to offer to cut their costs by moving them off AWS, but didn't push it very hard because I made far more money from the ones that insisted on using AWS. Some of them had good reasons. Many were just cargo-culting. You don't need to build datacenters, or rack servers yourself, or even own servers, or even ever having seen your servers in order to have racks of dedicated bare metal colo racks of servers. Plenty of service-providers will rent you those services by the hour, and plenty of server providers will rent or lease-to-own hardware for you, so you get monthly bills. I started working in this space in 1995, and while we've often had servers actually on prem, I've also managed racks worth of servers I've never seen in person, in data centres I've never seen, with the help of "remote hands" provided by colo provider employees I've never met.
  - coldtea 2y ago
    Not if they use for long term storage or datasets, records, files and such. That's if they use for infrastructure e.g. for some web service etc. And even then, it takes much less to use it, than to even begin designing such a solution.
- influx 2y ago
  Exactly, not many internal solutions have 11 9s of durability across multiple AZs like S3.
  - vidarh 2y ago
    Mostly because almost nobody needs it, and even fewer need it in that form. And if you need that, then even if AWS can deliver it, you need more than one supplier.
    - scarface_74 2y ago
      Because dealing with that much data spread across suppliers is not going to introduce complexity and latency.
      - vidarh 2y ago
        Of course it will introduce complexity and latency, but if durability is so important that you need 11 9's, then you don't have a realistic choice unless all you care about is the appearance of it. Put another way: I was around when WorldCom went bankrupt and their datacentre in half the building we were also in was locked down by administrators. Had a server fail? Tough. You wouldn't get in. They were not Amazon of today sized, but they had $20bn/year revenue. That's one of many failure modes that can take you out if you rely on one provider. I'm uneasy about relying on one provider for far lower durability projects - I usually ensure we at least have backups somewhere else. But if you care enough that you pick AWS due to a need for 11 nines, and you don't include the long list of failure scenarios that include anything from a bankruptcy (yes, it's unlikely), to them kicking you off intentionally or by accident, or you failing to pay a bill, or any number of other scenarios, they might have 11 nines durability, but you don't have a guarantee you'll get access to it. I'm not arguing against using AWS. It's expensive, but it has its uses. I use it at my current employer because our customers want it, and it's not a const-sensitive product. Picking it as part of a strategy to manage risks is entirely valid. But it's certainly not going to save you from complexity unless you cut corners because they get you to ignore risks.
- nostrebored 2y ago
  “We’re highly available within a single data center” - CIO of a bank that I was working with For storage in particular this gets dicey. Is there an off site backup? What data loss is acceptable for that recovery? What availability requirements do you have? Tales of faulty drives destroying companies are everywhere.
  - dylan604 2y ago
    After the '96 OKC bombing, banks came to a whole new appreciation of redundancy. I was only a few years out of high school working freelance gigs, but one of them was following the progress of a specific bank implementing primary/secondary/tertiary sites and how long the secondary becomes the primary and the tertiary is promoted to secondary. Obviously, this was pre-cloud, so it was even more of a herculean task. It was also the first time I saw a large tape library with robot changer, so it sticks out in the memory banks.
    - dylan604 2y ago
      errr, '95 OKC bombing. <facepalm> I'd love to call it a typo, but I definitely misremembered the date.
- prmoustache 2y ago
  > For some companies this would mean creating a whole department to manage that. If you are already managing that much data, you already have a whole department to manage that. And in the end I haven't seen a significant decrease of people to manage stuff in the cloud. Sure you don't have to manage the hardware infra but you still manage tons of smaller virtual infrastructures on a myriad of AWS accounts. Sysadmins just get relabeled cloudops, devops and/or devsecops , end up being paid more, and thus more expensive. That is at least my observation. I get paid more than I used to but the management hasn't been simplified. The only thing that has changed dramatically is you aren't dependent anymore on delivery time for storage, servers and network equipment.
  - paxys 2y ago
    Managing the data and managing the infrastructure to host that data are very different problems. As discussed elsewhere in this this thread you can store 100 petabytes of data in glacier for $1.2M/yr. That's literally the cost of 1-2 experienced engineers.
    - prmoustache 2y ago
      1. chance is that such amount of data is not sitting alone. It is online to be accessed. So you can't just factor glacier as a single cost. 2. a datacenter doesn't require many employees, most of them not being engineers themselves. To get back to my remark on point 1, while managing your own datacenters just to host storage would be expensive, it is usually mutualized with the rest of the infrastructure you need to access and use the data. 3. colo datacenters
    - vidarh 2y ago
      That might be the cost of 1-2 experienced engineers in Silicon Valley. So don't host there. $1.2m is also for the Deep Archival storage class which has 12 hour retrieval times, and so competes with a handful of tape robots w/caching. It's not something you would have 1-2 experienced engineers spend full time managing - it's something you'd have remote hands and a support contract in place for, and your engineers spending a small fraction of their time managing.
cozzyd 2y ago
Maxar is a case that maybe makes sense. They have boatloads of satellite imagery that needs to be delivered to customers on demand.

coldtea 2y ago

Good luck achieving the realibility and security of AWS by managing it yourself at anything "competitive". For some companies this would mean creating a whole department to manage that.

spamizbad 2y ago
With that much data you're paying a fortune for AWS Operations people either way. Every company I've seen that tries to large-scale AWS thing lean ends up driving themselves into a ditch where they spend millions on contractors, consultants and 3rd party vendors to dig themselves out of. And when they're in that ditch they are neither secure nor reliable.
- nostrebored 2y ago
  I can set up an S3 bucket, efs, access, and backups, all reliable and geographically distributed in a single working day.
  - spicyusername 2y ago
    Yes, indeed. This is exactly the sneaky thing about the cloud. It seems so "simple". Now try doing that for multiple multi-thousand person orgs with competing processes, architectures, product cycles, etc. You'll find very quickly that costs spiral out of control in a very unpredictable and irreversible way. Self-hosting is something a company has much tighter control of the costs for. Most of those costs are known up front as capex purchases and predictable opex expenditures. They tend to seem expensive compared to the PoC S3 demo you just did, but often aren't compared to the actual realized expenses after migrating.
    - nostrebored 2y ago
      … I have owned core product teams in multi thousand person companies and can confidently say that your spiraling costs sound like a lack of planning and product ownership. Platform teams are your friend. Costs being well known for self hosting is just wrong. It completely ignores cost of ownership and tail risk for your infra
- scarface_74 2y ago
  You’re right. Because every company you have seen with a large scale implementation on AWS must mean that all large companies - including Netflix - are dumb.
  - coldtea 2y ago
    Is there an argument in your comment? For my argument not "all large" companies need to be dumb, as in the strawman above. It's enough to make more sense for the average company to do so. And there are tons of companies, large, SME and others, with big data to store, that have no ops or IT department as such, or just a couple of people. They're not going to build datacenters of their own, distribute them globally, research into reliability and failover solutions, add some custom or adapt some third party stack, and so on. The salary for just an extra person could tower their AWS costs.
    - scarface_74 2y ago
      I can’t quite grok from your original comment are you saying that every company who tried to duplicate what AWS does on prem themselves end up spending too much money on consultants etc or are you saying that companies that use AWS still spend too much money on consultants and they don’t have any cost savings?
      - coldtea 2y ago
        Am saying that: - most companies will do better to not try to replicate AWS (or the share of AWS functionality that they need) - most companies aren't tech companies (not talking about the Netflixes and co of the world here) and would botch it, and at best get a costly + high maintenance bad copy
    - vidarh 2y ago
      When I did contracting in this space, my AWS based customers invariably needed far more of my time for similar sized setups than the non-cloud customers. The assumption that the non-cloud users will need to pay for any extra people is the opposite of my experience actually running both types of setup. I used to offer to cut their costs by moving them off AWS, but didn't push it very hard because I made far more money from the ones that insisted on using AWS. Some of them had good reasons. Many were just cargo-culting. You don't need to build datacenters, or rack servers yourself, or even own servers, or even ever having seen your servers in order to have racks of dedicated bare metal colo racks of servers. Plenty of service-providers will rent you those services by the hour, and plenty of server providers will rent or lease-to-own hardware for you, so you get monthly bills. I started working in this space in 1995, and while we've often had servers actually on prem, I've also managed racks worth of servers I've never seen in person, in data centres I've never seen, with the help of "remote hands" provided by colo provider employees I've never met.
- coldtea 2y ago
  Not if they use for long term storage or datasets, records, files and such. That's if they use for infrastructure e.g. for some web service etc. And even then, it takes much less to use it, than to even begin designing such a solution.
influx 2y ago
Exactly, not many internal solutions have 11 9s of durability across multiple AZs like S3.
- vidarh 2y ago
  Mostly because almost nobody needs it, and even fewer need it in that form. And if you need that, then even if AWS can deliver it, you need more than one supplier.
  - scarface_74 2y ago
    Because dealing with that much data spread across suppliers is not going to introduce complexity and latency.
    - vidarh 2y ago
      Of course it will introduce complexity and latency, but if durability is so important that you need 11 9's, then you don't have a realistic choice unless all you care about is the appearance of it. Put another way: I was around when WorldCom went bankrupt and their datacentre in half the building we were also in was locked down by administrators. Had a server fail? Tough. You wouldn't get in. They were not Amazon of today sized, but they had $20bn/year revenue. That's one of many failure modes that can take you out if you rely on one provider. I'm uneasy about relying on one provider for far lower durability projects - I usually ensure we at least have backups somewhere else. But if you care enough that you pick AWS due to a need for 11 nines, and you don't include the long list of failure scenarios that include anything from a bankruptcy (yes, it's unlikely), to them kicking you off intentionally or by accident, or you failing to pay a bill, or any number of other scenarios, they might have 11 nines durability, but you don't have a guarantee you'll get access to it. I'm not arguing against using AWS. It's expensive, but it has its uses. I use it at my current employer because our customers want it, and it's not a const-sensitive product. Picking it as part of a strategy to manage risks is entirely valid. But it's certainly not going to save you from complexity unless you cut corners because they get you to ignore risks.
nostrebored 2y ago
“We’re highly available within a single data center” - CIO of a bank that I was working with For storage in particular this gets dicey. Is there an off site backup? What data loss is acceptable for that recovery? What availability requirements do you have? Tales of faulty drives destroying companies are everywhere.
- dylan604 2y ago
  After the '96 OKC bombing, banks came to a whole new appreciation of redundancy. I was only a few years out of high school working freelance gigs, but one of them was following the progress of a specific bank implementing primary/secondary/tertiary sites and how long the secondary becomes the primary and the tertiary is promoted to secondary. Obviously, this was pre-cloud, so it was even more of a herculean task. It was also the first time I saw a large tape library with robot changer, so it sticks out in the memory banks.
  - dylan604 2y ago
    errr, '95 OKC bombing. <facepalm> I'd love to call it a typo, but I definitely misremembered the date.
prmoustache 2y ago
> For some companies this would mean creating a whole department to manage that. If you are already managing that much data, you already have a whole department to manage that. And in the end I haven't seen a significant decrease of people to manage stuff in the cloud. Sure you don't have to manage the hardware infra but you still manage tons of smaller virtual infrastructures on a myriad of AWS accounts. Sysadmins just get relabeled cloudops, devops and/or devsecops , end up being paid more, and thus more expensive. That is at least my observation. I get paid more than I used to but the management hasn't been simplified. The only thing that has changed dramatically is you aren't dependent anymore on delivery time for storage, servers and network equipment.
- paxys 2y ago
  Managing the data and managing the infrastructure to host that data are very different problems. As discussed elsewhere in this this thread you can store 100 petabytes of data in glacier for $1.2M/yr. That's literally the cost of 1-2 experienced engineers.
  - prmoustache 2y ago
    1. chance is that such amount of data is not sitting alone. It is online to be accessed. So you can't just factor glacier as a single cost. 2. a datacenter doesn't require many employees, most of them not being engineers themselves. To get back to my remark on point 1, while managing your own datacenters just to host storage would be expensive, it is usually mutualized with the rest of the infrastructure you need to access and use the data. 3. colo datacenters
  - vidarh 2y ago
    That might be the cost of 1-2 experienced engineers in Silicon Valley. So don't host there. $1.2m is also for the Deep Archival storage class which has 12 hour retrieval times, and so competes with a handful of tape robots w/caching. It's not something you would have 1-2 experienced engineers spend full time managing - it's something you'd have remote hands and a support contract in place for, and your engineers spending a small fraction of their time managing.

cozzyd 2y ago

Maxar is a case that maybe makes sense. They have boatloads of satellite imagery that needs to be delivered to customers on demand.