6 ms·

If you have petabytes of data in AWS you already have a number of on-staff engineers with significant six figure salaries. If the problem is that your group of

If you have petabytes of data in AWS you already have a number of on-staff engineers with significant six figure salaries. If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation. For reference, a few petabytes of data is not actually that many systems these days, if you have something like a bunch of 72-drive supermicros or equivalent with 14-16TB drives in them. Set up properly this can be administered by one FTE (of course with additional staffing/tech resources for when that FTE is on vacation/unavailable, and appropriate training for other persons who might have admin on the setup). my very rough calculation here says that a 36-drive ZFS RAIDZ2 composed of 16TB drives is something like 492TB (447TiB) usable storage capacity. so five such arrays would be 2460TB. compared to the monthly AWS bill for 2.0 to 2.5TB of data you could probably afford to entirely duplicate the whole setup in a twin identical set of hardware at a geographically diverse off-site location.

amluto 5y ago
You have to factor in the cost of egress from AWS to your nice colocated drive.
- walrus01 5y ago
  welcome to the hotel california... Last thing I remember, I was Running for the door I had to find the passage back To the place I was before "Relax, " said the night man, "We are programmed to receive. You can check-out any time you like, But you can never leave! "
  - amluto 5y ago
    And this is why I don’t think AWS will lower egress fees in response to R2. AWS may be more interested in discouraging people from using egress than in capturing the revenue from egress. I predict that, at most, we’ll see a narrowly tailored reduction in egress fees that is designed to be entirely useless for communication between server applications.
PragmaticPulp 5y ago
> If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation. It's not about whether or not the engineers can make the colocated setup work. It's that you're going to pay a lot of hidden costs with a colocated setup. Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours. Each additional engineer you have to hire to help with the colocated setup is $200-400K fully loaded out of your company's budget. If you have to hire 3 additional engineers to fill out your colocated on-call schedule and help set up and maintain the system, that's easily an extra $1 million per year on your budget. Cloud is expensive, but $1 million goes a long way. It's easy to look at a potential AWS bill and a potential colocation and hardware bill and declare colocation the winner, but then you still have to set up and maintain it all as well as constantly train everyone on it. With AWS, you can hire engineers with AWS experience and they'll understand the big picture of how to work with things on day 1. With a custom setup, you're at the whims of whichever employees set up the system because they know it best. Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not.
- walrus01 5y ago
  > Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not. Everything above really sounds like it's just regurgitating AWS sales person talking points. Sounds like a systemic management / CTO-level problem to me if a company isn't willing to put in place the hiring practices and compensation, documentation systems and operational procedures to deal with that sort of concern. If your core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with. > Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours. If a company can't hire datacenter techs to install hardware, cables, and swap hardware as smart remote hands, maintaining as little as a couple of 45RU cabinets of gear, you also have other management/systemic problems to deal with. I'm looking at this from the point of view of a facilities based bare metal ISP that owns/runs all of its own hardware, and can tell you it's not rocket science.
markus_zhang 5y ago
We are probably looking at the new future in which Cloud computing == Mainframes of the 50s~80s and fewer and fewer people even know how to run the whole scene on-premise. People who got into cloud computing early (mostly by luck) get to win big bucks and better life styles while others try to dispel the magic from left and right.
- walrus01 5y ago
  ultimately, though, somebody has to own, house and run those mainframes, so it's just abstracting the work away to some other group of people. lots of people made careers out of running mainframes and minicomputers in the 1955-1985 time frame. in the case of things like aws, azure, etc, it's just a lot more centralized in a smaller number of gargantuan companies.
  - markus_zhang 5y ago
    Since the future is pretty much set, I think it's more relevant to try to obtain skills (albeit more difficult to obtain because fewer companies have them) and jump into the wagon.
chucky_z 5y ago
Engineers who can put together and maintain this hardware don't exist on the job market.
- walrus01 5y ago
  Engineers who can put together and maintain 2PB of storage don't exist on the job market? I'd say you don't know the right people or aren't looking in the right places. 2PB is really not that much stuff these days. It's less than two cabinets of equipment and that amount of space (80RU or so) includes routers, switches, AC power distribution, OOB, etc.
  - spookthesunset 5y ago
    They exist but finding and recruiting those people costs a non trivial amount of time and money. Both of which could be spent on whatever secret sauce your company does.
    - Dylan16807 5y ago
      The thing is, the bar for managing a rack or two of off the shelf hardware is maybe an inch higher than the bar for managing just one server. Is it super hard to find someone that can handle physical servers at all?
__turbobrew__ 5y ago
I work in a very data heavy HPC space and you are glossing over many things. * to get performant access to a storage cluster is non-trivial, there are many different variables in place which must be correctly tuned to get good performance. Network topology, high quality NICs and switches, a tuned Linux kernel, client side caching settings, network packet sizes, file system block sizes, erasure coding settings, etc. * your solution mentions nothing of backups, offsite failovers, or disaster recovery plans. * your solution mentions nothing of a physical datacenter: fire suppression, battery backups, hvac, power supplies, backup generators, server racks, sound isolation, workspaces for hardware maintenance, network cable routing. * If you have multiple geolocations you need to have dark fiber or ip transit between locations with multiple ISPs to have high speed connections between sites without downtime. In addition to the raw costs you have to factor in the lead time of building a qualified infrastructure team, building out the requirements, provisioning hardware and datacenter space, setting everything up, and then tuning everything. With infinite money this is probably still a 2 year lead time at minimum. I do agree that running multi petabyte workloads in AWS is probably not optimal, but when you are a startup in the growth stage it is probably better worth your time throwing VC money at AWS and building out your product. Eventually, the business should probably migrate to self managed infrastructure once the right product fit has been found and the business is looking to streamline.

amluto 5y ago

You have to factor in the cost of egress from AWS to your nice colocated drive.

walrus01 5y ago
welcome to the hotel california... Last thing I remember, I was Running for the door I had to find the passage back To the place I was before "Relax, " said the night man, "We are programmed to receive. You can check-out any time you like, But you can never leave! "
- amluto 5y ago
  And this is why I don’t think AWS will lower egress fees in response to R2. AWS may be more interested in discouraging people from using egress than in capturing the revenue from egress. I predict that, at most, we’ll see a narrowly tailored reduction in egress fees that is designed to be entirely useless for communication between server applications.

PragmaticPulp 5y ago

> If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation. It's not about whether or not the engineers can make the colocated setup work. It's that you're going to pay a lot of hidden costs with a colocated setup. Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours. Each additional engineer you have to hire to help with the colocated setup is $200-400K fully loaded out of your company's budget. If you have to hire 3 additional engineers to fill out your colocated on-call schedule and help set up and maintain the system, that's easily an extra $1 million per year on your budget. Cloud is expensive, but $1 million goes a long way. It's easy to look at a potential AWS bill and a potential colocation and hardware bill and declare colocation the winner, but then you still have to set up and maintain it all as well as constantly train everyone on it. With AWS, you can hire engineers with AWS experience and they'll understand the big picture of how to work with things on day 1. With a custom setup, you're at the whims of whichever employees set up the system because they know it best. Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not.

walrus01 5y ago
> Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not. Everything above really sounds like it's just regurgitating AWS sales person talking points. Sounds like a systemic management / CTO-level problem to me if a company isn't willing to put in place the hiring practices and compensation, documentation systems and operational procedures to deal with that sort of concern. If your core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with. > Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours. If a company can't hire datacenter techs to install hardware, cables, and swap hardware as smart remote hands, maintaining as little as a couple of 45RU cabinets of gear, you also have other management/systemic problems to deal with. I'm looking at this from the point of view of a facilities based bare metal ISP that owns/runs all of its own hardware, and can tell you it's not rocket science.

markus_zhang 5y ago

We are probably looking at the new future in which Cloud computing == Mainframes of the 50s~80s and fewer and fewer people even know how to run the whole scene on-premise. People who got into cloud computing early (mostly by luck) get to win big bucks and better life styles while others try to dispel the magic from left and right.

walrus01 5y ago
ultimately, though, somebody has to own, house and run those mainframes, so it's just abstracting the work away to some other group of people. lots of people made careers out of running mainframes and minicomputers in the 1955-1985 time frame. in the case of things like aws, azure, etc, it's just a lot more centralized in a smaller number of gargantuan companies.
- markus_zhang 5y ago
  Since the future is pretty much set, I think it's more relevant to try to obtain skills (albeit more difficult to obtain because fewer companies have them) and jump into the wagon.

chucky_z 5y ago

Engineers who can put together and maintain this hardware don't exist on the job market.

walrus01 5y ago
Engineers who can put together and maintain 2PB of storage don't exist on the job market? I'd say you don't know the right people or aren't looking in the right places. 2PB is really not that much stuff these days. It's less than two cabinets of equipment and that amount of space (80RU or so) includes routers, switches, AC power distribution, OOB, etc.
- spookthesunset 5y ago
  They exist but finding and recruiting those people costs a non trivial amount of time and money. Both of which could be spent on whatever secret sauce your company does.
  - Dylan16807 5y ago
    The thing is, the bar for managing a rack or two of off the shelf hardware is maybe an inch higher than the bar for managing just one server. Is it super hard to find someone that can handle physical servers at all?

__turbobrew__ 5y ago

I work in a very data heavy HPC space and you are glossing over many things. * to get performant access to a storage cluster is non-trivial, there are many different variables in place which must be correctly tuned to get good performance. Network topology, high quality NICs and switches, a tuned Linux kernel, client side caching settings, network packet sizes, file system block sizes, erasure coding settings, etc. * your solution mentions nothing of backups, offsite failovers, or disaster recovery plans. * your solution mentions nothing of a physical datacenter: fire suppression, battery backups, hvac, power supplies, backup generators, server racks, sound isolation, workspaces for hardware maintenance, network cable routing. * If you have multiple geolocations you need to have dark fiber or ip transit between locations with multiple ISPs to have high speed connections between sites without downtime. In addition to the raw costs you have to factor in the lead time of building a qualified infrastructure team, building out the requirements, provisioning hardware and datacenter space, setting everything up, and then tuning everything. With infinite money this is probably still a 2 year lead time at minimum. I do agree that running multi petabyte workloads in AWS is probably not optimal, but when you are a startup in the growth stage it is probably better worth your time throwing VC money at AWS and building out your product. Eventually, the business should probably migrate to self managed infrastructure once the right product fit has been found and the business is looking to streamline.