6 ms·

Haha! Back in ~2014 or so my company was spending nearly $30,000/month on an EC2 "compute-optimized" cluster to transcode live video streams to multiple rendit

Haha! Back in ~2014 or so my company was spending nearly $30,000/month on an EC2 "compute-optimized" cluster to transcode live video streams to multiple renditions. One of our engineers said hey, why don't we try to colo some real hardware? We did a test with a single bare-metal 8-core Xeon server and it completely destroyed the performance of the EC2 "compute-optimized" cluster! After that we colo'd 4 big Xeon servers for about $1,600/month total. Looking back on it now it's just so insane...$30,000? no way.

jamal-kumar 4y ago
I don't use AWS for a damn thing because of exorbitant costs. I just don't get why people think that it's necessary other than that they're the types to get drawn into marketing hype. There's just so many better things for your company to be spending the money on.
- redredrobot 4y ago
  The argument is - engineers are expensive so why pay for the expertise to setup and run machines? There's just so many better things for your company to be spending the money on.
  - jamal-kumar 4y ago
    What if my company is pretty much all engineers? We don't let pencil pushers with MBAs anywhere near what we're doing, and it's going great. I know this isn't the most usual configuration but if undervaluing my skills and trying to bottom dollar on them is going to be their rules, then I'm just going to do my own thing, and they're just going to have to scrape the bottom of the barrel for talent. I hope the zeitgeist changes any time soon. God knows how many unicorns have been sacrificed with that kind of paradigm which could be successful companies by now.
    - nemothekid 4y ago
      >What if my company is pretty much all engineers? I don't know how that changes the equation. No one is undervaluing your skills; it's would you rather spend your time driving to a colo center to replace a RAID array or working on $product. With AWS you are outsourcing an IT team, not just processors and how you approach pricing should reflect that.
      - oogali 4y ago
        I hear this bad argument often (“replacing hard drives”) and I don’t understand why. It’s as if we’re mentally stuck in a bad hacking movie from 1999. If you’re doing colocation to save money, you’ve also figured out that going to the datacenter sucks and it’s a terrible place to do work. You’re not building your own servers from scratch, you’re generally purchasing them from a vendor who offers a warranty and optional on-site service. Or you’re leasing them from a hosting company who will take care of those pesky RAID alarms for you. You (or your hosting providers) have likely outfitted your server with remote out-of-band access to allow you to get into BIOS or the RAID controller without physically being in front of the server. And finally, you have remote access to power cycle the server (or a batphone at your hosting provider to do it on your behalf). I want to say that these datacenter-visit-prevention techniques have been near standard practice for a decade-and-a-half. Or is it just me and my circle that do this?
        franga2000 4y ago
        > Or is it just me and my circle that do this? Nope, this seems to be the norm. I've worked on a couple colo servers that nobody at the company had ever actually seen in person. They figured out colo in Germany was the best deal, so they had some servers delivered straight to the DC and the staff there installed them and plugged them into an IP KVM. Not sure if this is a standard service most providers offer, but I'm sure a big enough cheque would convince most - and considering the cost of transporting both the hardware and engineer to install it, that cheque can be quite large.
        jamal-kumar 4y ago
  - 112233 4y ago
    Never understood that argument. How exactly an expertise of setting up and running (your cloud provider) instances is cheaper than expertise of setting up a physical machine?
- m_kos 4y ago
  I agree that cloud compute is usually (too) expensive, but it is sometimes useful and cost-effective. Not long ago, I need to very quickly run a one-off analysis, which required over 200 GB of RAM. It was much cheaper and faster to spin up one VM on GCP for a day than having to order parts, etc.
  - zimpenfish 4y ago
    Similar - wanted to see how much ram gron would need for a particular file and needed much more than the 16GB or 32GB available at home. Quick and easy to spin up a 192GB EC2 to verify it needed ~95GB all told. No way I could have done that without something like AWS.
    - zasdffaa 4y ago
      Depends... set the pagefile to something large, run the program. If it doesn't thrash too hard you can observe it to its largest growth. You'd probably want an SSD. Maybe that would work. Maybe.
  - jamal-kumar 4y ago
    What was the price?
    - selcuka 4y ago
      No idea about GCP but a 32-CPU Digital Ocean droplet with 256GB of RAM costs $1,280/month, which is less than $43/day.
    - m_kos 4y ago
      IIRC, about $120.
zimbu668 4y ago
So you saved, $28,400 a month. Did that make a material difference to the company? I often tell people above me I could save us $10k a month at AWS and generally the response is, "Yeaaaaaaah that's great, could you do XYZ instead to help us land an additional $1M in ARR?"
- KSteffensen 4y ago
  The people above you suck. That's a terrible attitude. Signalling that saving money or just generally improving systems doesn't matter will not build a culture of technical excellence and ambition. Also, it's not peanuts. How many extra developers are $28400 per month?
  - tlamponi 4y ago
    no idea of $ but say they'd get 5000€ a month (quite OK for most places europe), the company would need to pay out a bit more here (for some tax/social benefit things), 7000€ would be pretty realistic, so 4 developers could be hired. This then means that while op wouldn't immediately add 1M ARR, the original dev and the new devs could soon add 5M ARR (stupid extrapolation, it won't be as much in practice, at least normally)
    - azmodeus 4y ago
      I think you underestimate European taxes as an employer. In general it costs 2x to employ people in my experience. Taxes and social security contributions are massive
      - tlamponi 4y ago
        No I don't, I'm employed in Europe in a leadership role and thus talk with the CEO about salaries and general money flow, and I'm also able to read my pay slip, which is quite detailed and also lists all "Lohnnebenkosten" (side/extra costs for employer on top of my brutto money), iow. what the employer really pays.
    - 4y ago
davnn 4y ago
For some use cases though, the cloud is just amazingly cheap and fast. We are currently scaling our (computationally cheap) batch re-training jobs on AWS Lambda and it‘s quite incredible that you can train thousands of models in parallel with TBs of RAM. There is no on-premise alternative.

jamal-kumar 4y ago

I don't use AWS for a damn thing because of exorbitant costs. I just don't get why people think that it's necessary other than that they're the types to get drawn into marketing hype. There's just so many better things for your company to be spending the money on.

redredrobot 4y ago
The argument is - engineers are expensive so why pay for the expertise to setup and run machines? There's just so many better things for your company to be spending the money on.
- jamal-kumar 4y ago
  What if my company is pretty much all engineers? We don't let pencil pushers with MBAs anywhere near what we're doing, and it's going great. I know this isn't the most usual configuration but if undervaluing my skills and trying to bottom dollar on them is going to be their rules, then I'm just going to do my own thing, and they're just going to have to scrape the bottom of the barrel for talent. I hope the zeitgeist changes any time soon. God knows how many unicorns have been sacrificed with that kind of paradigm which could be successful companies by now.
  - nemothekid 4y ago
    >What if my company is pretty much all engineers? I don't know how that changes the equation. No one is undervaluing your skills; it's would you rather spend your time driving to a colo center to replace a RAID array or working on $product. With AWS you are outsourcing an IT team, not just processors and how you approach pricing should reflect that.
    - oogali 4y ago
      I hear this bad argument often (“replacing hard drives”) and I don’t understand why. It’s as if we’re mentally stuck in a bad hacking movie from 1999. If you’re doing colocation to save money, you’ve also figured out that going to the datacenter sucks and it’s a terrible place to do work. You’re not building your own servers from scratch, you’re generally purchasing them from a vendor who offers a warranty and optional on-site service. Or you’re leasing them from a hosting company who will take care of those pesky RAID alarms for you. You (or your hosting providers) have likely outfitted your server with remote out-of-band access to allow you to get into BIOS or the RAID controller without physically being in front of the server. And finally, you have remote access to power cycle the server (or a batphone at your hosting provider to do it on your behalf). I want to say that these datacenter-visit-prevention techniques have been near standard practice for a decade-and-a-half. Or is it just me and my circle that do this?
      - franga2000 4y ago
        > Or is it just me and my circle that do this? Nope, this seems to be the norm. I've worked on a couple colo servers that nobody at the company had ever actually seen in person. They figured out colo in Germany was the best deal, so they had some servers delivered straight to the DC and the staff there installed them and plugged them into an IP KVM. Not sure if this is a standard service most providers offer, but I'm sure a big enough cheque would convince most - and considering the cost of transporting both the hardware and engineer to install it, that cheque can be quite large.
      - jamal-kumar 4y ago
- 112233 4y ago
  Never understood that argument. How exactly an expertise of setting up and running (your cloud provider) instances is cheaper than expertise of setting up a physical machine?
m_kos 4y ago
I agree that cloud compute is usually (too) expensive, but it is sometimes useful and cost-effective. Not long ago, I need to very quickly run a one-off analysis, which required over 200 GB of RAM. It was much cheaper and faster to spin up one VM on GCP for a day than having to order parts, etc.
- zimpenfish 4y ago
  Similar - wanted to see how much ram gron would need for a particular file and needed much more than the 16GB or 32GB available at home. Quick and easy to spin up a 192GB EC2 to verify it needed ~95GB all told. No way I could have done that without something like AWS.
  - zasdffaa 4y ago
    Depends... set the pagefile to something large, run the program. If it doesn't thrash too hard you can observe it to its largest growth. You'd probably want an SSD. Maybe that would work. Maybe.
- jamal-kumar 4y ago
  What was the price?
  - selcuka 4y ago
    No idea about GCP but a 32-CPU Digital Ocean droplet with 256GB of RAM costs $1,280/month, which is less than $43/day.
  - m_kos 4y ago
    IIRC, about $120.

zimbu668 4y ago

So you saved, $28,400 a month. Did that make a material difference to the company? I often tell people above me I could save us $10k a month at AWS and generally the response is, "Yeaaaaaaah that's great, could you do XYZ instead to help us land an additional $1M in ARR?"

KSteffensen 4y ago
The people above you suck. That's a terrible attitude. Signalling that saving money or just generally improving systems doesn't matter will not build a culture of technical excellence and ambition. Also, it's not peanuts. How many extra developers are $28400 per month?
- tlamponi 4y ago
  no idea of $ but say they'd get 5000€ a month (quite OK for most places europe), the company would need to pay out a bit more here (for some tax/social benefit things), 7000€ would be pretty realistic, so 4 developers could be hired. This then means that while op wouldn't immediately add 1M ARR, the original dev and the new devs could soon add 5M ARR (stupid extrapolation, it won't be as much in practice, at least normally)
  - azmodeus 4y ago
    I think you underestimate European taxes as an employer. In general it costs 2x to employ people in my experience. Taxes and social security contributions are massive
    - tlamponi 4y ago
      No I don't, I'm employed in Europe in a leadership role and thus talk with the CEO about salaries and general money flow, and I'm also able to read my pay slip, which is quite detailed and also lists all "Lohnnebenkosten" (side/extra costs for employer on top of my brutto money), iow. what the employer really pays.
  - 4y ago

davnn 4y ago

For some use cases though, the cloud is just amazingly cheap and fast. We are currently scaling our (computationally cheap) batch re-training jobs on AWS Lambda and it‘s quite incredible that you can train thousands of models in parallel with TBs of RAM. There is no on-premise alternative.