5 ms·

> "So we moved to 32 core, 64 GB RAM VMs. By changing a single line of configuration, we upgraded every engineer’s machine." On GitHub, that instance type is $

> "So we moved to 32 core, 64 GB RAM VMs. By changing a single line of configuration, we upgraded every engineer’s machine." On GitHub, that instance type is $2.88/hour or $2,073 monthly per developer for a single instance. (Granted, that's running 24/7 but still - wow, that's expensive for a single instance)

sharms 5y ago
While that does sound quite expensive, companies typically have a limited budget for employee laptops, but appear to have almost unlimited budget for cloud hosted testing environments, which usually have many instances (and services like RDS / Redis etc)
- koyote 5y ago
  Our company is the opposite, I have a max-specced laptop and we are actively trying to reduce the massive cost of cloud computing. A powerful laptop is a drop in the ocean compared to how much cloud VMs cost per developer.
switch007 5y ago
Something tells me that a Microsoft company isn’t paying Azure RRP prices
yurishimo 5y ago
This was the thing that jumped out at me! I'm not sure how the VSCode application interacts with these to potentially put them to sleep, but dang that is $$$ for a dev machine at the scale github is operating at. I guess lucky for them to be owned by Microsoft now and so that's just a cost of doing business. I have a hard time believing they could have even considered an approach like this if they were still paying for hardware out of pocket. Kudos to the team for figuring it out though.
iruoy 5y ago
I assume they're getting quite a sizable discount being owned by Microsoft. I do wonder how many VM's they have running though at different points of the day and how many spares they have prepared.
015a 5y ago
That pricing is a ~70% markup over standard Azure rates. Companies typically buy laptops with an assumed 3 year lifespan; so, let's assume a high-end $3000 laptop, that would be ~$83/month. Of course, you need a computer to access Github Codespaces, so this isn't saving all of that money. Maybe companies can cut some costs by buying cheaper laptops? Or if you want a more apples-to-apples; a Lenovo ThinkStation P620 runs ~$2800 for 32 threads and 64gb memory. DIY would also land somewhere around that $3K mark; a Threadripper 2950x (32 thread) is $1100; 64gb of memory is around $250. This, of course, is the most expensive option they have; the more standard option would be 4/8; for which you would pay ~$85/mo (all the time, normal work hours), for the privilege of a machine that's far less powerful than any laptop any of our developers have. For god's sake, my M1 MacBook Air is an 8/16, with four of those cores far more powerful than any three year old Xeon Azure has running. And it was like $1400. I understand maintaining dev machines isn't the easiest thing in the world. I have never once worked anywhere where it was such a problem that the company would justify tens of thousands of dollars in spend. Because it kind of is one-or-the-other; time invested in making Github Codespaces work for your company is time not invested in maintaining local dev environments. And Github's ideal end-state is companies totally rely on Codespaces, so the local dev environments languish. Its not worth it.
- djanogo 5y ago
  Even with Codespace your company will still have to buy you a laptop, to run corporate VPN, security/virus software, background screen recording, and zoom/video calls. All of this will still need powerful reliable business grade laptop. The cost difference between laptop meant for remote development vs regular dev laptop is less than $1k. ($300 for CPU upgrade, $200 for RAM, $200 for SSD upgrade)?
  - ghthor 5y ago
    :sigh: this is to close to home. Nothing like malfunctioning virus software + background screen recording to eat all your local machine ram and cpu cycles.
  - cameronbrown 5y ago
    I think a lot of these things are less important by the day though. Are VPNs or advanced security software even necessary if everything is in Notion/Google Docs and something like Codespaces is being used?
hibikir 5y ago
The kinds of bills that some large company development environments end up running are surprisingly high, but those companies also think the opportunity cost of losing productivity on application team developers to also very high. That's how you find companies that decide to hire very expensive teams to, say, add types to a dynamic language, rewrite a standard library, invest in massive parallelism to run thousands of CPU hour integration tests in 15 minutes, or many other projects like that which no normal company would ever consider. If it gets some project team to cut delivery times by 3 weeks, it's all considered worthwhile. Where we run into trouble is when we try to copy the shape of those big companies, and the kind of initiatives they fund, into startups, or mid-range companies. In quite a few ways, those top companies are better at development, but the base costs are only worthwhile if you have a money fountain of real revenue that grows far faster than your dev expenses. Trying to copy them with very different conditions is going to lead to tough problems, possibly company killing, and it's the kind of imitation we see all over the industry. So yes, someone like Github is going to have eye popping expenses, because the alternatives to avoid those expenses just don't make sense to them. If you are not working in a multi billion dollar company, their practices can be interesting, but it makes as much sense to emulate them as it'd make for you to hire an entire Formula 1 pit crew to keep your commuter car in good shape.
lkbm 5y ago
If spinning up/down is fast+smooth (e.g., no reconfiguring stuff each time), you could drop that down to ~40h/week taking you to under $500/month. Still pricey, but combined with not having to pay the ~70% markup another commenter mentioned, it could be under $2,000/year.
- banana_giraffe 5y ago
  Add on top of that, spinning down should be automatic. I've worked on bespoke platforms like this where it wasn't. Devs would often forget and/or be more willing to run long running tasks after work hours. The net result was pretty much having to assume an instance would be online 24x7 unless there were at least gentle pokes to disable instances.
koyote 5y ago
And that's on top of the price for every engineer's machine that is used to access the instance... Sure, you might not need to buy machines that are as beefy (although I bet you'll still need a decent machine to run all your chrome/electron windows) but enterprise maintenance on these machines is still a significant cost. I definitely don't think one can look at this from the 'saving money' angle.

sharms 5y ago

While that does sound quite expensive, companies typically have a limited budget for employee laptops, but appear to have almost unlimited budget for cloud hosted testing environments, which usually have many instances (and services like RDS / Redis etc)

koyote 5y ago
Our company is the opposite, I have a max-specced laptop and we are actively trying to reduce the massive cost of cloud computing. A powerful laptop is a drop in the ocean compared to how much cloud VMs cost per developer.

switch007 5y ago

Something tells me that a Microsoft company isn’t paying Azure RRP prices

yurishimo 5y ago

This was the thing that jumped out at me! I'm not sure how the VSCode application interacts with these to potentially put them to sleep, but dang that is $$$ for a dev machine at the scale github is operating at. I guess lucky for them to be owned by Microsoft now and so that's just a cost of doing business. I have a hard time believing they could have even considered an approach like this if they were still paying for hardware out of pocket. Kudos to the team for figuring it out though.

iruoy 5y ago

I assume they're getting quite a sizable discount being owned by Microsoft. I do wonder how many VM's they have running though at different points of the day and how many spares they have prepared.

015a 5y ago

That pricing is a ~70% markup over standard Azure rates. Companies typically buy laptops with an assumed 3 year lifespan; so, let's assume a high-end $3000 laptop, that would be ~$83/month. Of course, you need a computer to access Github Codespaces, so this isn't saving all of that money. Maybe companies can cut some costs by buying cheaper laptops? Or if you want a more apples-to-apples; a Lenovo ThinkStation P620 runs ~$2800 for 32 threads and 64gb memory. DIY would also land somewhere around that $3K mark; a Threadripper 2950x (32 thread) is $1100; 64gb of memory is around $250. This, of course, is the most expensive option they have; the more standard option would be 4/8; for which you would pay ~$85/mo (all the time, normal work hours), for the privilege of a machine that's far less powerful than any laptop any of our developers have. For god's sake, my M1 MacBook Air is an 8/16, with four of those cores far more powerful than any three year old Xeon Azure has running. And it was like $1400. I understand maintaining dev machines isn't the easiest thing in the world. I have never once worked anywhere where it was such a problem that the company would justify tens of thousands of dollars in spend. Because it kind of is one-or-the-other; time invested in making Github Codespaces work for your company is time not invested in maintaining local dev environments. And Github's ideal end-state is companies totally rely on Codespaces, so the local dev environments languish. Its not worth it.

djanogo 5y ago
Even with Codespace your company will still have to buy you a laptop, to run corporate VPN, security/virus software, background screen recording, and zoom/video calls. All of this will still need powerful reliable business grade laptop. The cost difference between laptop meant for remote development vs regular dev laptop is less than $1k. ($300 for CPU upgrade, $200 for RAM, $200 for SSD upgrade)?
- ghthor 5y ago
  :sigh: this is to close to home. Nothing like malfunctioning virus software + background screen recording to eat all your local machine ram and cpu cycles.
- cameronbrown 5y ago
  I think a lot of these things are less important by the day though. Are VPNs or advanced security software even necessary if everything is in Notion/Google Docs and something like Codespaces is being used?

hibikir 5y ago

The kinds of bills that some large company development environments end up running are surprisingly high, but those companies also think the opportunity cost of losing productivity on application team developers to also very high. That's how you find companies that decide to hire very expensive teams to, say, add types to a dynamic language, rewrite a standard library, invest in massive parallelism to run thousands of CPU hour integration tests in 15 minutes, or many other projects like that which no normal company would ever consider. If it gets some project team to cut delivery times by 3 weeks, it's all considered worthwhile. Where we run into trouble is when we try to copy the shape of those big companies, and the kind of initiatives they fund, into startups, or mid-range companies. In quite a few ways, those top companies are better at development, but the base costs are only worthwhile if you have a money fountain of real revenue that grows far faster than your dev expenses. Trying to copy them with very different conditions is going to lead to tough problems, possibly company killing, and it's the kind of imitation we see all over the industry. So yes, someone like Github is going to have eye popping expenses, because the alternatives to avoid those expenses just don't make sense to them. If you are not working in a multi billion dollar company, their practices can be interesting, but it makes as much sense to emulate them as it'd make for you to hire an entire Formula 1 pit crew to keep your commuter car in good shape.

lkbm 5y ago

If spinning up/down is fast+smooth (e.g., no reconfiguring stuff each time), you could drop that down to ~40h/week taking you to under $500/month. Still pricey, but combined with not having to pay the ~70% markup another commenter mentioned, it could be under $2,000/year.

banana_giraffe 5y ago
Add on top of that, spinning down should be automatic. I've worked on bespoke platforms like this where it wasn't. Devs would often forget and/or be more willing to run long running tasks after work hours. The net result was pretty much having to assume an instance would be online 24x7 unless there were at least gentle pokes to disable instances.

koyote 5y ago

And that's on top of the price for every engineer's machine that is used to access the instance... Sure, you might not need to buy machines that are as beefy (although I bet you'll still need a decent machine to run all your chrome/electron windows) but enterprise maintenance on these machines is still a significant cost. I definitely don't think one can look at this from the 'saving money' angle.