10 ms·

Funny enough, no matter how big these things get, they never seem to make me a lot more productive. If my CI build time at least would go down. Software is so

Funny enough, no matter how big these things get, they never seem to make me a lot more productive. If my CI build time at least would go down. Software is so slow compared to hardware. It's embarrassing that we haven't moved not even a hundredth of what hardware has the last 30 years. Why get this?

joseph_grobbles 3y ago
It's a variation of Parkinson's Law -- we just keep expanding what we are doing to fit the hardware available to us, then claiming that nothing has changed. CI is a fairly new thing. The idea of constantly doing all that compute work again and again was unfathomable not that long ago for most teams. We layer on and load in loads of ancillary processes, checks, lints, and so on, because we have the headroom. And then we reminisce about the days when we did a bi-monthly build on a "build box", forgetting how minimalist it actually was.
- jebarker 3y ago
  This is true, but there's still choice in how we expand and the default seems to be to do it as wastefully as possible.
  - pbjtime 3y ago
    It's what the economy rewards. Simple as that
    - jebarker 3y ago
      I don't totally agree. It's what a short-term view of the economy rewards for sure. But even if that was the only view of the economy I've seen plenty of low-performance software written purely out of cargo culting and/or inability or lack of will to do anything better.
  - lazide 3y ago
    As wastefully as we can get away with no? Not the same thing.
    - jebarker 3y ago
      Yes, your wording is better, I don't believe people are actively trying to be as wasteful as possible.
jiggawatts 3y ago
People get set in their ways. Sometimes entire industries need a shake-up. This never occurs voluntarily, and people will wail and thrash about even as you try to help them get out of their rut. Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A typical modern CI/CD pipeline is like a smorgasboard of worst-case scenarios for performance. Let me list just some of them: - Everything is typically done from scratch, with minimal or no caching. - Synchronous I/O from a single thread, often to a remote cloud-hosted replicated disk... for an ephemeral build job. - Tens of thousands of tiny files, often smaller than the physical sector size. - Layers upon layers of virtualisation. - Many small HTTP downloads, from a single thread with no pipelining. Often un-cached despite being identified by stable identifiers -- and hence infinitely cacheable safely. - Spinning up giant, complicated, multi-process workflows for trivial tasks such as file copies. (CD agent -> shell -> cp command) Bonus points for generating more kilobytes of logs than the kilobytes of files processed. - Repeating the same work over and over (C++ header compilation). - Generating reams of code only to shrink it again through expensive processes or just throw it away (Rust macros). I could go on, but it's too painful...
- mschuster91 3y ago
  > Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A lot of what you describe originates from lessons learned in "classic" build environments: - broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step) - someone hot-fixing something on a build server which never got documented and/or impacted other builds, leading to long and weird debugging efforts when setting up more build servers - its worse counterpart, someone setting up the environment on a build server and never documenting it, leading to serious issues when that person inevitably left and then something broke - OS package upgrades breaking things (e.g. Chrome/FF upgrades and puppeteer using it), and a resulting reluctancy in upgrading build servers' software - attackers hacking build systems because of vulnerabilities, in the worst case embedding malware into deliverables or stealing (powerful) credentials - colliding versions of software stacks / libraries / other dependencies leading to issues when new projects are to be built on old build servers In contrast to that, my current favourite system of running GitLab CI on AWS EKS with ephemeral runner pods is orders of magnitude better: - every build gets its own fresh checkout of everything, so no chance of leftover build files or an attacker persisting malware without being noticed (remember, everything comes out of git) - no SSH or other access to the k8s nodes possible - every build gets a reproducible environment, so when something fails in a build, it's trivial to replicate locally, and all changes are documented
moffkalast 3y ago
https://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law
Aurornis 3y ago
> Software is so slow compared to hardware. It's embarrassing that we haven't moved not even a hundredth of what hardware has the last 30 years I don’t understand this mentality. What, exactly, did you expect to get faster? If you run the same software on older hardware it’s going to be much slower. We’re just doing more because we can now. From my perspective, things are pretty darn fast on modern hardware compared to what I was dealing with 5-10 years ago. I had embedded systems builds that would take hours and hours on my local machine years ago. Now they’re done in tens of minutes on my machine that uses a relatively cheap consumer CPU. I can clean build a complete kernel in a couple minutes. In my text editor I can do a RegEx search across large projects and get results nearly instantly! Having NVMe SSDs and high core count consumer CPUs makes amazing things possible. Software is improving, too. Have you seen how fast the new Bun package manager is? I can pick from dozens of open source database options for different jobs that easily enable high performance, large scale operations that would have been unthinkable or required expensive enterprise software a decade ago (even with today’s hardware). > Why get this? If you really think nothing has improved, you might be experiencing a sort of hedonic adaptation: Every advancement gets internalized as the new baseline and you quickly forget how slow things were previously. I remember the same thing happened when SSDs came out: They made an amazing improvement in desktop responsiveness over mechanical HDDs, but many people almost immediately forgot how slow HDDs were. It’s only when you go back and use a slow HDD-based desktop that you realize just how slow things were in the past.
- Almondsetat 3y ago
  why is opening word or excel or powerpoint not instantaneous? plenty of people, me included, have to sieve through vast amounts of document and constantly open/close them
  - mejutoco 3y ago
    Maybe they are waiting to make them right before making them fast? /s (Make it work, make it right, make it fast)
  - JohnBooty 3y ago
    Application "bloat" is one obvious thing, but operating systems are also doing a lot more "work" to open an app these days -- address space randomization, checking file integrity, checking for developer signing certs, mitigations against hardware side channel attacks like rowhammer or whatever, etc. Those things aren't free. But, I don't know the relative performance hit there compared to good old software bloat.
jayd16 3y ago
Yeah, I remember compressing 4k120hz video of my AI upscaled ray traced RPG play through in real time and streaming it to all my friends' phones in the 90s. Times never change. But really, it's easy to forget the massive leaps we make every year.
- josephg 3y ago
  Maybe I'm just getting old, but diablo 4 doesn't look that much better to my eyes than diablo 3 did. I'm sure the textures are higher resolution and so on, but 5 minutes in I don't notice or care. Its how the game plays that matters, and that has almost never been hardware limited. That said, I'm really looking forward to the day we can embed LLMs and generative AI into video games for like world generation & dialog. I can't wait to play games where on startup, the game generates truly unique cities for me to explore filled with fictional characters. I want to wander around unique worlds and talk - using natural language - with the people who populate them. I'm giddy with excitement over the amazing things video games can bring in the next few years. I feel like a kid again looking forward to christmas.
- hypercube33 3y ago
  there was a video posted to hacker news about core windows apps taking longer to launch on new enough hardware. Stuff like calc, word, paint taking longer to start on win11 vs win 2000 even though machines have much faster everything so I think the point stands - why is everything slower now?
  - lazide 3y ago
    Windows sucks.
switchbak 3y ago
I get your take here, but as someone who's worked very hard at times to optimize builds (amongst other things), the business just generally doesn't respect those efforts and certainly doesn't reward them. Often times they're actively punished with a reflexive assumption that they're not "serious" efforts worth the time of the business. (There's the odd exception, but this is very widespread in my experience) Sure, there's a balance to be made between cutting wood and sharpening the saw. Who do we blame when the boss-man won't allow anyone to sharpen the tools even though we're obviously wasting outrageous amounts of time? You blame the people that won't allow those investments to be made. When you multiply that across an entire industry, add some trendy fashionable tech (that's also just fast-enough to be tolerable), and this is how we end up in the shitty circumstance you describe. And yet I still wouldn't trade my fancy IDE and slow CI pipelines for a copy of Turbo Pascal 7, as fast as it would be!
redox99 3y ago
> Funny enough, no matter how big these things get, they never seem to make me a lot more productive. Maybe because the stuff you do isn't bottlenecked by compute? In my case every hardware upgrade resulted in a big productivity improvement. Better CPU: Cut my C++ build times in half (from 10 to 5 minutes if I change an important .h) Better GPUs: Cut my AI training time by a few X, massively improving iteration times. Also allow me to run bigger models that more easily reach my target accuracy.

joseph_grobbles 3y ago

It's a variation of Parkinson's Law -- we just keep expanding what we are doing to fit the hardware available to us, then claiming that nothing has changed. CI is a fairly new thing. The idea of constantly doing all that compute work again and again was unfathomable not that long ago for most teams. We layer on and load in loads of ancillary processes, checks, lints, and so on, because we have the headroom. And then we reminisce about the days when we did a bi-monthly build on a "build box", forgetting how minimalist it actually was.

jebarker 3y ago
This is true, but there's still choice in how we expand and the default seems to be to do it as wastefully as possible.
- pbjtime 3y ago
  It's what the economy rewards. Simple as that
  - jebarker 3y ago
    I don't totally agree. It's what a short-term view of the economy rewards for sure. But even if that was the only view of the economy I've seen plenty of low-performance software written purely out of cargo culting and/or inability or lack of will to do anything better.
- lazide 3y ago
  As wastefully as we can get away with no? Not the same thing.
  - jebarker 3y ago
    Yes, your wording is better, I don't believe people are actively trying to be as wasteful as possible.

jiggawatts 3y ago

People get set in their ways. Sometimes entire industries need a shake-up. This never occurs voluntarily, and people will wail and thrash about even as you try to help them get out of their rut. Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A typical modern CI/CD pipeline is like a smorgasboard of worst-case scenarios for performance. Let me list just some of them: - Everything is typically done from scratch, with minimal or no caching. - Synchronous I/O from a single thread, often to a remote cloud-hosted replicated disk... for an ephemeral build job. - Tens of thousands of tiny files, often smaller than the physical sector size. - Layers upon layers of virtualisation. - Many small HTTP downloads, from a single thread with no pipelining. Often un-cached despite being identified by stable identifiers -- and hence infinitely cacheable safely. - Spinning up giant, complicated, multi-process workflows for trivial tasks such as file copies. (CD agent -> shell -> cp command) Bonus points for generating more kilobytes of logs than the kilobytes of files processed. - Repeating the same work over and over (C++ header compilation). - Generating reams of code only to shrink it again through expensive processes or just throw it away (Rust macros). I could go on, but it's too painful...

mschuster91 3y ago
> Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A lot of what you describe originates from lessons learned in "classic" build environments: - broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step) - someone hot-fixing something on a build server which never got documented and/or impacted other builds, leading to long and weird debugging efforts when setting up more build servers - its worse counterpart, someone setting up the environment on a build server and never documenting it, leading to serious issues when that person inevitably left and then something broke - OS package upgrades breaking things (e.g. Chrome/FF upgrades and puppeteer using it), and a resulting reluctancy in upgrading build servers' software - attackers hacking build systems because of vulnerabilities, in the worst case embedding malware into deliverables or stealing (powerful) credentials - colliding versions of software stacks / libraries / other dependencies leading to issues when new projects are to be built on old build servers In contrast to that, my current favourite system of running GitLab CI on AWS EKS with ephemeral runner pods is orders of magnitude better: - every build gets its own fresh checkout of everything, so no chance of leftover build files or an attacker persisting malware without being noticed (remember, everything comes out of git) - no SSH or other access to the k8s nodes possible - every build gets a reproducible environment, so when something fails in a build, it's trivial to replicate locally, and all changes are documented

moffkalast 3y ago

https://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law

Aurornis 3y ago

> Software is so slow compared to hardware. It's embarrassing that we haven't moved not even a hundredth of what hardware has the last 30 years I don’t understand this mentality. What, exactly, did you expect to get faster? If you run the same software on older hardware it’s going to be much slower. We’re just doing more because we can now. From my perspective, things are pretty darn fast on modern hardware compared to what I was dealing with 5-10 years ago. I had embedded systems builds that would take hours and hours on my local machine years ago. Now they’re done in tens of minutes on my machine that uses a relatively cheap consumer CPU. I can clean build a complete kernel in a couple minutes. In my text editor I can do a RegEx search across large projects and get results nearly instantly! Having NVMe SSDs and high core count consumer CPUs makes amazing things possible. Software is improving, too. Have you seen how fast the new Bun package manager is? I can pick from dozens of open source database options for different jobs that easily enable high performance, large scale operations that would have been unthinkable or required expensive enterprise software a decade ago (even with today’s hardware). > Why get this? If you really think nothing has improved, you might be experiencing a sort of hedonic adaptation: Every advancement gets internalized as the new baseline and you quickly forget how slow things were previously. I remember the same thing happened when SSDs came out: They made an amazing improvement in desktop responsiveness over mechanical HDDs, but many people almost immediately forgot how slow HDDs were. It’s only when you go back and use a slow HDD-based desktop that you realize just how slow things were in the past.

Almondsetat 3y ago
why is opening word or excel or powerpoint not instantaneous? plenty of people, me included, have to sieve through vast amounts of document and constantly open/close them
- mejutoco 3y ago
  Maybe they are waiting to make them right before making them fast? /s (Make it work, make it right, make it fast)
- JohnBooty 3y ago
  Application "bloat" is one obvious thing, but operating systems are also doing a lot more "work" to open an app these days -- address space randomization, checking file integrity, checking for developer signing certs, mitigations against hardware side channel attacks like rowhammer or whatever, etc. Those things aren't free. But, I don't know the relative performance hit there compared to good old software bloat.

jayd16 3y ago

Yeah, I remember compressing 4k120hz video of my AI upscaled ray traced RPG play through in real time and streaming it to all my friends' phones in the 90s. Times never change. But really, it's easy to forget the massive leaps we make every year.

josephg 3y ago
Maybe I'm just getting old, but diablo 4 doesn't look that much better to my eyes than diablo 3 did. I'm sure the textures are higher resolution and so on, but 5 minutes in I don't notice or care. Its how the game plays that matters, and that has almost never been hardware limited. That said, I'm really looking forward to the day we can embed LLMs and generative AI into video games for like world generation & dialog. I can't wait to play games where on startup, the game generates truly unique cities for me to explore filled with fictional characters. I want to wander around unique worlds and talk - using natural language - with the people who populate them. I'm giddy with excitement over the amazing things video games can bring in the next few years. I feel like a kid again looking forward to christmas.
hypercube33 3y ago
there was a video posted to hacker news about core windows apps taking longer to launch on new enough hardware. Stuff like calc, word, paint taking longer to start on win11 vs win 2000 even though machines have much faster everything so I think the point stands - why is everything slower now?
- lazide 3y ago
  Windows sucks.

switchbak 3y ago

I get your take here, but as someone who's worked very hard at times to optimize builds (amongst other things), the business just generally doesn't respect those efforts and certainly doesn't reward them. Often times they're actively punished with a reflexive assumption that they're not "serious" efforts worth the time of the business. (There's the odd exception, but this is very widespread in my experience) Sure, there's a balance to be made between cutting wood and sharpening the saw. Who do we blame when the boss-man won't allow anyone to sharpen the tools even though we're obviously wasting outrageous amounts of time? You blame the people that won't allow those investments to be made. When you multiply that across an entire industry, add some trendy fashionable tech (that's also just fast-enough to be tolerable), and this is how we end up in the shitty circumstance you describe. And yet I still wouldn't trade my fancy IDE and slow CI pipelines for a copy of Turbo Pascal 7, as fast as it would be!

redox99 3y ago

> Funny enough, no matter how big these things get, they never seem to make me a lot more productive. Maybe because the stuff you do isn't bottlenecked by compute? In my case every hardware upgrade resulted in a big productivity improvement. Better CPU: Cut my C++ build times in half (from 10 to 5 minutes if I change an important .h) Better GPUs: Cut my AI training time by a few X, massively improving iteration times. Also allow me to run bigger models that more easily reach my target accuracy.