5 ms·

I don't know what the author is building that a build is taking 4 hours, but I am aware that there are huge software projects with build times like that and muc

by _ph_ 7y ago

I don't know what the author is building that a build is taking 4 hours, but I am aware that there are huge software projects with build times like that and much longer. I agree that getting quick feedback to changes is vital for quick overall development. This shows the values of development stacks which reduce the build times a lot. As a resultmore incremental development can be done. For example, the build times were one of the architectural goals when designing the Go language. Dynamic languages are also a great tool to fight build times. Even if it is unsuitable for large projects to be entirely built on dynamic languages, a great technique is using a mix of static and dynamic. Build the core of the application in a static language, but more as a set of libraries to be called from the dynamic layer. The dynamic layer is where most of the application logic resides, easy to be changed. And parts which have stabilized reasonably can still be moved over to the static part.

mister_hn 7y ago
Have you ever tried to build Chromium or Firefox yourself? If you have a slow machine, the build can take even 10 hours or more
- _ph_ 7y ago
  No, I have not. But wouldn't be those code bases which compile very slow because of being written in C++? C++ seems to baloon in building time for large code bases. On the other side, the big Threadripper manages to compile the Linux Kernel in 40s. I don't think those web browsers are that much bigger than Linux than the compile times differ. Firefox is further slowed down by using more and more Rust based code - Rust is known to compile slowly. However I would say this is a good tradeoff, since Rust brings much more safety to one of the most safety-critical applications on a typical desktop.
  - rhoyerboat 7y ago
    the general issue is that the feature rich web browser contains libraries wrapped around basically every leaf and stem of windows, mac and *nix desktop interface family forests. (if your building on say, a completely fresh stage3 gentoo system, that equals some pretty good linker load. cairo comes to mind as a huge culprit, whether its webkit or mozilla.)
  - zlynx 7y ago
    It is extremely easy to let your C++ code get into long build times. People get fond of putting code into the header, for example. This adds that compile time to every file that includes that file. And it prevents you from using forward declaration, which then requires including those definitions in your header. With discipline and tracking the build time, and tools that record how long each build step takes, you can keep people from doing things like declaring a Boost multimap in a header file. They can go ahead and use it but hide it in a cpp source file behind a pointer or a virtual interface. I've noticed that some people fall in love with C++11 move constructors, and they are pretty cool, but unlike a reference or pointer they require full type definitions in order to pass "by value." So that's annoying. C code is definitely faster to build, unless you've got some crazy programmer who reimplemented C++ templates using five level deep nested macros.
- jbverschoor 7y ago
  Then maybe they don't have proper dependencies set. Inclusing too many things, too many acessible stuff, globals, no libraries?
- cozzyd 7y ago
  Doesn't Firefox use -flto? That might be why.
- thechao 7y ago
  Fast builds for C++ require unending diligence from day 1. Part of my team’s CI is a build time check. Our code base runs to 10mm lines of code; clean uncached builds are 3m; cached builds are 30s; incremental builds are in the 100s of ms range. The entire system is a single, short Makefile combined with ccache. There’s no fine-grain dependency tracking, so our builds never go wrong.
scarface74 7y ago
So by using a dynamic language, you trade the cost of a long build time with possible runtime errors that would have easily been caught by the build system? Instead of paying the cost once per build you are now inflicting runtime errors on your customers.
- tyingq 7y ago
  Is there a strong, real world correlation between heavy use of dynamic languages, and a poor user experience (versus competitors, for example)? The theory that it would makes sense to me, but I suspect other factors outshadow any effect it might have. For example, I haven't heard that Facebook, Instacart, Booking.com, WordPress.com, etc, have stability or experience issues worse than their competitors.
- _ph_ 7y ago
  Well, the costs are once per every build, which can take many hours, as the author describes it, even be only available on the next day. Static typechecking is a valuable tool, but it shouldn't be as the solution for all correctness problems. Testing is another. And when the build times become too large, testing suffers. Finally, it is absoultely possible to have dynamic languages with type declarations which can be automatically checked.
  - scarface74 7y ago
    So now you trade reduced build times for increase in the amount of developer time for creating extra tests that check for type correctness and you increase the “build” time by having your build system run more tests? And once you have type declaration that can “automatically” be checked - something has to run against the same code base to check the correctness of your code and you’re back to increasing your build time?
    - _ph_ 7y ago
      No. With dynamic languages, one gets not only "reduced" build times, they are instant. Testing for correctness needs to be done with any kind of language. Type checking is not a replacement for that. But you can add the necessary type declarations to dynamic language to facilitate static type checking if desired. These checks don't influence the build time significantly. As written in my first post, using dynamic languages is only one way of improving development times. The other of course is, use languages which allow for fast compilation.
castratikron 7y ago
I wrote physics simulation code some time ago. The build was fast, but since the simulation speed was about 10ns/day the tests would take several hours to run. And the only way to know if your change fixed something is if the tests passed. So it took several hours to get feedback for a change. But it also took about that long to come up with a new idea to try. Eventually I had a pipeline running where I could fire off a few ideas throughout the day, keep track of the change in the build, and iterate that way.
dpc_pw 7y ago
You're going to lose that gained time as soon as your "build" includes running tests. And you'll need more tests because in dynamic langueses generally nothing works until you check it with a test. Mixing lanuages is a waste. It makes everything harder operationally, technically and business wise. The solution to long builds is projects structure that supports caching and parallelism. No one ever changes more than couple thousands lines even in a multi million line project, so why would everything had to get rebuilt? I've seen ci pipelines building millions of lines in under a minute, no sweat.

by _ph_ 7y ago

mister_hn 7y ago

Have you ever tried to build Chromium or Firefox yourself? If you have a slow machine, the build can take even 10 hours or more

_ph_ 7y ago
No, I have not. But wouldn't be those code bases which compile very slow because of being written in C++? C++ seems to baloon in building time for large code bases. On the other side, the big Threadripper manages to compile the Linux Kernel in 40s. I don't think those web browsers are that much bigger than Linux than the compile times differ. Firefox is further slowed down by using more and more Rust based code - Rust is known to compile slowly. However I would say this is a good tradeoff, since Rust brings much more safety to one of the most safety-critical applications on a typical desktop.
- rhoyerboat 7y ago
  the general issue is that the feature rich web browser contains libraries wrapped around basically every leaf and stem of windows, mac and *nix desktop interface family forests. (if your building on say, a completely fresh stage3 gentoo system, that equals some pretty good linker load. cairo comes to mind as a huge culprit, whether its webkit or mozilla.)
- zlynx 7y ago
  It is extremely easy to let your C++ code get into long build times. People get fond of putting code into the header, for example. This adds that compile time to every file that includes that file. And it prevents you from using forward declaration, which then requires including those definitions in your header. With discipline and tracking the build time, and tools that record how long each build step takes, you can keep people from doing things like declaring a Boost multimap in a header file. They can go ahead and use it but hide it in a cpp source file behind a pointer or a virtual interface. I've noticed that some people fall in love with C++11 move constructors, and they are pretty cool, but unlike a reference or pointer they require full type definitions in order to pass "by value." So that's annoying. C code is definitely faster to build, unless you've got some crazy programmer who reimplemented C++ templates using five level deep nested macros.
jbverschoor 7y ago
Then maybe they don't have proper dependencies set. Inclusing too many things, too many acessible stuff, globals, no libraries?
cozzyd 7y ago
Doesn't Firefox use -flto? That might be why.
thechao 7y ago
Fast builds for C++ require unending diligence from day 1. Part of my team’s CI is a build time check. Our code base runs to 10mm lines of code; clean uncached builds are 3m; cached builds are 30s; incremental builds are in the 100s of ms range. The entire system is a single, short Makefile combined with ccache. There’s no fine-grain dependency tracking, so our builds never go wrong.

scarface74 7y ago

So by using a dynamic language, you trade the cost of a long build time with possible runtime errors that would have easily been caught by the build system? Instead of paying the cost once per build you are now inflicting runtime errors on your customers.

tyingq 7y ago
Is there a strong, real world correlation between heavy use of dynamic languages, and a poor user experience (versus competitors, for example)? The theory that it would makes sense to me, but I suspect other factors outshadow any effect it might have. For example, I haven't heard that Facebook, Instacart, Booking.com, WordPress.com, etc, have stability or experience issues worse than their competitors.
_ph_ 7y ago
Well, the costs are once per every build, which can take many hours, as the author describes it, even be only available on the next day. Static typechecking is a valuable tool, but it shouldn't be as the solution for all correctness problems. Testing is another. And when the build times become too large, testing suffers. Finally, it is absoultely possible to have dynamic languages with type declarations which can be automatically checked.
- scarface74 7y ago
  So now you trade reduced build times for increase in the amount of developer time for creating extra tests that check for type correctness and you increase the “build” time by having your build system run more tests? And once you have type declaration that can “automatically” be checked - something has to run against the same code base to check the correctness of your code and you’re back to increasing your build time?
  - _ph_ 7y ago
    No. With dynamic languages, one gets not only "reduced" build times, they are instant. Testing for correctness needs to be done with any kind of language. Type checking is not a replacement for that. But you can add the necessary type declarations to dynamic language to facilitate static type checking if desired. These checks don't influence the build time significantly. As written in my first post, using dynamic languages is only one way of improving development times. The other of course is, use languages which allow for fast compilation.

castratikron 7y ago

I wrote physics simulation code some time ago. The build was fast, but since the simulation speed was about 10ns/day the tests would take several hours to run. And the only way to know if your change fixed something is if the tests passed. So it took several hours to get feedback for a change. But it also took about that long to come up with a new idea to try. Eventually I had a pipeline running where I could fire off a few ideas throughout the day, keep track of the change in the build, and iterate that way.

dpc_pw 7y ago

You're going to lose that gained time as soon as your "build" includes running tests. And you'll need more tests because in dynamic langueses generally nothing works until you check it with a test. Mixing lanuages is a waste. It makes everything harder operationally, technically and business wise. The solution to long builds is projects structure that supports caching and parallelism. No one ever changes more than couple thousands lines even in a multi million line project, so why would everything had to get rebuilt? I've seen ci pipelines building millions of lines in under a minute, no sweat.