9 ms·

SCons is not slow

24 points by moonchild 6y ago

amelius 6y ago
I hope that FOSS can converge to a single build system as much as possible, because it's not very convenient to learn them all.
- CameronNemo 6y ago
  I don't think a single build system is necessary or desirable. meson (as well as waf and scons) depends on python, which is not so nice for bootstrapping. Thus desktop applications and low level userspace plumbing could benefit from different build systems. That said, I hope waf and scons die off soon (only a handful of packages even use them). CMake and Meson are much better choices.
  - bogwog 6y ago
    What’s wrong with waf? I’ve been using it lately and think it’s great. Writing build complex build tasks is easy, and the performance is excellent. I haven’t tried Meson, but CMake cant really do much besides compile the programming languages it explicitly supports. Automation tasks are pretty much impossible.
    - CameronNemo 6y ago
      Do you have a link to these wscripts, or a pastebin? I am curious on what types of automations you are referring to. I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. Only having to know a half dozen build systems is way easier than having to know waf, Scons, meson, autotools, make, setuptools, cargo, qmake, cmake is a pain. Only 25 packages in the distro I contribute to use waf. Most are audio recording tools that do not get much maintenance. 20 packages use scons. Meanwhile almost 400 use meson, ~500 use qmake, ~1000 use cmake, and over 2000 use autotools.
      - bogwog 6y ago
        > Do you have a link to these wscripts, or a pastebin? I am curious on what types of automations you are referring to. Not currently, but I do have a half-written article I plan to publish eventually once I complete this project I'm working on, to showcase my use of waf and how it's helped me. That's how much I like it! But to give you a quick example: my project (a cross-platform game) has a bunch of assets that I am embedding in the executable. To do this, I wrote a custom waf task that automatically generates a char array in a .cpp/.h bundle, and adds it to my build and include paths. Using that same tool, I am able to embed other assets that need to be transformed in various ways before being embedded. One of those is a bundle of binary files and meta-data combined together inside a single flatbuffer data blob. This involves compiling the schema, generating a small command-line utility to write the blob using the sources generated from the schema compiler, using that utility to generate the blob, and finally embedding the blob using the asset embedding tool. (Plus some of the binary files themselves need to be processed before all of this happens) I could do that with shell scripts, but waf's design makes it easy to do, in a small amount of reusable code (everything I just described I implemented in <400 lines of python), and in a general-purpose and easy to read programming language. Plus the tasks automatically execute in parallel wherever possible. > I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. I've heard that complaint before, and I can see why it would be annoying for outside maintainers. Waf's customizability means two Waf projects might look nothing alike. So even if one becomes an expert at Waf (which is not very easy considering how bad the docs are), they might be completely lost in someone else's project. I failed a game jam once because I was using Amazon Lumberyard (which uses waf), and even though I completed the game, I couldn't figure out how to compile the damn thing before the deadline. But if it's any consolation, I don't plan on distributing my project through any linux distros :P
        Arelius 6y ago
        Having used WAF extensively in large legacy projects, one of it’s large problems is that it isn’t particularly batteries included, and relies on a lot of customizations, and doesn’t have a lot of rails so it’s really easy to break it’s reliability guarantees. And having a unreliable build where one is never sure if a problem will go away by deleting intermediates is super frustrating.
  - amelius 6y ago
    > I don't think a single build system is necessary or desirable. Why not? In the old days, it was "./configure; make". Today, it's much less certain what build system(s) you'll have to deal with when building a FOSS tool. That's not a problem when everything works, but as experience shows, more often than not you'll run into issues and then I would say it's nice if you are familiar with the system. And just like with other tools it's nice if everybody uses the same thing. Github wouldn't exist if everybody rolled their own version control system, for example.
    - progval 6y ago
      > Github wouldn't exist if everybody rolled their own version control system, for example. It also wouldn't exist if no one did. It's all about compromises. Having too many tools for the same job isn't good, but monoculture isn't either.
  - nonbirithm 6y ago
    If Cargo didn't exist I would believe you. But a group of people learning from past mistakes and doing most of the things right from the start has convinced me otherwise. Sadly when it comes to C++ I don't think we'll get a do-over. Too many build systems exist already, and important libraries from a decade or two ago will use those systems until the end of time, and so will the projects that depend on them indirectly.
- dgfitz 6y ago
  I’m going to sit in my little corner of the world and wish hg and qmake had won... while learning all of the other ways to generate Makefiles
- MCOfficer 6y ago
  FOSS build systems are https://xkcd.com/927/ https://xkcd.com/927/ on repeat.
- bawolff 6y ago
  Unifying around a single thing is not something FOSS likes to do.
  - qznc 6y ago
    This is not FOSS specific. In companies you can observe the same tension between the efficiency of unity and the flexibility of diversity.
  - mkoubaa 6y ago
    There's an xkcd for that. In seriousness, I wish the same thing, but it's disruptive and thankless work to ask of volunteers
  - cycloptic 6y ago
    I don't know, a lot of FOSS seems to have unified around Linux for whatever reason.
    - bawolff 6y ago
      Linux is really 5 million different systems though. Debian or redhat (or etc)? Systemd or init? Bash or zsh? Gnome or KDE? X11 or Wayland? GCC or clang? Almost every single component has multiple choices.
- moonchild 6y ago
  I've been tossing around thoughts for this recently. I think the solution is something that, like scons, is built on top of a proper programming language. Unlike scons, however, it should be imperative, rather than declarative. Core primitives, like 'a file is out of date iff its modification time or hash has changed', should be expressible 'in userspace', as it were—the core of the build system shouldn't know about them. (Though it should have some concept of a target being up-to-date and not being rebuilt, it shouldn't need to know about files.) This, combined with the ability to create proper abstractions, means you can emulate the primitives of all existing build systems and give them a common substrate. (Bonus: assuming the core primitives are done right, you can still get performance as good as ninja. Which is good, because you won't be able to compile to a build system like ninja.)
  - aseipp 6y ago
    The answer to this is "use Shake", which is similar to what you describe. The secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files — so you can depend on environment variables, file contents, results of dynamic commands, etc — you need to be able to express "dynamic" rules where you can compute new dependencies for a rule "on the fly" while it's executing, and you need elbow grease to write it as an API. Shake also has had a lot of research done about it comparing it to other systems, is used very successfully, and has an array of nice features other systems don't, including .ninja file compatibility and build profile reports: https://shakebuild.com/ https://shakebuild.com/ Shake is written in Haskell as a library (so your build rules have to be written in Haskell too, as programs) but you could either wire up a language via the FFI to make some classes of build rules easily expressible in JavaScript/Lua, or in theory rewrite the core of Shake in another language too. The theory is all there, though. My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity (multi language builds, requires parsing or other "advanced" features, etc). As a bonus it's also very fast, within a few % of Ninja when parsing Ninja files, and sometimes faster than that. I was surprised when it came in as 2x as fast as Ninja on a large-ish C codebase at work, due to having picked a different "build schedule" that distributed compilations across cores better.
- gravypod 6y ago
  A single-build-system world would need to see a build system that recognizes all languages and use cases as equally important. There's very few build systems that actually support this in a reproducible and hermetic way.
qppo 6y ago
I feel like if you close with "only 20% slower than make" you can't justify the title "is not slow." 20% is a pretty huge margin for large builds, which I wouldn't consider using make for in the first place. CMake + Ninja has shaved enormous amounts of time off my builds from make, so why should I use something that's even slower?
- Y_Y 6y ago
  As a counterpoint, my builds are mostly too fast to care, but modifying the CMake files that describe them is excruciating. It's a miracle of software engineering that CMake functions, but wow is the language awful.
  - cycloptic 6y ago
    Is there something specific you dislike about it besides the syntax? I frequently hear complaints about cmake not supporting one thing or another but a lot of those have been addressed in recent releases. It's not clear to me why you say it's a miracle either, AFAIK it does most of the same things any other makefile generator does. (Disclaimer: I'm not involved in the cmake project but I often see people struggling with it because of misuse, possibly because of expectations that it will work exactly like autotools)
  - cbsmith 6y ago
    Honestly, I switched to bazel and did not look back.
  - umvi 6y ago
    Yeah we switched from Scons to CMake and I have to admit I loved being able to use python for Scons. CMake's language is unintuitive and clunky. I wish CMake had been designed like LLVM with a front and back end so that there could be a python-based front end (for example).
    - bb88 6y ago
      Yeah, I've been unhappy with any automake system for a long time. Haven't tried ninja though. Scons was fine-ish. Now that because work forces me to develop on a locked down windows box, make is the last thing I want.
beervirus 6y ago
Rebuttal here: https://blog.melski.net/2013/12/11/update-scons-is-still-really-slow/ https://blog.melski.net/2013/12/11/update-scons-is-still-rea...
- cbsmith 6y ago
  I'm just kind of amazed that there is this kind of lack of understanding of performance & measurement amongst people working on optimizing low-level technical tools like build systems.
  - ncmncm 6y ago
    To be fair, not all such people; by the evidence, only SCons people.
    - onebaddog 6y ago
      Not even SCons people. (I'm the project maintainer)
    - cbsmith 6y ago
      It was absolutely not intended as a statement about all such people. The surprise is about any such people.
dimator 6y ago
Slow or not, it becomes cryptic to write Sconsfiles with any complexity, and it's impossible to debug.
- sgtnoodle 6y ago
  It's python, so you can write your own abstractions as you see fit. Like any general purpose programming language, you can make something maintainable or you can make a mess.
  - TazeTSchnitzel 6y ago
    Sometimes having a tool prevent you from shooting yourself in the foot is kinder. But it's more than that, a good build system does not need to be Turing complete, and having Turing completeness imposes certain requirements on it that may make it inherently slower.
    - sgtnoodle 6y ago
      Going along with the firearm analogy, it's only fully useful and powerful tool once you've removed it from its case and removed the trigger lock. Any given firearm may have additional safety features such as interlocks, but they're only effective if they stay out of the way when you intend to shoot. You avoid accidentally shooting yourself and others through training, practice and discipline: treat any firearm as if it were loaded, keep your finger off the trigger until you intend to shoot, etc. When setting up a build system for multiple people to use, it's important to agree on a general approach and some rules to follow in order to keep the system from turning into a mess. It's code like anything else, and benefits from training, practice and discipline. Using a language that most folk already know is almost always better than using an obscure one out of a sense that the obscurity makes it safer. When building up a build system, I suggest focusing on designing abstractions to make 90% of use cases trivial for other developers to use. Spitting out libraries and linking them into binaries, existing autocoding scripts, etc. The other 10% of cases that other folk run into, loudly volunteer to make yourself available to help them work through it. Most of the time, there's a way to do what they need that isn't horrendous, or you can take the opportunity to improve your abstractions. If your problem is that other developers you work with just go in with a hatchet and make a mess without informing anyone, perhaps it's time to adobt some processes to address that, such as better version control hygiene and code reviews.
mmorearty 6y ago
Scons is definitely slow in some scenarios. It has O(m * n) performance if a rule has m inputs and n outputs. I tdiscovered this when working with a code base that had a single rule with all .ts files as inputs, and all .js files as outputs. As the number of files we had got bigger, SCons got painfully slow. Of course we can rewrite our rules, but that shouldn’t be necessary, especially since it was quite hard to figure out why our builds got so slow.
dTal 6y ago
Ironically, hitting the front page of HN with a title of "SCons Is Not Slow" is, for me, excellent circumstantial evidence that SCons is, in fact, slow.
29athrowaway 6y ago
I have not tried SCons, and I have not heard about it in a while. Is it good? who does it compare to CMkae or Bazel? For context, I don't like autotools.
- sgtnoodle 6y ago
  It's good in that it's mature, cross platform, and it's python so you can build whatever you want on top of it. We use it at work for a relatively large codebase that targets multiple languages and architectures. I like python a lot better than cmake's syntax. It's nice to be able to define functions and importable modules to abstract away the ugly details of the build so that other developers can safely use them without being build system experts. Bazel is cool, but it's relatively immature. Last time I looked into it, you couldn't just install it easily. It's easy enough to make scons look like bazel. I suspect bazel handles very large codebases more efficiently.
- ncmncm 6y ago
  There is literally nobody who will admit to liking autotools. But they used to solve many problems we don't have anymore, so we used them. SCons is bad, in that it is slow, slow, slow, slow. If your time is worth nothing, it can in fact be made to do things. But other alternatives can, too, and faster.
ncmncm 6y ago
I don't know if SCons is still slow, but in 2018 it was definitely very, very slow. I doubt it has improved much, as its maintainers were in similar denial then. When we used it at MongoDB, somebody (thanks Mathias!) put together apparatus to make it generate a ninja config instead of building by itself. That was fast! Anyway, after you had the ninja config. But the edit-build-test cycle did not include a slow SCons run anymore, and that was huge. The MongoDB internal wiki had a page titled "SCons Isn't That Slow", with advice for how to live with how slow it really was, later including ("unofficial") ninja and ccache setup. (Which I added.) But at least it wasn't WAF.
- bb88 6y ago
  If forking is the issue, why didn't they just do multiprocessing instead with a bunch of worker processes and work going into queues?
- jhasse 6y ago
  Why is Waf slower? It has separate config / build steps.
  - ncmncm 6y ago
    I don't recall (from time-served at Bloomberg) that Waf was especially slow, but I found that to make it do anything useful, you had to write a lot of obscure, custom Python code nobody could understand.
    - jhasse 6y ago
      And you think this custom Python code wouldn't have been necessary with SCons?
      - ncmncm 6y ago
        Don't really know, tbh. Things I used SCons on were different from the things I used Waf on. SCons might need as much custom Python for those other uses as Waf. But that does not make Waf any better.
mypalmike 6y ago
Maybe add (2014) to the title? It looks like the text may have been revised a few times since, but the original appears to be at least that old.
gfxgirl 6y ago
SCons is the worst experience I've had with a build system. No one, even its advocates, ever really understood what was going on. Things became so much simpler and orders of magnitude faster when we switched away
- stingraycharles 6y ago
  You must not have used cmake. It’s fast but man is it cryptic and has a steep learning curve.
person_of_color 6y ago
Do people use tup?
- papaf 6y ago
  I used to use tup for small projects but then I switched to CMake/ninja. I can't say anything against tup other than it is not widely used. In terms of performance, I did not notice any difference between tup and ninja -- but these were small projects.
- chenglou 6y ago
  Tup requires FUSE, so is a deal breaking dependency for many corporate setups due to security. If you’re into Tup, you might also like https://github.com/droundy/fac https://github.com/droundy/fac Fun fact: ninja was inspired by Tup, but is a bit more geared toward pragmatic usages.
  - papaf 6y ago
    I think tup used to use LD_PRELOAD and seems to still support it: https://github.com/gittup/tup/issues/394#issuecomment-636378846 https://github.com/gittup/tup/issues/394#issuecomment-636378...
person_of_color 6y ago
We need a deep learning based build system.
budabudimir 6y ago
Indeed, I had the misfortune of using SCons on one project back in 2017-2019. It was ridiculous how slow it was. Even for the case when "Nothing to be done" was the result it took 20-40 seconds to output the message. Thankfully we switched to ninja.
onebaddog 6y ago
As the maintainer of SCons I can assure you that we are aware of SCons's performance issue. The page in question is actually quite old at this point. Since the content was written Ninja, Cmake, Bazel, and other build systems have launched and become mature. We are slowly working through the easiest of the list of known performance issues. In addition MongoDB has contributed their Ninja file output logic. We expect to integrate that in the next major release. Projects like MongoDB, Godot, VMware, PlatformIO, and many others use SCons. One of it's strengths is the ability to correctly build complex builds. Please engage with the community our users mailing list, IRC Channel, or discord server if you're a user. One frustration for the project is that users run into issues using SCons and never reach out and ask for help. Many times a small change in usage will resolve whatever has a user stuck.

24 points by moonchild 6y ago

I hope that FOSS can converge to a single build system as much as possible, because it's not very convenient to learn them all.

CameronNemo 6y ago
I don't think a single build system is necessary or desirable. meson (as well as waf and scons) depends on python, which is not so nice for bootstrapping. Thus desktop applications and low level userspace plumbing could benefit from different build systems. That said, I hope waf and scons die off soon (only a handful of packages even use them). CMake and Meson are much better choices.
- bogwog 6y ago
  What’s wrong with waf? I’ve been using it lately and think it’s great. Writing build complex build tasks is easy, and the performance is excellent. I haven’t tried Meson, but CMake cant really do much besides compile the programming languages it explicitly supports. Automation tasks are pretty much impossible.
  - CameronNemo 6y ago
    Do you have a link to these wscripts, or a pastebin? I am curious on what types of automations you are referring to. I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. Only having to know a half dozen build systems is way easier than having to know waf, Scons, meson, autotools, make, setuptools, cargo, qmake, cmake is a pain. Only 25 packages in the distro I contribute to use waf. Most are audio recording tools that do not get much maintenance. 20 packages use scons. Meanwhile almost 400 use meson, ~500 use qmake, ~1000 use cmake, and over 2000 use autotools.
    - bogwog 6y ago
      > Do you have a link to these wscripts, or a pastebin? I am curious on what types of automations you are referring to. Not currently, but I do have a half-written article I plan to publish eventually once I complete this project I'm working on, to showcase my use of waf and how it's helped me. That's how much I like it! But to give you a quick example: my project (a cross-platform game) has a bunch of assets that I am embedding in the executable. To do this, I wrote a custom waf task that automatically generates a char array in a .cpp/.h bundle, and adds it to my build and include paths. Using that same tool, I am able to embed other assets that need to be transformed in various ways before being embedded. One of those is a bundle of binary files and meta-data combined together inside a single flatbuffer data blob. This involves compiling the schema, generating a small command-line utility to write the blob using the sources generated from the schema compiler, using that utility to generate the blob, and finally embedding the blob using the asset embedding tool. (Plus some of the binary files themselves need to be processed before all of this happens) I could do that with shell scripts, but waf's design makes it easy to do, in a small amount of reusable code (everything I just described I implemented in <400 lines of python), and in a general-purpose and easy to read programming language. Plus the tasks automatically execute in parallel wherever possible. > I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. I've heard that complaint before, and I can see why it would be annoying for outside maintainers. Waf's customizability means two Waf projects might look nothing alike. So even if one becomes an expert at Waf (which is not very easy considering how bad the docs are), they might be completely lost in someone else's project. I failed a game jam once because I was using Amazon Lumberyard (which uses waf), and even though I completed the game, I couldn't figure out how to compile the damn thing before the deadline. But if it's any consolation, I don't plan on distributing my project through any linux distros :P
      - Arelius 6y ago
        Having used WAF extensively in large legacy projects, one of it’s large problems is that it isn’t particularly batteries included, and relies on a lot of customizations, and doesn’t have a lot of rails so it’s really easy to break it’s reliability guarantees. And having a unreliable build where one is never sure if a problem will go away by deleting intermediates is super frustrating.
- amelius 6y ago
  > I don't think a single build system is necessary or desirable. Why not? In the old days, it was "./configure; make". Today, it's much less certain what build system(s) you'll have to deal with when building a FOSS tool. That's not a problem when everything works, but as experience shows, more often than not you'll run into issues and then I would say it's nice if you are familiar with the system. And just like with other tools it's nice if everybody uses the same thing. Github wouldn't exist if everybody rolled their own version control system, for example.
  - progval 6y ago
    > Github wouldn't exist if everybody rolled their own version control system, for example. It also wouldn't exist if no one did. It's all about compromises. Having too many tools for the same job isn't good, but monoculture isn't either.
- nonbirithm 6y ago
  If Cargo didn't exist I would believe you. But a group of people learning from past mistakes and doing most of the things right from the start has convinced me otherwise. Sadly when it comes to C++ I don't think we'll get a do-over. Too many build systems exist already, and important libraries from a decade or two ago will use those systems until the end of time, and so will the projects that depend on them indirectly.
dgfitz 6y ago
I’m going to sit in my little corner of the world and wish hg and qmake had won... while learning all of the other ways to generate Makefiles
MCOfficer 6y ago
FOSS build systems are https://xkcd.com/927/ https://xkcd.com/927/ on repeat.
bawolff 6y ago
Unifying around a single thing is not something FOSS likes to do.
- qznc 6y ago
  This is not FOSS specific. In companies you can observe the same tension between the efficiency of unity and the flexibility of diversity.
- mkoubaa 6y ago
  There's an xkcd for that. In seriousness, I wish the same thing, but it's disruptive and thankless work to ask of volunteers
- cycloptic 6y ago
  I don't know, a lot of FOSS seems to have unified around Linux for whatever reason.
  - bawolff 6y ago
    Linux is really 5 million different systems though. Debian or redhat (or etc)? Systemd or init? Bash or zsh? Gnome or KDE? X11 or Wayland? GCC or clang? Almost every single component has multiple choices.
moonchild 6y ago
I've been tossing around thoughts for this recently. I think the solution is something that, like scons, is built on top of a proper programming language. Unlike scons, however, it should be imperative, rather than declarative. Core primitives, like 'a file is out of date iff its modification time or hash has changed', should be expressible 'in userspace', as it were—the core of the build system shouldn't know about them. (Though it should have some concept of a target being up-to-date and not being rebuilt, it shouldn't need to know about files.) This, combined with the ability to create proper abstractions, means you can emulate the primitives of all existing build systems and give them a common substrate. (Bonus: assuming the core primitives are done right, you can still get performance as good as ninja. Which is good, because you won't be able to compile to a build system like ninja.)
- aseipp 6y ago
  The answer to this is "use Shake", which is similar to what you describe. The secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files — so you can depend on environment variables, file contents, results of dynamic commands, etc — you need to be able to express "dynamic" rules where you can compute new dependencies for a rule "on the fly" while it's executing, and you need elbow grease to write it as an API. Shake also has had a lot of research done about it comparing it to other systems, is used very successfully, and has an array of nice features other systems don't, including .ninja file compatibility and build profile reports: https://shakebuild.com/ https://shakebuild.com/ Shake is written in Haskell as a library (so your build rules have to be written in Haskell too, as programs) but you could either wire up a language via the FFI to make some classes of build rules easily expressible in JavaScript/Lua, or in theory rewrite the core of Shake in another language too. The theory is all there, though. My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity (multi language builds, requires parsing or other "advanced" features, etc). As a bonus it's also very fast, within a few % of Ninja when parsing Ninja files, and sometimes faster than that. I was surprised when it came in as 2x as fast as Ninja on a large-ish C codebase at work, due to having picked a different "build schedule" that distributed compilations across cores better.
gravypod 6y ago
A single-build-system world would need to see a build system that recognizes all languages and use cases as equally important. There's very few build systems that actually support this in a reproducible and hermetic way.

qppo 6y ago

I feel like if you close with "only 20% slower than make" you can't justify the title "is not slow." 20% is a pretty huge margin for large builds, which I wouldn't consider using make for in the first place. CMake + Ninja has shaved enormous amounts of time off my builds from make, so why should I use something that's even slower?

Y_Y 6y ago
As a counterpoint, my builds are mostly too fast to care, but modifying the CMake files that describe them is excruciating. It's a miracle of software engineering that CMake functions, but wow is the language awful.
- cycloptic 6y ago
  Is there something specific you dislike about it besides the syntax? I frequently hear complaints about cmake not supporting one thing or another but a lot of those have been addressed in recent releases. It's not clear to me why you say it's a miracle either, AFAIK it does most of the same things any other makefile generator does. (Disclaimer: I'm not involved in the cmake project but I often see people struggling with it because of misuse, possibly because of expectations that it will work exactly like autotools)
- cbsmith 6y ago
  Honestly, I switched to bazel and did not look back.
- umvi 6y ago
  Yeah we switched from Scons to CMake and I have to admit I loved being able to use python for Scons. CMake's language is unintuitive and clunky. I wish CMake had been designed like LLVM with a front and back end so that there could be a python-based front end (for example).
  - bb88 6y ago
    Yeah, I've been unhappy with any automake system for a long time. Haven't tried ninja though. Scons was fine-ish. Now that because work forces me to develop on a locked down windows box, make is the last thing I want.

beervirus 6y ago

Rebuttal here: https://blog.melski.net/2013/12/11/update-scons-is-still-really-slow/ https://blog.melski.net/2013/12/11/update-scons-is-still-rea...

cbsmith 6y ago
I'm just kind of amazed that there is this kind of lack of understanding of performance & measurement amongst people working on optimizing low-level technical tools like build systems.
- ncmncm 6y ago
  To be fair, not all such people; by the evidence, only SCons people.
  - onebaddog 6y ago
    Not even SCons people. (I'm the project maintainer)
  - cbsmith 6y ago
    It was absolutely not intended as a statement about all such people. The surprise is about any such people.

dimator 6y ago

Slow or not, it becomes cryptic to write Sconsfiles with any complexity, and it's impossible to debug.

sgtnoodle 6y ago
It's python, so you can write your own abstractions as you see fit. Like any general purpose programming language, you can make something maintainable or you can make a mess.
- TazeTSchnitzel 6y ago
  Sometimes having a tool prevent you from shooting yourself in the foot is kinder. But it's more than that, a good build system does not need to be Turing complete, and having Turing completeness imposes certain requirements on it that may make it inherently slower.
  - sgtnoodle 6y ago
    Going along with the firearm analogy, it's only fully useful and powerful tool once you've removed it from its case and removed the trigger lock. Any given firearm may have additional safety features such as interlocks, but they're only effective if they stay out of the way when you intend to shoot. You avoid accidentally shooting yourself and others through training, practice and discipline: treat any firearm as if it were loaded, keep your finger off the trigger until you intend to shoot, etc. When setting up a build system for multiple people to use, it's important to agree on a general approach and some rules to follow in order to keep the system from turning into a mess. It's code like anything else, and benefits from training, practice and discipline. Using a language that most folk already know is almost always better than using an obscure one out of a sense that the obscurity makes it safer. When building up a build system, I suggest focusing on designing abstractions to make 90% of use cases trivial for other developers to use. Spitting out libraries and linking them into binaries, existing autocoding scripts, etc. The other 10% of cases that other folk run into, loudly volunteer to make yourself available to help them work through it. Most of the time, there's a way to do what they need that isn't horrendous, or you can take the opportunity to improve your abstractions. If your problem is that other developers you work with just go in with a hatchet and make a mess without informing anyone, perhaps it's time to adobt some processes to address that, such as better version control hygiene and code reviews.

mmorearty 6y ago

Scons is definitely slow in some scenarios. It has O(m * n) performance if a rule has m inputs and n outputs. I tdiscovered this when working with a code base that had a single rule with all .ts files as inputs, and all .js files as outputs. As the number of files we had got bigger, SCons got painfully slow. Of course we can rewrite our rules, but that shouldn’t be necessary, especially since it was quite hard to figure out why our builds got so slow.

dTal 6y ago

Ironically, hitting the front page of HN with a title of "SCons Is Not Slow" is, for me, excellent circumstantial evidence that SCons is, in fact, slow.

29athrowaway 6y ago

I have not tried SCons, and I have not heard about it in a while. Is it good? who does it compare to CMkae or Bazel? For context, I don't like autotools.

sgtnoodle 6y ago
It's good in that it's mature, cross platform, and it's python so you can build whatever you want on top of it. We use it at work for a relatively large codebase that targets multiple languages and architectures. I like python a lot better than cmake's syntax. It's nice to be able to define functions and importable modules to abstract away the ugly details of the build so that other developers can safely use them without being build system experts. Bazel is cool, but it's relatively immature. Last time I looked into it, you couldn't just install it easily. It's easy enough to make scons look like bazel. I suspect bazel handles very large codebases more efficiently.
ncmncm 6y ago
There is literally nobody who will admit to liking autotools. But they used to solve many problems we don't have anymore, so we used them. SCons is bad, in that it is slow, slow, slow, slow. If your time is worth nothing, it can in fact be made to do things. But other alternatives can, too, and faster.

ncmncm 6y ago

I don't know if SCons is still slow, but in 2018 it was definitely very, very slow. I doubt it has improved much, as its maintainers were in similar denial then. When we used it at MongoDB, somebody (thanks Mathias!) put together apparatus to make it generate a ninja config instead of building by itself. That was fast! Anyway, after you had the ninja config. But the edit-build-test cycle did not include a slow SCons run anymore, and that was huge. The MongoDB internal wiki had a page titled "SCons Isn't That Slow", with advice for how to live with how slow it really was, later including ("unofficial") ninja and ccache setup. (Which I added.) But at least it wasn't WAF.

bb88 6y ago
If forking is the issue, why didn't they just do multiprocessing instead with a bunch of worker processes and work going into queues?
jhasse 6y ago
Why is Waf slower? It has separate config / build steps.
- ncmncm 6y ago
  I don't recall (from time-served at Bloomberg) that Waf was especially slow, but I found that to make it do anything useful, you had to write a lot of obscure, custom Python code nobody could understand.
  - jhasse 6y ago
    And you think this custom Python code wouldn't have been necessary with SCons?
    - ncmncm 6y ago
      Don't really know, tbh. Things I used SCons on were different from the things I used Waf on. SCons might need as much custom Python for those other uses as Waf. But that does not make Waf any better.

mypalmike 6y ago

Maybe add (2014) to the title? It looks like the text may have been revised a few times since, but the original appears to be at least that old.

gfxgirl 6y ago

SCons is the worst experience I've had with a build system. No one, even its advocates, ever really understood what was going on. Things became so much simpler and orders of magnitude faster when we switched away

stingraycharles 6y ago
You must not have used cmake. It’s fast but man is it cryptic and has a steep learning curve.

person_of_color 6y ago

Do people use tup?

papaf 6y ago
I used to use tup for small projects but then I switched to CMake/ninja. I can't say anything against tup other than it is not widely used. In terms of performance, I did not notice any difference between tup and ninja -- but these were small projects.
chenglou 6y ago
Tup requires FUSE, so is a deal breaking dependency for many corporate setups due to security. If you’re into Tup, you might also like https://github.com/droundy/fac https://github.com/droundy/fac Fun fact: ninja was inspired by Tup, but is a bit more geared toward pragmatic usages.
- papaf 6y ago
  I think tup used to use LD_PRELOAD and seems to still support it: https://github.com/gittup/tup/issues/394#issuecomment-636378846 https://github.com/gittup/tup/issues/394#issuecomment-636378...

person_of_color 6y ago

We need a deep learning based build system.

budabudimir 6y ago

Indeed, I had the misfortune of using SCons on one project back in 2017-2019. It was ridiculous how slow it was. Even for the case when "Nothing to be done" was the result it took 20-40 seconds to output the message. Thankfully we switched to ninja.

onebaddog 6y ago

As the maintainer of SCons I can assure you that we are aware of SCons's performance issue. The page in question is actually quite old at this point. Since the content was written Ninja, Cmake, Bazel, and other build systems have launched and become mature. We are slowly working through the easiest of the list of known performance issues. In addition MongoDB has contributed their Ninja file output logic. We expect to integrate that in the next major release. Projects like MongoDB, Godot, VMware, PlatformIO, and many others use SCons. One of it's strengths is the ability to correctly build complex builds. Please engage with the community our users mailing list, IRC Channel, or discord server if you're a user. One frustration for the project is that users run into issues using SCons and never reach out and ask for help. Many times a small change in usage will resolve whatever has a user stuck.