4 ms·

I didn't read too deeply, but I'm glad to see all these sorts of systems slowly converge on a sane model. The part that saddens me is that they all end up look

I didn't read too deeply, but I'm glad to see all these sorts of systems slowly converge on a sane model. The part that saddens me is that they all end up looking like a glorified distributed make.

sixo 6y ago
Why does that sadden you? Make doesn't do the job. Therefore a new tool is required. What's the problem?
ptsneves 6y ago
The problem of DAG is solved by make very well. So well that it is why people insist on using it. In theory they would have made a good choice. The issue is that "make" made sure to put enough traps so that people with less knowledge of the pitfalls tend to fall through. How many times I have seen recursive make calls without the $(MAKE) variable lead to funny races or lost compilation flags? How many paralelizable targets running in single thread because somebody did not understand some strange out of order execution(due to wrong Dependency mapping)? How many programs which are only partially built? How many projects were not actually built for huge periods of time because somebody forgot to put PHONY target? How many Frankenstein build systems exist which create Meta languages around make(kbuild) ? My latest adventure was a dormant fork bomb for 4 years in a daily updated project in the clean target! Whole development server farms being resource starved because the clean target started being executed as part of the normal build. How did the fork bomb happen? The usual recursive make. Make is perfect actually,but we are not perfect in its usage.
pacala 6y ago
Make is a 44 year old implementation of the DAG model. It is not the only possible one, and it has plenty of flaws. At least in principle, there is room for a better implementation of the DAG model, and even different implementations tailored for different ecosystems. The Mother of all Demos was amazing 5 decades ago. And yet we're not stuck using the exact same UI toolkit as Douglas Engelbart.
donmcronald 6y ago
I'd hardly call any of them sane. There used to be debates about checking compilers into the VCS and now no one bats an eye when the entire build system is constantly churning underneath you. Today you have GitLab Runner 12.x. Next month it'll be 13.x. I imagine most other CI systems are similar, the hosted variants being especially bad because you can't even decide when to update. Modern CI feels like a shell script using a bunch of external APIs got tossed into a blender and poured into a YAML config. Then everyone claims builds are repeatable because they run in a container, but ignore the fact that if your Dockerfile has a single command that isn't working with exact versions for resources (apt-get anyone?) you can't rely on the build being repeatable between jobs in the same pipeline. A big problem I think we'll face in the future is there's no value in giving developers a product they can control. GitLab CI will evolve so you need a top tier plan to do anything interesting and they'll eventually own your build environment. GitHub will push things like Codespaces and they'll own your development environment. The idea that anyone would even touch GitHub Actions blows my mind. What happened to the developers that want to control their environments? Do you actually own your codebase if it's useless without a bunch of paid subscriptions you're using as part of your development workflow? What happens when your CI provider triples the price or shuts down entirely?

I didn't read too deeply, but I'm glad to see all these sorts of systems slowly converge on a sane model. The part that saddens me is that they all end up looking like a glorified distributed make.

sixo 6y ago

Why does that sadden you? Make doesn't do the job. Therefore a new tool is required. What's the problem?

ptsneves 6y ago

The problem of DAG is solved by make very well. So well that it is why people insist on using it. In theory they would have made a good choice. The issue is that "make" made sure to put enough traps so that people with less knowledge of the pitfalls tend to fall through. How many times I have seen recursive make calls without the $(MAKE) variable lead to funny races or lost compilation flags? How many paralelizable targets running in single thread because somebody did not understand some strange out of order execution(due to wrong Dependency mapping)? How many programs which are only partially built? How many projects were not actually built for huge periods of time because somebody forgot to put PHONY target? How many Frankenstein build systems exist which create Meta languages around make(kbuild) ? My latest adventure was a dormant fork bomb for 4 years in a daily updated project in the clean target! Whole development server farms being resource starved because the clean target started being executed as part of the normal build. How did the fork bomb happen? The usual recursive make. Make is perfect actually,but we are not perfect in its usage.

pacala 6y ago

Make is a 44 year old implementation of the DAG model. It is not the only possible one, and it has plenty of flaws. At least in principle, there is room for a better implementation of the DAG model, and even different implementations tailored for different ecosystems. The Mother of all Demos was amazing 5 decades ago. And yet we're not stuck using the exact same UI toolkit as Douglas Engelbart.

donmcronald 6y ago

I'd hardly call any of them sane. There used to be debates about checking compilers into the VCS and now no one bats an eye when the entire build system is constantly churning underneath you. Today you have GitLab Runner 12.x. Next month it'll be 13.x. I imagine most other CI systems are similar, the hosted variants being especially bad because you can't even decide when to update. Modern CI feels like a shell script using a bunch of external APIs got tossed into a blender and poured into a YAML config. Then everyone claims builds are repeatable because they run in a container, but ignore the fact that if your Dockerfile has a single command that isn't working with exact versions for resources (apt-get anyone?) you can't rely on the build being repeatable between jobs in the same pipeline. A big problem I think we'll face in the future is there's no value in giving developers a product they can control. GitLab CI will evolve so you need a top tier plan to do anything interesting and they'll eventually own your build environment. GitHub will push things like Codespaces and they'll own your development environment. The idea that anyone would even touch GitHub Actions blows my mind. What happened to the developers that want to control their environments? Do you actually own your codebase if it's useless without a bunch of paid subscriptions you're using as part of your development workflow? What happens when your CI provider triples the price or shuts down entirely?