4 ms·
I used GNU make to manage a data pipeline with a large number of (mostly Python) scripts written by a team. It had to process lots of data using many steps. Ma
by dwheeler 3y ago
I used GNU make to manage a data pipeline with a large number of (mostly Python) scripts written by a team. It had to process lots of data using many steps.
Make worked very well for this use case. It easily auto-generated dependencies, ran tests, and supported parallel execution. In a tiny fraction of a second it determined what needed to be done, and only did that.
We created a small collection that we released as open source software. You can get it here:
https://github.com/david-a-wheeler/make-booster https://github.com/david-a-wheeler/make-booster
I think a lot of hate on make is due to poor use. If your makefile is complex, refactor it. Auto-generate dependencies (it only takes a few lines in GNU make). And don't use recursive make, that way lies madness. I also think GNU make is the wiser tool; POSIX make lacks too much in many cases.
- WesolyKubeczek 3y ago> Auto-generate dependencies (it only takes a few lines in GNU make) Care to give an example? The manual is quite frustrating to piece this together, when it comes to dynamic dependencies, and god forbid, dynamic targets.
- dwheeler 3y agoCheck out: https://spin.atomicobject.com/2016/08/26/makefile-c-projects/ https://spin.atomicobject.com/2016/08/26/makefile-c-projects... https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ https://make.mad-scientist.net/papers/advanced-auto-dependen...