14 ms·
Makefiles. I always dismissed them as a C compiler thing. Something that could never be useful for Python programming. But nowadays every project I create has a
by aequitas 7y ago
Makefiles. I always dismissed them as a C compiler thing. Something that could never be useful for Python programming. But nowadays every project I create has a Makefile to bind together all task involved on that project. From bootstrapping the dev environment, running checks/test, starting a devserver, building releases and container images. Makefiles are just such a nice place to put scripts for these common tasks compared to normal shell scripts. The functional approach and dependency resolving of Make allows you to express them with minimal boilerplate and you get tab completion for free.
Sometimes I try to take more native solutions (eg. Tox, docker) but I always end up wrapping those tools in a Makefile somewhere forthe road since there are always missing links and because Make is ubiquitous on nearly every Linux and macOS it is just all you need to get a project started.
Example: https://gitlab.com/internet-cleanup-foundation/web-security-map/blob/master/Makefile https://gitlab.com/internet-cleanup-foundation/web-security-...
Running the 'make' command in this repo will setup all requirements and then run code checks and tests. Running it again will skip the setup part unless one of the dependency files has changed (or the setup env is removed).
- pletnes 7y agoOne of the worst problems with using windows (in my opinion) is that there’s no native GNU make.
- aequitas 7y agoIsn't non-native development on Windows a solved problem nowadays with WSL(2)?
- murderfs 7y agoWSL is currently horrendously (unusably, IMO) slow. WSL2 promises a 20x speed up, but it was already 100x slower than native Linux at some actually-realistic workloads that happen all the time when you're developing (e.g. `git grep`), so it's probably still too slow to be tolerable. I had the opposite problem of wanting to develop some stuff for Windows from a Linux environment, and I settled on running a linux VM and copying binaries over by scping to WSL, which works reasonably well.
- pletnes 7y agoA nice thing with WSL is that you get working make and rsync. But I would like make for coding on native windows. Many FOSS projects use Makefile as the parent post described.
- shadowfox 7y agoThere are a number of ports of GNU Make to Windows. MSYS2 [1], for examples, provides a reasonable development environment that includes Make. If you just want a Make, there is [2] which can be installed separately and is part of the GNUWin32 collection. [1] https://www.msys2.org/ https://www.msys2.org/ [2] http://gnuwin32.sourceforge.net/packages/make.htm http://gnuwin32.sourceforge.net/packages/make.htm
- deng 7y ago> One of the worst problems with using windows (in my opinion) is that there’s no native GNU make. GNU Make even comes with a vcproj file for building a native binary with Visual Studio. Worked fine for me. Building it with Guile support though is difficult, but fortunately Eli Zaretskii provides native binaries through his ezwinports, and they worked pretty much flawlessly for me. Of course you will usually need a shell to execute recipes, but Make itself runs natively. For more information, see README.W32 in the sources.
- vips7L 7y agoScoop has it in their repos as well (the gow package).
- justincormack 7y agoWindows does ship nmake but it is a little different.
- d0mine 7y agofabfile.py (Fabric) could be used as a Makefile in Python. If you don't ever need to ssh to other machines ti run your tasks, you could use pyinvoke library directly (tasks.py). https://www.fabfile.org/ https://www.fabfile.org/ It is easy to add command line arguments to the tasks, configure them using files (json, yaml), environment variables, to split the task definitions into several modules/namespaces.
- deleted 7y ago[deleted]
- bb88 7y agoHaving used fabric in the past, I've always found it just as easy to use a shell script and make files. There's always some level of bootstrapping a project (installing packages/software, compiling libraries and dependencies) where it's easier to just to write a shell script than to program python to do. E.g. How do you get fabric installed on a system? There's also been this longevity of sorts that Make seems to have gotten right. People just keep going back to it because it's simple.
- theossuary 7y agoI've been moving away from using shell scripts in a tools/ directory to using Python Invoke (http://www.pyinvoke.org http://www.pyinvoke.org), which is the library underlying fabric. I used bash scripts for years, but for a lot of reasons made the switch: - It was always painful to create small libraries of functions used across multiple scripts in a project - It's difficult to consistently configure them with per-user settings. I've written bash implementations of settings management, Invoke handles this for me. - I'd still have to reach for Python whenever I needed to do anything with arrays or dicts, etc. - Getting error handling correct can be a chore Invoke has a lot of nice to haves to: - Autogenerated help output - Autocomplete out of the box - Very easy to add tasks, just a Python function - Easy to run shell code when needed - Very powerful config management when needed - Supports namespacing, task dependencies, running multiple tasks at once and deduplicating them It's not perfect, but it's a lot better than my hand rolled scripts were.
- shavenwarthog2 7y agoIn 2019 Makefiles are a useful tool for automating project-level things. Too often webapps will require you to install X to install Y to run producing artifact Z. Since Make is old and baked and everywhere, specifying "make Z" is a useful project-level development process. It's not tied to a language (e.g. Tox) nor a huge runtime (Docker). Make is small enough in scope to be easy, and large enough to be capable without a lot of incantations. The big downside of Make, alas, is Windows compatibility.
- logicallee 7y ago>The big downside of Make, alas, is Windows compatibility. Isn't the big problem that you have no idea what it's doing to your system? Also that you aren't expected to be able to undo it. You can read the makefiles, of course, but it seems simpler not to have to. (Just update the necessary packages yourself, to the latest version.) Forgive me if this is naive of me.
- bch 7y agoIf you write your own Makefiles you know what they do. They’re not that hard to grok and even hand-rolled Makefile use is (IMO) underrated.
- dnautics 7y agoI always had trouble reading makefiles because the control flow is not very linear. At least with shell files basically everything is explicit.
- sk5t 7y agoThe dependency graph is one of the better reasons to use makefiles--think of the nonlinearity as a bonus!
- aequitas 7y agoI often find the non linear way of working with Make an advantage. Since it allows you to break a big piece of procedural shell code with lots of control flow (if X is installed don't install again, etc) into small self contained functional pieces with clean input and output boundaries which can be run individually. It also greatly improves code reuse as every target/recipe can be considered a function.
- trendoid 7y agoAny good resources for learning about make files that you can recommend?
- gilmi 7y agoMatt Might's article is really good: http://matt.might.net/articles/intro-to-make/ http://matt.might.net/articles/intro-to-make/
- messe 7y agoWorth noting that that's an introduction to GNU make, which, while the most common implementation, isn't the only one out there.
- antipaul 7y agohttps://github.com/pavopax/gists/blob/master/makefile-quick-start.md https://github.com/pavopax/gists/blob/master/makefile-quick-...
- ashton314 7y agoThis video helped me: https://www.youtube.com/watch?v=fkEz_oVh0B4 https://www.youtube.com/watch?v=fkEz_oVh0B4
- aequitas 7y agoThis is a nice video. The only thing I'm missing that should be covered imho (as you will encounter it even if you don't use it) is implicit/pattern rules: https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html#Pattern-Rules https://www.gnu.org/software/make/manual/html_node/Pattern-R...
- deng 7y agoThe GNU Make manual is excellent. For learning advanced techniques: "The GNU Make Book" by John Graham-Cumming.
- aequitas 7y ago
- RangerScience 7y agoWe're doing this, and I mostly love it. I haven't found a great way to do code re-use across projects yet, and I'm not super happy with the Make function syntax (but, maybe if it needs a function, I should turn it into a shell script that itself is called by the Make command...). All in all tho, it's a fantastic place to write down long CLI commands (ex: launching a dev docker container with the right networking and volume configurations) that you use a lot when working on the project. Our Jenkins pipeline also relies on the Makefiles, literally just invoking `make release`, which is also pretty awesome.
- aequitas 7y agoWhen using it in multiple projects and CI you also tend to develop some kind of Developer-API with common commands/targets. No matter what kind of project you run you always use the same target names to get started. No remembering which tool is used for this lanuage, just clone it, run `make` and you're off, `make test` to test, etc. Make does support includes (https://www.gnu.org/software/make/manual/html_node/Include.html https://www.gnu.org/software/make/manual/html_node/Include.h...) which allow for some form of code reuse across projects. But then you encounter the balance between DRY and clarity. There are always exceptions, so you try to make stuff to universal, but then its hard to grok the code. And I feel that if I start to use functions I'm using Make wrong and that kind of logic better fits in shell scripts called from the Makefile. Makefiles (the way I use them at least) should be simple to read and explain themselves. But it's often hard to balance this with the features Make provides, like implicit rules and automatic variables. And if I ever turn to generating Makefiles (other than for C projects where it kind of expected) I will probably retire.
- RangerScience 7y ago> common commands Oh absolutely. It's fantastic for that. Our build pipeline actually relies on that; every project has a "release" target that is basically for the CI to use. > Make includes Yeah, I looked into that, and I think I had the same conclusion. > scripts called from the Makefile That's what I'm thinking is the way to level up this kind of system. Although then, why have `make init` instead of just `./bin/init` ?
- noobermin 7y agoMakefiles are so old and quaint, why not use "{flavorofthemonth}".format(flavorofthemonth=np.random.choice(frameworks)) ?
- m463 7y agoI hate makefiles. That said, I wholeheartedly agree with the comment. It's sad that something so central to a project and so useful and important to so many people seems like it hasn't advanced ... ever. Developers generally do the minimum with Makefiles and get out. They are similar to 1040 forms in popularity. I've always had a dream of a redesigned "make" system... with import statements, object oriented rules, clear targets, rules and files clearly seperated, structure and organization... sigh.
- alexhutcheson 7y agoBazel is pretty nice.
- ptasci67 7y agoI will +1 that. I like Bazel a lot because it really forces you into a singular way of building a project which is clean and nice. That said, mileage may vary. It was originally built by Google so it has quirks. I find it is best suited for projects with compiled dependencies and large repos. Otherwise, I was going to add that Gradle as a build system is very advanced and improves upon make in many ways.
- dnautics 7y agoBazel is great until you have to install tensorflow from source into a container and are sitting wondering why you have to put and configure a JVM inside a container-destined-to-be-static-binary, temporarily to get a python program with no Java bindings installed. As I'm not the most intelligent developer, I'm sure there's a better, more sophisticated way to do this but I got really frustrated and gave up.
- imtringued 7y agoWhy not use multiple stages in your docker file?
- dwrodri 7y agoI'm sure bazel is a good tool when it is used properly, but as a greenhorn in the tech field, the constant version mismatching between bazel and tensorflow can become quite the pain when you have to build tensorflow from source.
- quickthrower2 7y agoI develop on Windows, and I like make as a lazy default you can just type in and as long as you maintain the make file, it will build the thing. It is also a nice document of what you can build and how. I also like it because Netlify supports it, so you can get it to run make to deploy your site when you push a commit, giving you a lot of control about your CI, while keeping it simple.
- ddebernardy 7y agoHave you ever considered using Rakefiles instead? https://github.com/ruby/rake https://github.com/ruby/rake
- aequitas 7y agoNever wrote them myself but have encountered them sometimes. Had no major issues with them then I believe. However I would probably write a Makefile to manage the Ruby environment and install Rake as they don't come installed by default.
- uijl 7y agoI really like this tutorial to get into Make files: https://swcarpentry.github.io/make-novice/ https://swcarpentry.github.io/make-novice/.
- kitd 7y agoI put Make in the same class as Vi. I hate using them but I have to learn them because they're the least of N evils, the most pragmatic way out of a hole.
- novaleaf 7y agoself taught dev here too. I have never used make files, but pretty sure I'm using NodeJs in a similar role. I use it to automate all my "scripting", including deploying of my SaaS product to the cloud and running unit tests. If it sounds interesting, check out https://www.npmjs.com/package/shelljs https://www.npmjs.com/package/shelljs PS: I do my primary development on windows, but my production environment is ubuntu. node apps "just work" on both environments. truely cross platform.
- escrichov 7y agoI also use similar Makefiles in my projects. I use "make release" to generate the docker container.
- lowlevel 7y agoI second this... a lot of times, broken make files are standing between you and victory, so it would be good to at least have some familiarity with them.
- sanderjd 7y agoI love Make in concept and kind of hate it in practice. There is sooo much incidental complexity and so many warts to work around. I think it's a concept that is ripe for a new approach that thoughtfully keeps the good, ditches the bad, and maybe even adds some useful capabilities that aren't already there. But of course I'm immediately skeptical of this idea a la https://xkcd.com/927/ https://xkcd.com/927/ (Standards). For instance, maybe this is what npm and all the rest thought they were doing. Certainly Rake in the ruby world tried to do this, and I never really liked it, so clearly they missed the mark somehow, at least for me. But then when I feel discouraged about the ability to improve on things, I think about how I felt this way when I first heard about Git. Why would you implement a new source control system when we already have subversion? Sure, svn has its frustrations and warts, but this new thing is just gonna have its own frustrations and warts and now we'll just have another frustrating warty thing and we haven't really gained anything. And this is totally true! Git is super frustrating and warty. Except that it's also way better than subversion, much faster and far more flexible. It was a revelation when I started using it. So I think back to Linus when he was thinking about creating git and think that he probably didn't have this discouraged uncertainty about improving things; he just had ideas for a better way and he went out and did it. (And yes, I know it was influenced by bitkeeper and other DVCs exist, so it's not like he invented the concept, but my point stands.) So maybe someone could make a better Make?
- majkinetor 7y agoOn Windows there is great Powershell module Invoke-Build.