4 ms·
Why does Mill use Scala?
- koito17 2y agoThe article fails to mention whether Scala can ensure code is deterministic and hermetic. Starlark code is deterministic and hermetic, but the article never mentions this. Unfortunately, Starlark does not have static typing, but I think types would make Starlark one of the best languages today for build configuration. In the Clojure community, there was a huge push for "builds are programs". I somewhat agree with this assertion, but I also think "one should restrict the class of programs a build belongs to". Neither Clojure nor Scala, compared to Starlark, seem to offer a way to ensure builds belong to a deterministic subset of programs. Thus I am still wondering "why Scala?". I have never used Scala, but reading this whole article gives me the impression that Mill is the Scala equivalent of Clojure's tools.build. That is not what I would want in a build system.
- fmbb 2y agoWhat is special in starlark that makes it hermetic and deterministic? Scala is deterministic. If you call functions that have side effects and nondeterministic behavior you can fall outside the comforts of determinism. But you can stumble upon library functions someone wrote in Starlark that accidentally put you there as well. The Starlark homepage says > Hermetic execution - Execution cannot access the file system, network, system clock. It is safe to execute untrusted code. But the last time I wrote Starlark it was to define build targets in Basel. And executing the builds definitely accessed my file system and the network, otherwise builds would have no results.
- cbeach 2y agoBazel enforces a hermetic sandbox for Starlark to operate within. All input files have to be declared, and the build process can only see files that are declared. Starlark cannot access arbitrary files at runtime, and it deliberately has no APIs for things like system time or random number generation, or global state. Scala, on the other hand, has no such restrictions. As much as I love Scala, I think it’s an odd choice for a pure, deterministic system. Although perhaps if you use Scala to build a DSL (an area where Scala shines) you could engineer a pure functional sandbox within Scala.
- fmbb 2y agoDo you only declare the file names, or also file content and owner and group and everything up front? Sure it deliberately has a bunch of restrictions in its base form. But in order to use it for anything you have to write custom actions or download and execute custom actions others wrote. And this will mutate your file system and run arbitrary executables.
- winwang 2y agoAs someone who worked a bit with Bazel actions and uses Scala: I think Bazel cuts down on the area where arbitrary I/O happens. Also, many executables come from pre-defined targets (e.g. java toolchains). One way to deal with I/O is if Mill injected its own implementation of a JVM filesystem/network layer, which I'm pretty sure it can do.
- recursivecaveat 2y agoInput and output files are tracked determistically via content hashing. It cannot reach outside of its sandbox and touch the network or actual filesystem. Those are all (generally pinned) inputs to the system. For a given version of a script and a tuple of input content hashes, you get the exact same output files every time. No way to accidentally iterate over a hashmap, embed a timestamp, or leave the filesystem in a weird state because you were interrupted.
- michaelmior 2y ago> Scala is deterministic. I don't see how Scala is any more deterministic than any other language.
- fmbb 2y agoThat’s what I’m saying. About Starlark.
- fire_lake 2y agoStarlark does not expose sources of randomness like date.now()
- Capricorn2481 2y agoIf you're deliberately using that to prove Scala doesn't work for config than sure
- fire_lake 2y agoIt’s about guarantees vs eternal vigilance.
- thirtyseven 2y agoBazel splits the build into multiple phases. Starlark only comes into play in the first two, load and analysis. During these phases, Starlark code doesn't have access to the filesystem, except in a few very limited cases like using the glob() function to expand a wildcard to a list of source files. Furthermore, it only generates an abstract graph of build actions. The Bazel engine is responsible for executing this graph in later stages, which might result in non-hermetic things happening but usually not. Starlark has intentionally limited functionality such as lacking Turing completeness or global variables. This provides guarantees that it can be executed in parallel and will have a finite runtime.
- toprerules 2y agoThe best config system I've ever seen used plain old Python to generate static configs. Everyone knows Python. Python is easy to do data munging in, as demonstrated by it's popularity as the #1 data science tool. There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause. Even Starlark is just a dialect of Python. You can spend decades building a complicated configuration language, use a bespoke functional language as Mill does, but if you're a single company that can enforce code quality and just wants to get the job done, I feel like everything else is just unnecessary and over-engineered to scratch some academic itch for a "better system" that enforces "purity" at the cost of velocity. I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one.
- MathMonkeyMan 2y agoThis does work well. A team I was on at a past job did exactly this. On Unix the service literally ran `std::system("python config.py >config.json")` on startup. The problem with this is that the answer to the question "what kind of configuration can I expect?" is "simulate the script and find out." If the script is written well, and is short, then the parameters that are filled in by the runtime environment are apparent. Over time, though, there is a risk that the script will not remain written well, and it almost certainly won't remain short.
- siriusfeynman 2y agoAn approach I use is splitting my config tools into 2 stages Stage 1 creates a "explicit" config that can be exported to plaintext that contains exactly what is going to be created/modified with no abstraction/simplification Stage 2 applies the "explicit" config You get to be as clever as you want in stage 1 to avoid excessive copy pasting or not being able to know what your tool is going to do because all you have to go on is some homegrown DSL
- iforgot22 2y agoYou run into the same problem with config DSLs, except now you're dealing with a DSL. Config is almost never going to be static.
- agentultra 2y agoI'm not big on programming languages being used for configuration. It adds a lot of complexity and maintenance burden. Configurations are often read and your reader isn't a compiler but now they have to imagine what the final configuration state will be after evaluating the "program" that generates the configuration. I think plain old configuration languages are better, even if verbose, since the usual text-based tooling works quite well for managing, searching, etc. I use nix a lot and the main thing that bothers me about it is the language. People are quick to get clever with it. It becomes a morass of code that is difficult to read for anyone but experts and when it breaks and your not that expert... good luck fixing it.
- winwang 2y agoUnfortunately, I find that real-world configs don't quite conform to easy understandability past tutorial examples, i.e. k8s and yaml.
- vander_elst 2y agoIt seems this is mainly a scala project and then they are using scala also for the configuration, it probably makes sense for them.
- kunley 2y ago"Mill is a fast, scalable, multi-language build tool that supports Java, Scala, Kotlin, and Python". So, while I understand this tool can resonate in the JVM world, I have no idea why one would want to pull Java into their toolset in order to build Python.
- wavemode 2y agoWhat does it even mean to "build" Python?
- tacticus 2y agoshouting at pip inconsistencies
- pletnes 2y agoI imagine creating a wheel file with your code, metadata and resources? Also recall that a lot of python packages have dependencies written in C, Fortran, Rust etc. Also, many tools exist to create executable programs - basically bundling the python interpreter with some .py files, etc.
- kunley 2y agoYeah had the same thought actually
- lamp_book 2y agoI don’t know who is using mill but I’d guess Scala people and Scala frequently implies Spark which in turn implies PySpark. For instance I could see it being useful for a team building a source connector which will be Scala-centric but need some PySpark additions as well. Even without Spark, Scala is frequently used for data centric applications and those kinds of teams would naturally be working with python as well.
- Kwpolska 2y agoBecause it’s a Scala project, written by a Scala fan, simple as that. No need to come up with extra justification.
- gdgghhhhh 2y agoThe title really confused me until I realized this has nothing to do with https://millcomputing.com/ https://millcomputing.com/ :-)
- pabs3 2y agoWonder when Mill Computing will become more publicly active and have hardware available.
- waste_monk 2y agoIt seemed quite promising but from the outside momentum appears to be almost completely stalled, with only a handful of posts per year on the forum. I'd be curious to know if there's progress being made behind the scenes.
- pabs3 2y agoFrom forum threads in the last year or two it sounds like they have gotten quite far and only need investment to progress further. https://millcomputing.com/topic/any-plans-for-2024/ https://millcomputing.com/topic/any-plans-for-2024/ https://millcomputing.com/topic/yearly-ping-and-see-how-things-are-going-thread/ https://millcomputing.com/topic/yearly-ping-and-see-how-thin... Edit: posted a HN thread about getting investors for Mill: https://news.ycombinator.com/item?id=43054697 https://news.ycombinator.com/item?id=43054697
- rpcope1 2y agoOne interesting thing is that there are comparisons to Maven and Gradle but not sbt. Do people just not use sbt anymore or is it omitted because it's also Scala and/or prone to becoming a mess?
- videogreg93 2y agoI used sbt in my previous job and didn't hate it. All I want from build systems is to get out of my way and it let us do that pretty well. Very simple to add new tasks as well.
- openplatypus 2y agoWe use it. It is actively developed. It doesn't get in the way. It does what it says on the tin.
- ATMLOTTOBEER 2y agoIt’s slow, and the task/setting key resolution makes no sense unless you waste several hours reading sbt docs (time I will never get back). I’m not saying dump sbt for mill but u gotta admit sbt kinda sucks. Afaik this is also the conclusion most larger orgs that use scala come to when they start calling scalac from bazel or similar.
- ezst 2y agohttps://mill-build.org/mill/comparisons/sbt.html https://mill-build.org/mill/comparisons/sbt.html
- Lyngbakr 2y agoI'm surprised that Lua wasn't included in the discussion. I'm not saying they necessarily should have chosen to use it, but it's a notable omission given its popularity as a config language.
- rubenvanwyk 2y agoA lot of these reasons also apply to Kotlin and Kotlin is arguably simpler than Scala, why not just use Kotlin? It is much more widely adopted than Scala.
- ncgl 2y agoThe first thing any self respecting python dev is going to do on a new repo is implement typechecking. And the second thing they're going to do is complain about pip/pyenv. These days I would trade my python experience for scala, even knowing it'd mean less job prospects. We make a lot of excuses for python.