4 ms·

This appears to be the same language that started as a custom build rule DSL for bazel (and evolving from lessons learned in blaze's Python extension), the page

This appears to be the same language that started as a custom build rule DSL for bazel (and evolving from lessons learned in blaze's Python extension), the page positions it as a much more general-purpose embedded language. I'm all for using the right tool for the job, and I'll acknowledge the benefits of hermeticity and parallelism in many contexts. But, I also think that you need to weigh the cost of adding another language to a project. I don't imagine myself using this anywhere outside of a build system, and even then I would first try to take the declarative rules of that build system as far as I can. It's probably better than shell scripts, though, at least most of the time? May depend on your team's prior familiarity with shell and/or love of Python. There are three implementations (Go, Rust, Java) which could be a good thing if the language doesn't change often. They would have to be kept in sync for any changes, as well as keep from drifting due to changes in the host language's semantics. I also think that the closeness in syntax with Python can be a disadvantage, especially if users are expecting more recent Python additions like the walrus operator or pattern matching to be available. I like to design languages, too, so these comments come from a place of love and understanding. I think it would help if there were some more specific justification (or examples) of why this is better than something more established (e.g. Lua or MicroPython) or more distinct (e.g. pure data in configs instead of embedded code). I do like that the language attempts to remain as simple as needed.

foooorsyth 2y ago
>There are three implementations (Go, Rust, Java) There are so many people at Google just goofing around, lol. People just doing hobby stuff for $350k/year. There is no justification to implement this custom dialect of Python for a build system that drives everyone crazy 3 times. Reminds me of when it was revealed that 400 people were working on Fuschia — a hobby OS that only shipped on a single smart home device.
- deleted 2y ago
  [deleted]
- ithkuil 2y ago
  IIRC initially blaze was literally using python as a config layer. That made it too easy to write build scripts that were too slow and too hard to optimize and that negatively affected the overall build experience. An increase in edit/build/run cycle efficiency of thousands of employees justifies an investment of a few engineers. They could have invented a new bespoke DSL. Instead they choose to stick to a well known and familiar language and just limit its expressiveness to a subset that would be easier to optimize. I think that's quite reasonable
  - kevindamm 2y ago
    I also remember blaze directly eval'ing Python pre-Skylark, but I think the bigger problem was build hermeticity. This was compounded by env-related issues making it hard to build Python hermetically in the first place, and the importance of cached object files and build outputs in managing the incredibly large monorepo that is Google's source tree. And that's all before considering the Py2->Py3 migration (which, fortunately, was tackled afterwards). I think a new bespoke DSL would have been a non-starter since so many build scripts had already been written by the time Skylark was being conceived.
- zellyn 2y ago
  One of my hobbies while at Google (2010-2015) was to watch the multiple failed attempts to get rid of Borg Config Language and actual Python in Blaze. It took a lot of work until they eventually succeeded. You're probably underestimating the rigor, the pain, and also the value involved in cleaning those things up: being able to cleanly operate programmatically at scale in the Google monorepo is extremely necessary. (This is also why I don't trust configuration languages built by people who _didn't_ observe the years of pain. Cue and jsonnet are notable projects that were able to incorporate a lot of lessons.)
  - zellyn 2y ago
    Also, I think the Rust version was built as a side-project, then handed off to Facebook. With regards to Fuchsia… I used to think building a new OS from the ground up was madness. Now I think _not_ doing that eventually is madness. I'd be a lot happier if Zircon and Fuchsia were done in Rust though…
    - surajrmal 2y ago
      A good deal of fuchsia is done in rust (roughly 50% and increasing over time). People over emphasize the importance of zircon needing to be in rust. It's more important for new code to be rust than for existing code to be rewritten. Zircon isn't a very fast growing part of the entire OS.
      - zellyn 2y ago
        Fair. That study on writing only new things in Rust was surprising! https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html https://security.googleblog.com/2024/09/eliminating-memory-s... I do think for an OS/Kernel, it's worth having everything in a memory-safe language, and possibly worth formal verification too, if the very core of it is small enough…
  - foooorsyth 2y ago
    I can appreciate the pain of actually getting things through in a large (100k+ people) organization. What do you consider the be the justification for three separate implementations of the same build config language? Genuine question. I am not doubting the need for the DSL itself.
- surajrmal 2y ago
  I'm not sure. I understand what you are trying to say - do you think we should give up on OS diversity? Are you making fun of a company for actually investing in something ambitious which can benefit many if successful? While I'm not sure you are in the same crowd, I always think it's interesting that the HN hivemind tends to be upset with the browser monoculture, but doesn't bat an eye at OS monoculture. It feels like you're really just channeling feelings about the company rather than the projects.
- surajrmal 2y ago
  Of the three, only two are Google produced. The rust implementation is written by Facebook for use in their build system. The java implementation is the original but is pretty tied to the the bazel build system and not really very suitable to other uses. The go implementation is meant to be embeddable into a varying number of applications. An example application is https://github.com/shac-project/shac https://github.com/shac-project/shac Within Google, there a large number of similar tools which are written in go and harness the starlark language. While there are plenty of other options, I will say I think starlark is often a great choice.
- deleted 2y ago
  [deleted]
yodsanklai 2y ago
If I understand correctly, Starlark is a strict subset of Python (at least very close to Python). Can anything be more established than Python? Also I don't think pure data is an option. The point is that you want to generate data, which has many benefits (avoid duplication, easier to test). It seems that Starlark is a good trade-off given the constraints. My main grief is that it doesn't have type.
- kevindamm 2y ago
  The syntax is a pure subset but the implementations are bespoke, so I wouldn't equate the established position of Python with whether Starlark is considered established. I agree that sometimes pure data isn't an option, and I've had to write some Skylark to assist blaze build rules too, but every time it also added tech debt, reduced the number of people on the team who completely understood the build system, and wasn't convenient/practical to test the build extension. The problem I'm referring to above is when the use of a source-controlled data file would have been sufficient but someone had to write a build extension because it's fun or something new to do (or whatever reason seemed convincing at the time). Then there are people who think that those same data files shouldn't be part of the build process at all and should be part of system turn-up/tear-down, stored in a global config or DB where versioning is alongside the data instead of alongside the program build. Certain kinds of migration are made more difficult if everything is built into the binary. Of course, that strategy comes with its own caveats.
- phyrex 2y ago
  The rust implementation has types

foooorsyth 2y ago

>There are three implementations (Go, Rust, Java) There are so many people at Google just goofing around, lol. People just doing hobby stuff for $350k/year. There is no justification to implement this custom dialect of Python for a build system that drives everyone crazy 3 times. Reminds me of when it was revealed that 400 people were working on Fuschia — a hobby OS that only shipped on a single smart home device.

deleted 2y ago
[deleted]
ithkuil 2y ago
IIRC initially blaze was literally using python as a config layer. That made it too easy to write build scripts that were too slow and too hard to optimize and that negatively affected the overall build experience. An increase in edit/build/run cycle efficiency of thousands of employees justifies an investment of a few engineers. They could have invented a new bespoke DSL. Instead they choose to stick to a well known and familiar language and just limit its expressiveness to a subset that would be easier to optimize. I think that's quite reasonable
- kevindamm 2y ago
  I also remember blaze directly eval'ing Python pre-Skylark, but I think the bigger problem was build hermeticity. This was compounded by env-related issues making it hard to build Python hermetically in the first place, and the importance of cached object files and build outputs in managing the incredibly large monorepo that is Google's source tree. And that's all before considering the Py2->Py3 migration (which, fortunately, was tackled afterwards). I think a new bespoke DSL would have been a non-starter since so many build scripts had already been written by the time Skylark was being conceived.
zellyn 2y ago
One of my hobbies while at Google (2010-2015) was to watch the multiple failed attempts to get rid of Borg Config Language and actual Python in Blaze. It took a lot of work until they eventually succeeded. You're probably underestimating the rigor, the pain, and also the value involved in cleaning those things up: being able to cleanly operate programmatically at scale in the Google monorepo is extremely necessary. (This is also why I don't trust configuration languages built by people who _didn't_ observe the years of pain. Cue and jsonnet are notable projects that were able to incorporate a lot of lessons.)
- zellyn 2y ago
  Also, I think the Rust version was built as a side-project, then handed off to Facebook. With regards to Fuchsia… I used to think building a new OS from the ground up was madness. Now I think _not_ doing that eventually is madness. I'd be a lot happier if Zircon and Fuchsia were done in Rust though…
  - surajrmal 2y ago
    A good deal of fuchsia is done in rust (roughly 50% and increasing over time). People over emphasize the importance of zircon needing to be in rust. It's more important for new code to be rust than for existing code to be rewritten. Zircon isn't a very fast growing part of the entire OS.
    - zellyn 2y ago
      Fair. That study on writing only new things in Rust was surprising! https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html https://security.googleblog.com/2024/09/eliminating-memory-s... I do think for an OS/Kernel, it's worth having everything in a memory-safe language, and possibly worth formal verification too, if the very core of it is small enough…
- foooorsyth 2y ago
  I can appreciate the pain of actually getting things through in a large (100k+ people) organization. What do you consider the be the justification for three separate implementations of the same build config language? Genuine question. I am not doubting the need for the DSL itself.
surajrmal 2y ago
I'm not sure. I understand what you are trying to say - do you think we should give up on OS diversity? Are you making fun of a company for actually investing in something ambitious which can benefit many if successful? While I'm not sure you are in the same crowd, I always think it's interesting that the HN hivemind tends to be upset with the browser monoculture, but doesn't bat an eye at OS monoculture. It feels like you're really just channeling feelings about the company rather than the projects.
surajrmal 2y ago
Of the three, only two are Google produced. The rust implementation is written by Facebook for use in their build system. The java implementation is the original but is pretty tied to the the bazel build system and not really very suitable to other uses. The go implementation is meant to be embeddable into a varying number of applications. An example application is https://github.com/shac-project/shac https://github.com/shac-project/shac Within Google, there a large number of similar tools which are written in go and harness the starlark language. While there are plenty of other options, I will say I think starlark is often a great choice.
deleted 2y ago
[deleted]

yodsanklai 2y ago

If I understand correctly, Starlark is a strict subset of Python (at least very close to Python). Can anything be more established than Python? Also I don't think pure data is an option. The point is that you want to generate data, which has many benefits (avoid duplication, easier to test). It seems that Starlark is a good trade-off given the constraints. My main grief is that it doesn't have type.

kevindamm 2y ago
The syntax is a pure subset but the implementations are bespoke, so I wouldn't equate the established position of Python with whether Starlark is considered established. I agree that sometimes pure data isn't an option, and I've had to write some Skylark to assist blaze build rules too, but every time it also added tech debt, reduced the number of people on the team who completely understood the build system, and wasn't convenient/practical to test the build extension. The problem I'm referring to above is when the use of a source-controlled data file would have been sufficient but someone had to write a build extension because it's fun or something new to do (or whatever reason seemed convincing at the time). Then there are people who think that those same data files shouldn't be part of the build process at all and should be part of system turn-up/tear-down, stored in a global config or DB where versioning is alongside the data instead of alongside the program build. Certain kinds of migration are made more difficult if everything is built into the binary. Of course, that strategy comes with its own caveats.
phyrex 2y ago
The rust implementation has types