4 ms·

Since 2012 we have, as a field, learned better. You see, we all realized that just building code wasn't enough! What your 100 line JavaScript microservice real

Since 2012 we have, as a field, learned better. You see, we all realized that just building code wasn't enough! What your 100 line JavaScript microservice really needs to be accompanied by is a build file that determines how to create an entire Linux VM. Also, because scale, your project also needs to know how to scale that VM out across multiple clusters! The only answer of course is more build systems! Oh but JavaScript sucks, so make that Typescript, so shove a transpiler in front of your scripting language. Because somehow we've gone down the path of needing build systems for interpreted languages. I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/ (I legit like Babel and TS, but you have to admit it is /weird/ that we're going through the complexity of having a build system but are reaping none of the performance benefits!)

ReactiveJelly 6y ago
Technically containers are not VMs And re: build systems for interpreted languages, what would the alternative be? Asking Firefox and Chrome to integrate a language they didn't design? Making it a runtime module like MoonScript, and letting everyone who loads the page pay for compilation, on battery-constrained computers, millions of times over?
- EE84M3i 6y ago
  >Technically containers are not VMs Some build systems do build proper VM images, like AMIs.
  - rumanator 6y ago
    You're talking about a cloud service provider, whose bread and butter is selling you time on their VMs. Of course they manage machine images.
qubidt 6y ago
It's the classic question of "Reform or Revolution?" These systems were built as incremental mitigations to old problems. I don't think it'll change until they collapse under their own weight and people switch over to something else completely.
baddox 6y ago
From my perspective, the main reason for most browser JS projects having a build system is to essentially concatenate modules and to transpile newer JS features so they work on older browsers. That’s really just not as crazy as you make it sound. I don’t know about the Linux VM stuff being part of a browser JS project, maybe I’m misunderstanding that part but it does sound crazy.
- ori_b 6y ago
  > I don’t know about the Linux VM stuff being part of a browser JS project, maybe I’m misunderstanding that part but it does sound crazy. A bunch of stuff I see comes with a dockerfile these days. While I get building in docker for reproducibility, kind of -- shipping with one is a pretty big red flag. It means that it's likely to be very painful to port the code out of docker and integrate it into some other build. Which means that it'll be painful to integrate it with other code. Leave the build environment set up to the users. Embrace build diversity, it makes your code less fragile.
  - Someone 6y ago
    At my job, a colleague shared a solution for tunneling into servers that involved having a Docker image running for each server, with some tunneling tricks inside the image. Those are the times I wonder whether I’m getting old. I kept using a shell alias that does ssh -ND, even though that requires maintaining a config file. But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment?
    - ori_b 6y ago
      > But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment? I often need to run it somewhere other than docker -- possibly integrating it into a different program. Docker doesn't make this impossible, but it's a sign that portability was not considered when the code was written, and I'm in for pain.
MagnumPIG 6y ago
It's more like the very least we could collectively achieve while remaining functional.
jnxx 6y ago
> I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/ I think it can be done better, and above all, much simpler. I think "redo" looks like a very simple, good solution - here is the documentation: https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/ Of course it is only a tool and it does not convert an overcomplicated bloated system into a three-liner. But unnécessary complexity in build systems sucks big time, and interferes with decent testing, so I think this is a real gem.
- bonestormii_ 6y ago
  > (Disclaimer: my current implementation is not as fast as make for some things, because it's written in python. Eventually I'll rewrite it an C and it'll be very, very fast.) Lol. Whatever, python community... Anyway, I don't think redo really provides an answer to the insanity that is the js build system(s). I'm also a bit of a DJB fan, like python as a language, etc. However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system. He uses SQLite instead of the file system as a database paradigm favored by DJB. DJB's software is all about technical correctness and simplicity of design. There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it. The whole "I'm moving my python code to C tomorrow" thing is becoming a bit of a smell IMHO. Shell script or a compiled language would be preferable in my opinion.
  - jnxx 6y ago
    > However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system. I think that Python as a draft/concept language, for getting the semantics right, and after that rewriting it into a standalone C program is sound, and that is the stated plan for the project. I think what originally matters is not how it is implemented, but but how it is defined, what are the exact semantics, this is what was the Python development about. Because it is very simple and around 3400 lines of python, creating additional implementations should be not too much work, and my understanding is that a C implementation is the next planned step. > He uses SQLite instead of the file system as a database paradigm favored by DJB. Speed is one issue here, it also needs to support efficient multi-processing. Based on my own limited experience with something similar, my personal favorite for this kind of thing is LMDB (https://en.wikipedia.org/wiki/LMDB https://en.wikipedia.org/wiki/LMDB), which is well-tested, but I guess it would be very hard to make that into a portable version which runs, for example, on Windows, too. > There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it. From following the list (https://groups.google.com/forum/#!forum/redo-list https://groups.google.com/forum/#!forum/redo-list) and blog, my impression is that the main author is a bit of a quality and reliability fanatic. So, for that kind of project, exactly the right person. He provided a test suite which can be used to test alternative implementations, as there are already a few around. > Shell script or a compiled language would be preferable in my opinion. There exists a basic (non-parallel) implementation in shell. shell is however going to be a bit too slow for a "production" implementation for large projects where speed matters. My favorite language for this might be guile (I think it matches Guix well), or perhaps Common Lisp. For maximum portability, I believe C is really the best thing. I'd have nothing to say against an ultra-fast implementation for POSIX systems, written in Rust which calls into LMDB. But while this might be beautiful from a technical perspective, I don't think that there will be a noticeable speed difference in normal use.
dfee 6y ago
Your remark is funny, as I was looking at Bazel for TS last night. It seems that Bazel and PNPM don't work together, so I pretty much just have to punt on it, unfortunately (yes, I have a monorepo). Alas, it seems like a ton of extra effort to have BUILD files everywhere that repeat the imports of their sibling TS files. And on top of that, how am I going to ensure they stay up to date? What if someone forgets to manually add a dependency to the BUILD file after they've already done an `import {} from ...`? What if I don't want to vendor dependencies? Unfortunately, it seems like Bazel + TS could be really cool, but the amount of extra effort won't pay off because there are foot-guns all over the place.

ReactiveJelly 6y ago

Technically containers are not VMs And re: build systems for interpreted languages, what would the alternative be? Asking Firefox and Chrome to integrate a language they didn't design? Making it a runtime module like MoonScript, and letting everyone who loads the page pay for compilation, on battery-constrained computers, millions of times over?

EE84M3i 6y ago
>Technically containers are not VMs Some build systems do build proper VM images, like AMIs.
- rumanator 6y ago
  You're talking about a cloud service provider, whose bread and butter is selling you time on their VMs. Of course they manage machine images.

qubidt 6y ago

It's the classic question of "Reform or Revolution?" These systems were built as incremental mitigations to old problems. I don't think it'll change until they collapse under their own weight and people switch over to something else completely.

baddox 6y ago

From my perspective, the main reason for most browser JS projects having a build system is to essentially concatenate modules and to transpile newer JS features so they work on older browsers. That’s really just not as crazy as you make it sound. I don’t know about the Linux VM stuff being part of a browser JS project, maybe I’m misunderstanding that part but it does sound crazy.

ori_b 6y ago
> I don’t know about the Linux VM stuff being part of a browser JS project, maybe I’m misunderstanding that part but it does sound crazy. A bunch of stuff I see comes with a dockerfile these days. While I get building in docker for reproducibility, kind of -- shipping with one is a pretty big red flag. It means that it's likely to be very painful to port the code out of docker and integrate it into some other build. Which means that it'll be painful to integrate it with other code. Leave the build environment set up to the users. Embrace build diversity, it makes your code less fragile.
- Someone 6y ago
  At my job, a colleague shared a solution for tunneling into servers that involved having a Docker image running for each server, with some tunneling tricks inside the image. Those are the times I wonder whether I’m getting old. I kept using a shell alias that does ssh -ND, even though that requires maintaining a config file. But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment?
  - ori_b 6y ago
    > But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment? I often need to run it somewhere other than docker -- possibly integrating it into a different program. Docker doesn't make this impossible, but it's a sign that portability was not considered when the code was written, and I'm in for pain.

MagnumPIG 6y ago

It's more like the very least we could collectively achieve while remaining functional.

jnxx 6y ago

> I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/ I think it can be done better, and above all, much simpler. I think "redo" looks like a very simple, good solution - here is the documentation: https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/ Of course it is only a tool and it does not convert an overcomplicated bloated system into a three-liner. But unnécessary complexity in build systems sucks big time, and interferes with decent testing, so I think this is a real gem.

bonestormii_ 6y ago
> (Disclaimer: my current implementation is not as fast as make for some things, because it's written in python. Eventually I'll rewrite it an C and it'll be very, very fast.) Lol. Whatever, python community... Anyway, I don't think redo really provides an answer to the insanity that is the js build system(s). I'm also a bit of a DJB fan, like python as a language, etc. However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system. He uses SQLite instead of the file system as a database paradigm favored by DJB. DJB's software is all about technical correctness and simplicity of design. There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it. The whole "I'm moving my python code to C tomorrow" thing is becoming a bit of a smell IMHO. Shell script or a compiled language would be preferable in my opinion.
- jnxx 6y ago
  > However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system. I think that Python as a draft/concept language, for getting the semantics right, and after that rewriting it into a standalone C program is sound, and that is the stated plan for the project. I think what originally matters is not how it is implemented, but but how it is defined, what are the exact semantics, this is what was the Python development about. Because it is very simple and around 3400 lines of python, creating additional implementations should be not too much work, and my understanding is that a C implementation is the next planned step. > He uses SQLite instead of the file system as a database paradigm favored by DJB. Speed is one issue here, it also needs to support efficient multi-processing. Based on my own limited experience with something similar, my personal favorite for this kind of thing is LMDB (https://en.wikipedia.org/wiki/LMDB https://en.wikipedia.org/wiki/LMDB), which is well-tested, but I guess it would be very hard to make that into a portable version which runs, for example, on Windows, too. > There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it. From following the list (https://groups.google.com/forum/#!forum/redo-list https://groups.google.com/forum/#!forum/redo-list) and blog, my impression is that the main author is a bit of a quality and reliability fanatic. So, for that kind of project, exactly the right person. He provided a test suite which can be used to test alternative implementations, as there are already a few around. > Shell script or a compiled language would be preferable in my opinion. There exists a basic (non-parallel) implementation in shell. shell is however going to be a bit too slow for a "production" implementation for large projects where speed matters. My favorite language for this might be guile (I think it matches Guix well), or perhaps Common Lisp. For maximum portability, I believe C is really the best thing. I'd have nothing to say against an ultra-fast implementation for POSIX systems, written in Rust which calls into LMDB. But while this might be beautiful from a technical perspective, I don't think that there will be a noticeable speed difference in normal use.

dfee 6y ago

Your remark is funny, as I was looking at Bazel for TS last night. It seems that Bazel and PNPM don't work together, so I pretty much just have to punt on it, unfortunately (yes, I have a monorepo). Alas, it seems like a ton of extra effort to have BUILD files everywhere that repeat the imports of their sibling TS files. And on top of that, how am I going to ensure they stay up to date? What if someone forgets to manually add a dependency to the BUILD file after they've already done an `import {} from ...`? What if I don't want to vendor dependencies? Unfortunately, it seems like Bazel + TS could be really cool, but the amount of extra effort won't pay off because there are foot-guns all over the place.