4 ms·

Rust (and other fast, compiled languages) is the future of Javascript infrastructure [0]. Whereas before one might have thought that a language's tooling should

Rust (and other fast, compiled languages) is the future of Javascript infrastructure [0]. Whereas before one might have thought that a language's tooling should be written in said language, there might come a time when that's too slow, and we instead need to move to faster languages. That's what has happened to Python (numpy, pandas, scipy etc are all written in C and simply provide an interface in Python), and now it's happening to Javascript as well, with Deno, swc in Rust, Bun in Zig, esbuild in Go, and so on. I foresee a future where we won't have to deal with slow tooling and can have instantaneous updates once again. [0] https://leerob.io/blog/rust https://leerob.io/blog/rust

jpgvm 4y ago
Eventually you realise this split sucks though. it creates a world where programmers of the higher level language can't reasonably debug their most important libraries. This is different from not being able to debug the runtime, which has a smaller API surface than important libraries and frameworks generally have (though I guess it depends on if you count stdlib etc sometimes). It's generally much better to simply go to a language that still operates at the same or similar level of abstraction but with higher performance. i.e going from Typescript to Kotlin isn't a substantial move from an abstraction level perspective but you no longer consider needing to write the code in C because it's not fast enough. (you might for other reasons, like runtime portability)
- cercatrova 4y ago
  How does that work if end users want to use JS though? They're not all going to move to Kotlin. Since one must meet people where they are, rather than where we want them to be, the best we can do is instead have the end user still program in whatever language they want but then translate that into a higher performance language. That's basically what Python is doing now. Users don't want to write C/C++, they want to use Python, but Python is too slow, so here we are.
  - jakobnissen 4y ago
    What you're describing is known as the "two language problem". The long-term solution is to move to fast, high-level languages. I think Julia and LuaJIT are the only examples currently, but surely more will crop up. This is s fundamental problem that will not go away on its own. Efforts like Numba can alleviate some of the pain of a slow Python, but it turns out it's generally not possible to retroactively fit a JIT compiler onto a language designed not designed to be compiled. In my experience, once I tried moving my work code to Julia, it became clear to me just how much energy I had spent trying to overcome the performance wall of Python, and I can't imagine going back now.
    - v3ss0n 4y ago
      Python have PyPy and it's underrated. Imagine 7-20x performance boost without changing a line of code.
      - jakobnissen 4y ago
        IIUC, the performance of PyPy is still quite far behind normal compiled languages, due to limitations inherent in Python. And then it has limitations in its interop with Numpy and C libraries, at least last I checked. That's the general picture when trying to overcome Python's performance wall: Sometimes you can use Cython, or Numba, or PyPy or Numpy vectorization, or call into C. But only sometimes, and they each come with their own set of awful restrictions and caveats. It's such a breath of fresh air to switch to a fast language and just forget all those hacks and workarounds. I get why people just migrate to static languages
    - zem 4y ago
      second time this week i've had occasion to mention lush[1], which was in many ways an early attempt at what julia is doing now. from their homepage: 8<-------------------------------------------- Lush brings the best of both worlds by wrapping three languages into one: (1) a weakly-typed, garbage-collected, dynamically scoped, interpreted language with a simple Lisp-like syntax, (2) a strongly-typed, lexically-scoped compiled language that uses the same Lisp-like syntax, and (3) the C language, which can be freely mixed with Lush code within a single program, even within a single function. It sounds complicated, but it is not. In fact, Lush is designed to be very simple to learn and easy to use. 8<-------------------------------------------- [1] http://lush.sourceforge.net/ http://lush.sourceforge.net/
- fnord123 4y ago
  'It's the platform, stupid.' The language is secondary to the platform. Having fast libraries is great. No one ever wishes that numpy was written in python. And switching from typescript to kotlin? That's changing platforms. If you're in the browser or in node and replace it with the JVM? That's a wild suggestion to avoid some fast libraries. Imo you're just wrong.
  - DrScientist 4y ago
    Totally agree with the 'It's the platform, stupid'. However, I do wonder if for javascript/typescript that's only for 'in browser' stuff. If you look at the server - one of the best platforms is the JVM - lots of quality libraries - easy to write stable long running processes etc. ie if you are driven by platform isn't it Javascript on the browser and Java on the JVM on the back? ( typescript or kotlin are both moving a bit away from the platform ). The beauty of the network ( message passing ) means the stuff on the server doesn't need to be in the same language as that on the client. Now I'll admit there is some benefit to having front and backend the same, in terms of validation code, or perhaps occassionally moving logic from server to client or back - but does this override the platform effect? Also note you can run js on the JVM/GraalVM - so client/server validation libraries could be written in js and reused both sides.
    - AYBABTME 4y ago
      JVM is not good for tooling, at least not in comparison to Go or Rust, which are both faster to write, easier to distribute and faster to execute for short lived CLI invocations.
  - myko 4y ago
    Like many other languages targeting the web Kotlin will "compile" to JS and is deployable to the web.
- lucasyvas 4y ago
  I would argue this means the users of the higher level language need to learn the language their tooling is written in, or consider switching to the underlying compiled language altogether where possible. My entire career has been with scripting languages, but I think the pendulum is swinging again and they are back on notice. Anything that isn't strictly a scripting need is orders of magnitude more ergonomic to do in a compiled language than it was a decade ago. Why fight it? I'm still watching the developments of WASM closely, because while JS does the trick and always will, if you can build an app in a language that produces an equal or better result in WASM, why wouldn't you consider it from an engineering point of view? Then there would no longer be a language split. There's a lot of "ifs" here for sure, but the possibility exists that the languages we currently use are not the best for the task. I think when we see projects like this, it's a good opportunity to reflect.
cies 4y ago
Tools for interpreted dyntyped languages can be fast, helping the dev flow (edit-run-check) in those dyntyped langs to be fast. But those tools are then in written in statically typed langs that do not have such fast dev flows (as they have an extra step: edit-COMPILE-run-check). I wonder how we can have the best of both worlds. I see a glimpse of this using Kotlin: I can use the JVM to run in debug mode and have very fast compile times due to incremental builds and hot code reloads), then I can compile the code to "native" (binary) for production scenarios. This way we can have strong type guarantees AND fast dev flows.
- yencabulator 4y ago
  This is a false dichotomy. Plenty of Javascript projects build slower than a comparable Go project, because of webpack etc bloat and not caching results as well.
  - cies 4y ago
    True that. But changing a few lines in and hot-reloading is simply not possible in case of compiled langs; that was what I was driving at. One could say JSX/TS/TSX are compiled langs too...
    - yencabulator 4y ago
      Erlang is a bytecode VM, so "compiled" for sure, and is pretty much the pioneer of hot upgrades. Plenty of Lisps and Schemes are compiled (though often to bytecode) and can typically dynamically replace parts of the program. I think ability to hot upgrade seamlessly is more a statement of "how much of the data structure shape is carried at runtime" / "how likely it is old and new API data structures happen to interoperate", and how likely it is that one can convert old runtime state to new runtime state. And that one can be seen as a trade-off; close to the metal control over memory layout is a performance gain, but in practice trades off this kind of flexibility (in theory you could make it work, but it's probably a lot of work). https://www.theseus-os.com/ https://www.theseus-os.com/ is an experimental kernel that can restart/reload/upgrade Rust components at ELF library boundaries. (State internal to a component has to be discarded unless you program a converter. Then again most JS web development hot reload discards internal state of a component.)
andsoitis 4y ago
This seems a little giddy to me. In your linked article they claim that parts of key JS infra is being rewritten in Rust, but the claim seems exaggerated from where I sit. For instance, they list webpack but is that a good example of huge momentum to rewrite webpack in rust? You Python examples are of libraries that are used at runtime, not infrastructure. Coming back to JS, this svelte project is noted by the author as being very early stage and it seems more like a learning project for them. I don’t need to add that svelte isn’t nearly as broadly used as other JS tech. Don’t get me wrong, I’m not saying writing tools in faster compiled languages shouldn’t be done, but one might find that the excitement and prediction seems premature. There are trade offs - speed isn’t free.
- cercatrova 4y ago
  > huge momentum to rewrite webpack in rust Yes, see swc and esbuild (in Go) for examples. They are taking off in the JS world such that many other frameworks are eschewing Webpack and wrapping them due to their sheer speed, like NextJS with swc and Vite with esbuild. > libraries that are used at runtime, not infrastructure Sure, I was just giving an example. I would say though that due to JS being interpreted (okay, JITed), the infrastructure would have to run Webpack anyway, in order to bundle/compile the JS. > this svelte project is noted by the author as being very early stage and it seems more like a learning project for them I'm not talking about this project in particular, just the JS ecosystem as a whole is trending towards compiled-language tooling. But I will say though that this trend is not early stage as the examples above like swc/esbuild have been some years in the making. If anything this will simply accelerate such progress.
lucideer 4y ago
> I foresee a future where we won't have to deal with slow tooling and can have instantaneous updates once again. You quote Python examples around data processing (an appropriate niche use-case for perf-oriented implementations), but then your final line is actually about build tooling. The problem here is that things like esbuild are written to support dysfunction. Javascript-written build tooling build "reasonable" projects very fast. The dependency bloat in the NPM ecosystem is well-documented, and the actual level abstraction complexity of apps/frameworks/projects is also commonly considered to be excessive. This leads to slow builds, slow apps, and low maintainability/debuggability. Solving the first of those problems in isolation "supports dysfunction" in that it makes the latter problems less likely to be solved. It's especially bad when that solution comes with further maintainability/debuggability compromises.
- cercatrova 4y ago
  Slow tooling, slow runtime, it's all the same to me, I want fast programs. Not sure what you mean by esbuild written to support dysfunction, could you elaborate on why that's the case? If you mean that well-developed JS-made tooling can be on par with esbuild and similar, I will have to disagree as benchmarks show that compiled languages like Go and Rust are orders of magnitude faster, and whatever architectural improvements can be made in the JS-made tooling can also be made in these compiled-languages' tooling as well.
  - lucideer 4y ago
    Writing software in a fast language -vs- whatever quick-to-write-/maintainable-/domain-specific-/sandboxed-/etc. language is always some kind of compromise of perf. vs whatever your other priorities are. In particular, writing tooling for a language in that language will have massive advantages in terms of contributors, but may have perf. drawbacks if that language is not as performant as alternatives. Weighing up those pros & cons is important - the performant option is not always the best, it depends on your priorities. Performance generally increases in priority depending on how much of a bottleneck it is. If webpack takes 0.2 seconds and esbuild would take 0.02 seconds, that's a 10x perf. gain but is probably not a compelling reason to switch. If webpack is taking 20 seconds however, that's a bottleneck that's worth looking at. My point above is that an app that takes 20 seconds for webpack to build is overengineered in the most common cases - taking that badly written app and running it through esbuild to "fix" your problem isn't really fixing your problem, it's just hiding it under the bed. This is what's called "supporting dysfunction". Numpy, pandas, scipy are poor comparators because they're processing data, not code: they're tasks that depend on data scale, rather than on how many over-abstracted layers of code someone is trying to compile all at once. The former is not (necessarily) a sign of dysfunction. It's much more likely to be a valid use-case.

jpgvm 4y ago

Eventually you realise this split sucks though. it creates a world where programmers of the higher level language can't reasonably debug their most important libraries. This is different from not being able to debug the runtime, which has a smaller API surface than important libraries and frameworks generally have (though I guess it depends on if you count stdlib etc sometimes). It's generally much better to simply go to a language that still operates at the same or similar level of abstraction but with higher performance. i.e going from Typescript to Kotlin isn't a substantial move from an abstraction level perspective but you no longer consider needing to write the code in C because it's not fast enough. (you might for other reasons, like runtime portability)

cercatrova 4y ago
How does that work if end users want to use JS though? They're not all going to move to Kotlin. Since one must meet people where they are, rather than where we want them to be, the best we can do is instead have the end user still program in whatever language they want but then translate that into a higher performance language. That's basically what Python is doing now. Users don't want to write C/C++, they want to use Python, but Python is too slow, so here we are.
- jakobnissen 4y ago
  What you're describing is known as the "two language problem". The long-term solution is to move to fast, high-level languages. I think Julia and LuaJIT are the only examples currently, but surely more will crop up. This is s fundamental problem that will not go away on its own. Efforts like Numba can alleviate some of the pain of a slow Python, but it turns out it's generally not possible to retroactively fit a JIT compiler onto a language designed not designed to be compiled. In my experience, once I tried moving my work code to Julia, it became clear to me just how much energy I had spent trying to overcome the performance wall of Python, and I can't imagine going back now.
  - v3ss0n 4y ago
    Python have PyPy and it's underrated. Imagine 7-20x performance boost without changing a line of code.
    - jakobnissen 4y ago
      IIUC, the performance of PyPy is still quite far behind normal compiled languages, due to limitations inherent in Python. And then it has limitations in its interop with Numpy and C libraries, at least last I checked. That's the general picture when trying to overcome Python's performance wall: Sometimes you can use Cython, or Numba, or PyPy or Numpy vectorization, or call into C. But only sometimes, and they each come with their own set of awful restrictions and caveats. It's such a breath of fresh air to switch to a fast language and just forget all those hacks and workarounds. I get why people just migrate to static languages
  - zem 4y ago
    second time this week i've had occasion to mention lush[1], which was in many ways an early attempt at what julia is doing now. from their homepage: 8<-------------------------------------------- Lush brings the best of both worlds by wrapping three languages into one: (1) a weakly-typed, garbage-collected, dynamically scoped, interpreted language with a simple Lisp-like syntax, (2) a strongly-typed, lexically-scoped compiled language that uses the same Lisp-like syntax, and (3) the C language, which can be freely mixed with Lush code within a single program, even within a single function. It sounds complicated, but it is not. In fact, Lush is designed to be very simple to learn and easy to use. 8<-------------------------------------------- [1] http://lush.sourceforge.net/ http://lush.sourceforge.net/
fnord123 4y ago
'It's the platform, stupid.' The language is secondary to the platform. Having fast libraries is great. No one ever wishes that numpy was written in python. And switching from typescript to kotlin? That's changing platforms. If you're in the browser or in node and replace it with the JVM? That's a wild suggestion to avoid some fast libraries. Imo you're just wrong.
- DrScientist 4y ago
  Totally agree with the 'It's the platform, stupid'. However, I do wonder if for javascript/typescript that's only for 'in browser' stuff. If you look at the server - one of the best platforms is the JVM - lots of quality libraries - easy to write stable long running processes etc. ie if you are driven by platform isn't it Javascript on the browser and Java on the JVM on the back? ( typescript or kotlin are both moving a bit away from the platform ). The beauty of the network ( message passing ) means the stuff on the server doesn't need to be in the same language as that on the client. Now I'll admit there is some benefit to having front and backend the same, in terms of validation code, or perhaps occassionally moving logic from server to client or back - but does this override the platform effect? Also note you can run js on the JVM/GraalVM - so client/server validation libraries could be written in js and reused both sides.
  - AYBABTME 4y ago
    JVM is not good for tooling, at least not in comparison to Go or Rust, which are both faster to write, easier to distribute and faster to execute for short lived CLI invocations.
- myko 4y ago
  Like many other languages targeting the web Kotlin will "compile" to JS and is deployable to the web.
lucasyvas 4y ago
I would argue this means the users of the higher level language need to learn the language their tooling is written in, or consider switching to the underlying compiled language altogether where possible. My entire career has been with scripting languages, but I think the pendulum is swinging again and they are back on notice. Anything that isn't strictly a scripting need is orders of magnitude more ergonomic to do in a compiled language than it was a decade ago. Why fight it? I'm still watching the developments of WASM closely, because while JS does the trick and always will, if you can build an app in a language that produces an equal or better result in WASM, why wouldn't you consider it from an engineering point of view? Then there would no longer be a language split. There's a lot of "ifs" here for sure, but the possibility exists that the languages we currently use are not the best for the task. I think when we see projects like this, it's a good opportunity to reflect.

cies 4y ago

Tools for interpreted dyntyped languages can be fast, helping the dev flow (edit-run-check) in those dyntyped langs to be fast. But those tools are then in written in statically typed langs that do not have such fast dev flows (as they have an extra step: edit-COMPILE-run-check). I wonder how we can have the best of both worlds. I see a glimpse of this using Kotlin: I can use the JVM to run in debug mode and have very fast compile times due to incremental builds and hot code reloads), then I can compile the code to "native" (binary) for production scenarios. This way we can have strong type guarantees AND fast dev flows.

yencabulator 4y ago
This is a false dichotomy. Plenty of Javascript projects build slower than a comparable Go project, because of webpack etc bloat and not caching results as well.
- cies 4y ago
  True that. But changing a few lines in and hot-reloading is simply not possible in case of compiled langs; that was what I was driving at. One could say JSX/TS/TSX are compiled langs too...
  - yencabulator 4y ago
    Erlang is a bytecode VM, so "compiled" for sure, and is pretty much the pioneer of hot upgrades. Plenty of Lisps and Schemes are compiled (though often to bytecode) and can typically dynamically replace parts of the program. I think ability to hot upgrade seamlessly is more a statement of "how much of the data structure shape is carried at runtime" / "how likely it is old and new API data structures happen to interoperate", and how likely it is that one can convert old runtime state to new runtime state. And that one can be seen as a trade-off; close to the metal control over memory layout is a performance gain, but in practice trades off this kind of flexibility (in theory you could make it work, but it's probably a lot of work). https://www.theseus-os.com/ https://www.theseus-os.com/ is an experimental kernel that can restart/reload/upgrade Rust components at ELF library boundaries. (State internal to a component has to be discarded unless you program a converter. Then again most JS web development hot reload discards internal state of a component.)

andsoitis 4y ago

This seems a little giddy to me. In your linked article they claim that parts of key JS infra is being rewritten in Rust, but the claim seems exaggerated from where I sit. For instance, they list webpack but is that a good example of huge momentum to rewrite webpack in rust? You Python examples are of libraries that are used at runtime, not infrastructure. Coming back to JS, this svelte project is noted by the author as being very early stage and it seems more like a learning project for them. I don’t need to add that svelte isn’t nearly as broadly used as other JS tech. Don’t get me wrong, I’m not saying writing tools in faster compiled languages shouldn’t be done, but one might find that the excitement and prediction seems premature. There are trade offs - speed isn’t free.

cercatrova 4y ago
> huge momentum to rewrite webpack in rust Yes, see swc and esbuild (in Go) for examples. They are taking off in the JS world such that many other frameworks are eschewing Webpack and wrapping them due to their sheer speed, like NextJS with swc and Vite with esbuild. > libraries that are used at runtime, not infrastructure Sure, I was just giving an example. I would say though that due to JS being interpreted (okay, JITed), the infrastructure would have to run Webpack anyway, in order to bundle/compile the JS. > this svelte project is noted by the author as being very early stage and it seems more like a learning project for them I'm not talking about this project in particular, just the JS ecosystem as a whole is trending towards compiled-language tooling. But I will say though that this trend is not early stage as the examples above like swc/esbuild have been some years in the making. If anything this will simply accelerate such progress.

lucideer 4y ago

> I foresee a future where we won't have to deal with slow tooling and can have instantaneous updates once again. You quote Python examples around data processing (an appropriate niche use-case for perf-oriented implementations), but then your final line is actually about build tooling. The problem here is that things like esbuild are written to support dysfunction. Javascript-written build tooling build "reasonable" projects very fast. The dependency bloat in the NPM ecosystem is well-documented, and the actual level abstraction complexity of apps/frameworks/projects is also commonly considered to be excessive. This leads to slow builds, slow apps, and low maintainability/debuggability. Solving the first of those problems in isolation "supports dysfunction" in that it makes the latter problems less likely to be solved. It's especially bad when that solution comes with further maintainability/debuggability compromises.

cercatrova 4y ago
Slow tooling, slow runtime, it's all the same to me, I want fast programs. Not sure what you mean by esbuild written to support dysfunction, could you elaborate on why that's the case? If you mean that well-developed JS-made tooling can be on par with esbuild and similar, I will have to disagree as benchmarks show that compiled languages like Go and Rust are orders of magnitude faster, and whatever architectural improvements can be made in the JS-made tooling can also be made in these compiled-languages' tooling as well.
- lucideer 4y ago
  Writing software in a fast language -vs- whatever quick-to-write-/maintainable-/domain-specific-/sandboxed-/etc. language is always some kind of compromise of perf. vs whatever your other priorities are. In particular, writing tooling for a language in that language will have massive advantages in terms of contributors, but may have perf. drawbacks if that language is not as performant as alternatives. Weighing up those pros & cons is important - the performant option is not always the best, it depends on your priorities. Performance generally increases in priority depending on how much of a bottleneck it is. If webpack takes 0.2 seconds and esbuild would take 0.02 seconds, that's a 10x perf. gain but is probably not a compelling reason to switch. If webpack is taking 20 seconds however, that's a bottleneck that's worth looking at. My point above is that an app that takes 20 seconds for webpack to build is overengineered in the most common cases - taking that badly written app and running it through esbuild to "fix" your problem isn't really fixing your problem, it's just hiding it under the bed. This is what's called "supporting dysfunction". Numpy, pandas, scipy are poor comparators because they're processing data, not code: they're tasks that depend on data scale, rather than on how many over-abstracted layers of code someone is trying to compile all at once. The former is not (necessarily) a sign of dysfunction. It's much more likely to be a valid use-case.