6 ms·

As I mentioned in another comment [0], compiled languages are the future of interpreted languages' tooling. > Whereas before one might have thought that a lang

As I mentioned in another comment [0], compiled languages are the future of interpreted languages' tooling. > Whereas before one might have thought that a language's tooling should be written in said language, there might come a time when that's too slow, and we instead need to move to faster languages. > That's what has happened to Python (numpy, pandas, scipy etc are all written in C and simply provide an interface in Python), and now it's happening to Javascript as well, with Deno, swc in Rust, Bun in Zig, esbuild in Go, and so on. [0] https://news.ycombinator.com/item?id=32577837 https://news.ycombinator.com/item?id=32577837

mikepurvis 4y ago
Is that a bad thing? Libraries like Tensorflow were never going to be written natively in Python, but it's a reasonable choice for deployment (whls can be installed anywhere), for user interface (Python can interop with anything), and for overall ergonomics/ecosystem (import antigravity).
- cercatrova 4y ago
  No, I'm arguing that it's a good thing. There are some who argue that a language's tooling should be written in said language, but I argue that that's putting the cart before the horse, that the tooling should be fast and efficient first and extensible second, because there are many more who are tool users than tool creators, so we should optimize for the former. There is also no reason that someone who is so interested in tooling could not also learn the tool's language, like Rust, and contribute that way.
  - deleted 4y ago
    [deleted]
  - mikepurvis 4y ago
    I guess there's kind of a vague philosophical argument to be made for dogfooding, but that's perhaps more relevant early on in the lifecycle of an ecosystem. On the other hand, if Python had been really committed to dogfooding at the level of something like golang, then the primary implementation would look a lot more like pypy than it would like cpython.
hardwaregeek 4y ago
I agree, but I do think the big question will be supporting plugins without recompiling. The answer is probably WebAssembly or one of the native-node interops but it's still not trivial.
tinalumfoil 4y ago
Python is kind of a special case. It's slow, widely used and often used as "glue" calling out to other programs and libraries that do the real work. There's also a lot of overlap for python and systems programmers, so there's tooling and that community knows how to do these things. There's more of a culture in python that to do anything serious (read fast) you write that part somewhere else and call into it. If you look at JS tooling like eslint, tslint, uglifyjs, tsc, webpack, ~~esbuild~~, etc, it's all JavaScript (moving to typescript if anything). EDIT: I stand corrected in that esbuild is not written in Javascript, but I think that's a disadvantage to it's success if anything and would point out webpack is more popular
- cercatrova 4y ago
  As I mentioned above, new JS tooling is beginning to be written in non-JS/TS languages. esbuild which you mention specifically is in Go, not JS.
  - deleted 4y ago
    [deleted]
- oefrha 4y ago
  esbuild is written in Go, which is a large part of why it’s way faster than other dog slow tools especially webpack in your list.
- guilherme-puida 4y ago
  > EDIT: I stand corrected in that esbuild is not written in Javascript, but I think that's a disadvantage to it's success if anything and would point out webpack is more popular. Webpack exists since 2012, of course it's more popular. For a long time there were no other alternatives, so people had to deal with its madness.
bee_rider 4y ago
Hmm, maybe this is a dumb question, but Numpy, Pandas, Scipy, etc are all libraries. Are libraries considered tooling? When I think of tooling, I think of linters, debuggers, IDEs, and so on. I can definitely see why libraries should be written in faster languages, they are used all over the place. And anyway, something Numpy is most importantly an interface to pre-existing high performance libraries (BLAS, LAPACK) which were already written in faster languages. I think it is less obvious that a tooling in the second sense should be written in a lower level language. There's the obvious flexibility tradeoff, the fact that a person interested in a linter for a language probably is most familiar with that language. I'm not familiar with the computational problems in linters. Is getting better CPU performance a huge concern?
- cercatrova 4y ago
  A linter like Prettier can take several seconds in large codebases while the one by Rome (rome.tools) takes fractions of a second, because the former is in JS and the latter is in Rust. If I'm a linter user, I really don't care what the linter is written in, because I'm never going to open up the source code to look at it. It likely exposes a configuration file which is agnostic to any programming language, and that is what I will primarily be editing. Same thing with Webpack vs esbuild or swc, the latter two are simply instantaneous while Webpack can take minutes, even, to build an app. That it's in JS does not matter to me since I don't care about Webpack source code, I only care about what the tool itself does.
- SAI_Peregrinus 4y ago
  Is a compiler "toolchain" package tooling? Is the standard library included with that "toolchain" tooling, or a library?
  - bee_rider 4y ago
    So, I don't think it is really necessary for there to be a perfectly bright line between the two. But for the sake of argument, I'll say that my opinion is that the compiler, linker, etc are part of the tooling, while the standard library is a library. All of the libraries are linked to the project by some tooling, and a particular distribution of software could include any number of libraries (e.g., conda), I don't see why the standard library needs a special cutout. It will get even more blurry if you start asking about runtimes, macros, and all-template libraries.
_hl_ 4y ago
A counterpoint is the typescript compiler, arguably one of the most complex pieces of dynamic language tooling out there. It's written in typescript itself, and reasonably fast given that it has to implement a hugely sophisticated type system. This shows that it's entirely possible to write fast tooling in dynamic language if you know how to.
- bobbylarrybobby 4y ago
  Is the compiler expected to run only where a JS interpreter exists? (Which, at the time TS was created, consisted solely of Node?)
agumonkey 4y ago
Honestly I believe the low level core, dynamic shell is as old as programming. It's a bit part of the make it right then make it fast.
pjmlp 4y ago
As I learned during my Tcl days, languages like Python should stay as scripting languages, because as you pretty well describe, you end up writing everything in C anyway. A lesson learned 20 years ago, before everything .com went bust.
deleted 4y ago
[deleted]
dceddia 4y ago
I’ve wondered how much those speedups are due to Rust/Go/Zig/etc, versus how much is due to being able to design the system with fresh eyes and an eye for performance. JS is almost surely slower than Rust, for instance, but it’s not 10-100x in most cases, at least in this random set of benchmarks I found [0]. 0: https://programming-language-benchmarks.vercel.app/rust-vs-javascript https://programming-language-benchmarks.vercel.app/rust-vs-j...
- the_duke 4y ago
  Benchmarks are often not a great indicator for real world performance, especially for Javascript. Benchmarks are very constrained, repetitive and often numerical code that is executed a lot of times with similar input. This gives the JIT a lot of opportunity to optimise and specialise for specific types. These tools are usually not long-running , so code has to be JITed all over again on each invocation. A lot of code will probably only be executed a few times, so might only be executed by the interpreter or low tier JIT. There will also be a lot of AST construction, walking and modification, which is really difficult to optimise without the guidance of types. A lower level language can use better data structures that don't require so much pointer chasing. Also lots of potential GC pressure if there are multiple internal representations to construct. JS can be insanely fast for what it is, but alot of the weaknesses are apparent with short-running tooling. There is also a reason why most complex,JS heavy web apps seem to get laggy and slow eventually...
- TakeBlaster16 4y ago
  A major benefit of Rust/Go/Zig is control over data layout. If your language doesn't have value types, the CPU spends a lot of time pointer chasing and trashes the cache. In the worst case a cache miss could have the CPU sitting idle for hundreds of clock cycles while it waits for a RAM fetch to finish. That has a big effect in large programs with many layers of data structures, but it's not usually captured in microbenchmarks.
- terafo 4y ago
  As already mentioned in neighboring comment, these languages give you control over data layout. But what's really powerful - is that control over layout opens up a possibility of using SIMD instructions, which by itself may have 10x speed up over optimized scalar code in the same languages, speed up over JS is, of course, greater.

mikepurvis 4y ago

Is that a bad thing? Libraries like Tensorflow were never going to be written natively in Python, but it's a reasonable choice for deployment (whls can be installed anywhere), for user interface (Python can interop with anything), and for overall ergonomics/ecosystem (import antigravity).

cercatrova 4y ago
No, I'm arguing that it's a good thing. There are some who argue that a language's tooling should be written in said language, but I argue that that's putting the cart before the horse, that the tooling should be fast and efficient first and extensible second, because there are many more who are tool users than tool creators, so we should optimize for the former. There is also no reason that someone who is so interested in tooling could not also learn the tool's language, like Rust, and contribute that way.
- deleted 4y ago
  [deleted]
- mikepurvis 4y ago
  I guess there's kind of a vague philosophical argument to be made for dogfooding, but that's perhaps more relevant early on in the lifecycle of an ecosystem. On the other hand, if Python had been really committed to dogfooding at the level of something like golang, then the primary implementation would look a lot more like pypy than it would like cpython.

hardwaregeek 4y ago

I agree, but I do think the big question will be supporting plugins without recompiling. The answer is probably WebAssembly or one of the native-node interops but it's still not trivial.

tinalumfoil 4y ago

Python is kind of a special case. It's slow, widely used and often used as "glue" calling out to other programs and libraries that do the real work. There's also a lot of overlap for python and systems programmers, so there's tooling and that community knows how to do these things. There's more of a culture in python that to do anything serious (read fast) you write that part somewhere else and call into it. If you look at JS tooling like eslint, tslint, uglifyjs, tsc, webpack, ~~esbuild~~, etc, it's all JavaScript (moving to typescript if anything). EDIT: I stand corrected in that esbuild is not written in Javascript, but I think that's a disadvantage to it's success if anything and would point out webpack is more popular

cercatrova 4y ago
As I mentioned above, new JS tooling is beginning to be written in non-JS/TS languages. esbuild which you mention specifically is in Go, not JS.
- deleted 4y ago
  [deleted]
oefrha 4y ago
esbuild is written in Go, which is a large part of why it’s way faster than other dog slow tools especially webpack in your list.
guilherme-puida 4y ago
> EDIT: I stand corrected in that esbuild is not written in Javascript, but I think that's a disadvantage to it's success if anything and would point out webpack is more popular. Webpack exists since 2012, of course it's more popular. For a long time there were no other alternatives, so people had to deal with its madness.

bee_rider 4y ago

Hmm, maybe this is a dumb question, but Numpy, Pandas, Scipy, etc are all libraries. Are libraries considered tooling? When I think of tooling, I think of linters, debuggers, IDEs, and so on. I can definitely see why libraries should be written in faster languages, they are used all over the place. And anyway, something Numpy is most importantly an interface to pre-existing high performance libraries (BLAS, LAPACK) which were already written in faster languages. I think it is less obvious that a tooling in the second sense should be written in a lower level language. There's the obvious flexibility tradeoff, the fact that a person interested in a linter for a language probably is most familiar with that language. I'm not familiar with the computational problems in linters. Is getting better CPU performance a huge concern?

cercatrova 4y ago
A linter like Prettier can take several seconds in large codebases while the one by Rome (rome.tools) takes fractions of a second, because the former is in JS and the latter is in Rust. If I'm a linter user, I really don't care what the linter is written in, because I'm never going to open up the source code to look at it. It likely exposes a configuration file which is agnostic to any programming language, and that is what I will primarily be editing. Same thing with Webpack vs esbuild or swc, the latter two are simply instantaneous while Webpack can take minutes, even, to build an app. That it's in JS does not matter to me since I don't care about Webpack source code, I only care about what the tool itself does.
SAI_Peregrinus 4y ago
Is a compiler "toolchain" package tooling? Is the standard library included with that "toolchain" tooling, or a library?
- bee_rider 4y ago
  So, I don't think it is really necessary for there to be a perfectly bright line between the two. But for the sake of argument, I'll say that my opinion is that the compiler, linker, etc are part of the tooling, while the standard library is a library. All of the libraries are linked to the project by some tooling, and a particular distribution of software could include any number of libraries (e.g., conda), I don't see why the standard library needs a special cutout. It will get even more blurry if you start asking about runtimes, macros, and all-template libraries.

_hl_ 4y ago

A counterpoint is the typescript compiler, arguably one of the most complex pieces of dynamic language tooling out there. It's written in typescript itself, and reasonably fast given that it has to implement a hugely sophisticated type system. This shows that it's entirely possible to write fast tooling in dynamic language if you know how to.

bobbylarrybobby 4y ago
Is the compiler expected to run only where a JS interpreter exists? (Which, at the time TS was created, consisted solely of Node?)

agumonkey 4y ago

Honestly I believe the low level core, dynamic shell is as old as programming. It's a bit part of the make it right then make it fast.

pjmlp 4y ago

As I learned during my Tcl days, languages like Python should stay as scripting languages, because as you pretty well describe, you end up writing everything in C anyway. A lesson learned 20 years ago, before everything .com went bust.

deleted 4y ago

[deleted]

dceddia 4y ago

I’ve wondered how much those speedups are due to Rust/Go/Zig/etc, versus how much is due to being able to design the system with fresh eyes and an eye for performance. JS is almost surely slower than Rust, for instance, but it’s not 10-100x in most cases, at least in this random set of benchmarks I found [0]. 0: https://programming-language-benchmarks.vercel.app/rust-vs-javascript https://programming-language-benchmarks.vercel.app/rust-vs-j...

the_duke 4y ago
Benchmarks are often not a great indicator for real world performance, especially for Javascript. Benchmarks are very constrained, repetitive and often numerical code that is executed a lot of times with similar input. This gives the JIT a lot of opportunity to optimise and specialise for specific types. These tools are usually not long-running , so code has to be JITed all over again on each invocation. A lot of code will probably only be executed a few times, so might only be executed by the interpreter or low tier JIT. There will also be a lot of AST construction, walking and modification, which is really difficult to optimise without the guidance of types. A lower level language can use better data structures that don't require so much pointer chasing. Also lots of potential GC pressure if there are multiple internal representations to construct. JS can be insanely fast for what it is, but alot of the weaknesses are apparent with short-running tooling. There is also a reason why most complex,JS heavy web apps seem to get laggy and slow eventually...
TakeBlaster16 4y ago
A major benefit of Rust/Go/Zig is control over data layout. If your language doesn't have value types, the CPU spends a lot of time pointer chasing and trashes the cache. In the worst case a cache miss could have the CPU sitting idle for hundreds of clock cycles while it waits for a RAM fetch to finish. That has a big effect in large programs with many layers of data structures, but it's not usually captured in microbenchmarks.
terafo 4y ago
As already mentioned in neighboring comment, these languages give you control over data layout. But what's really powerful - is that control over layout opens up a possibility of using SIMD instructions, which by itself may have 10x speed up over optimized scalar code in the same languages, speed up over JS is, of course, greater.