24 ms·

How Turborepo is porting from Go to Rust

116 points by hardwaregeek 3y ago

djoldman 3y ago
This was referenced, which I thought was interesting: https://github.com/golang/go/issues/13492 https://github.com/golang/go/issues/13492 golang cannot be compiled as a C static library with musl.
- deleted 3y ago
  [deleted]
phamilton 3y ago
Ah, another tale that includes getting bit by static linking and alpine. Maybe I've just had a bad sample size, but I just haven't experienced a big enough win by using alpine to justify the weird scenarios that come up on occasion. The idea is nice. A small, stripped down container that will load quickly and have very few maintenance issues (due to basically zero dependencies). But my debian-slim images work well enough and when I do hit a problem there's more community around it and it's more straightforward to fix.
- deusum 3y ago
  People keep saying they need a light and fast distro - no systemd, no glibc - but it breaks a lot of other software. So, you need to be prepared to patch a lot of stuff, and build packages from source. Personally, I would love more of them to consider FreeBSD. It's got all of those features, a linux compatibility layer if necessary, a more permissive license, etc. I'd just love to see a lot more developers helping over there.
  - packetlost 3y ago
    Except you can't run FreeBSD in Kubernetes really. Or really anywhere besides in VMs in the Linux monoculture.
  - richardwhiuk 3y ago
    Moving to FreeBSD would require patching even more things.
  - ASalazarMX 3y ago
    > a more permissive license I see this as a net disadvantage. The great insight of the GPL licences is forcing changes to be contributed back to the project, companies can't easily privatize a public effort.
    - szundi 3y ago
      And this makes GPL based software like Linux stronger, more popular, more community to help
- dontlaugh 3y ago
  It's also much easier to set up debugging tools when necessary.
- klooney 3y ago
  Glibc is very good. They should get plaudits somewhere by someone occasionally.
- adameasterling 3y ago
  Alpine is a trap! For a Python product, we tried to make it work for about a year before throwing our hands up and switching to a plain Ubuntu image. We’ve had no regrets.
- returningfory2 3y ago
  Small Docker image or small binary size is this decade's canonical premature optimization.
  - didntcheck 3y ago
    Especially base images. The size of your own layers might be worth worrying about, but not the shared base that should only be stored once for arbitrarily many containers
- rollcat 3y ago
  I am concerned about the Linux/glibc monoculture, and its effect on the broader ecosystem. Whatever breaks on Linux/musl, is also likely to break on other operating systems. Whatever breaks loudly elsewhere, is also quite likely silently broken on Linux/glibc. If our default response is to double down on barely patching it enough to limp along, no wonder it keeps breaking.
  - onlypositive 3y ago
    Most the issues with alpine Linux people face are down to package maintainers breaking backwards compatibility constantly. Package renames are a regular problem.
    - singron 3y ago
      One thing I love about NixOS so far is that when they do a rename, they can print a deprecation warning every time you use the old name while evaluating your config. When I get a moment, I go through and fix them. You could probably do something similar in a traditional distro by creating an alias package that prints a warning on install, but then you actually have to watch the install logs to see that.
    - throw16180339 3y ago
      Alpine has at least the following deficiencies: DNS TCP resolution doesn't work (https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-on-alpine-linux https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-o...). It's a feature that it doesn't retry truncated lookups or obey the relevant RFCs (https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296). Alpine has a 128k stack size (https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/ https://ariadne.space/2021/06/25/understanding-thread-stack-...), while most other OSes have at least a 512k stack.
      - rollcat 3y ago
        Just to be clear, I'm not blindly defending musl/Alpine, rather noting the detrimental effects of monoculture. Quite often fixing a build or a package on another system or architecture is a matter of a 3-5 line diff, and I've contributed quite a few over the years. It usually boils down to an incorrect assumption by the author/maintainer, stuff like "#ifdef __linux__" (when what you actually mean is: "any UNIX-like system with X11"), or hardcoding CC=gcc (where CC=cc just works).
- auspex 3y ago
  Alpine is also based on MUSCL rather than glibc which generally has fewer well known CVEs which helps from a security standpoint.
  - danpalmer 3y ago
    Is that because it's got better security, or fewer eyes on it?
    - mike_hock 3y ago
      Both.
yndoendo 3y ago
Often people complain how shells in Unix/POSIX communicate via strings and praise PowerShell for communication with objects. Yet stringification still is the most simplistic means for communication between multiple domains as shown in this article.
- cesarb 3y ago
  In this case, the relevant aspect was not stringification, but serialization. It would have worked just as fine had they used a binary encoding like CBOR instead of JSON.
- jwmullally 3y ago
  Yes, its interesting to compare the de-facto standards around input and output parsing of CLI tools in Unix land. Input has been a solved problem since the 70s for simple use cases, it's output that was always a pain. Inputs are arbitrary string STDIN and tokenized command line args (argv/argv). Outputs are arbitrary string STDOUT/STDERR and well-formed simple uint8_t return codes. For input, most tools use the command line args for specifying tunable parameters. These are easy to adjust on the shell or in shell scripts, as opposed to deserializing & modifying & reserializing input files or STDIN. Most use the getopt convention (`myprog --input myfile -b 123`), which makes them well-formed enough to be a stable API and describable by simple grammars. As they are (usually) position independent key-value pairs (with some value arrays for multiple files etc), it makes it easy to add options without breaking existing callers. Look at the mountain of shell scripts out there that continue to run even as tools are updated. For output, nothing similar exists. It would be interesting if UNIX tools could also output argc/argv and shells had builtin functions to easily parse (getopt style) and index into those. Or even just have a flat key-value envvar style return list. (I guess you could have a convention of the last line of STDOUT being dedicated to that and piping it into some tool that sets a bunch of prefixed env vars). Would have made everyday output parsing from random tools a whole lot easier instead of grep/sed/awk and the dealing with changing output as people updated the tools ("ifconfig" versus "ip" well-formed grammar). Arrays and nested data structures are where everything gets complicated, and you need something like Powershell (powerful but clunky IMHO) or JSON and full programming languages. jshn is cool but verbose as its necessarily just a POSIX shell "extension".
uhtred 3y ago
Coming soon: How Turborepo is porting from Rust to Zig/whatever hottest language is to have on resume next year These blog posts make me laugh. Pretty sure they could achieve whatever they are doing with any language, heck even PHP would probably work. Just not cool though is it.
- Georgelemental 3y ago
  They explain their motivation here: https://vercel.com/blog/turborepo-migration-go-rust https://vercel.com/blog/turborepo-migration-go-rust
  - uhtred 3y ago
    [flagged]
    - brodouevencode 3y ago
      Were you not satisfied with the justification? If not why not?
      - peppermint_gum 3y ago
        That article did little to convince me they aren't just chasing the latest trends. I mean, the only concrete example in that post is that Go's stdlib function for setting file permissions is bad. It's such a minor thing, it's ridiculous.
        robot_no_421 3y ago
        Article was not written to convince skeptical contrarians, it was written to explain their decisions. I doubt they care that you aren't convinced of their choices when you have no skin in their game. If you read their article clearly laying our their reasons for their decision to migrate to Rust and your take away was "I disagree with their reasons, they really did it for a different reason", idk what to tell you. You're basically turning a statement they made into an argument they aren't making.
horeszko 3y ago
Their previous blog post (linked in the first sentence) about why the switch from Go to Rust is insightful. I'm currently learning Go but have had my eye on Rust as well and have been wondering about the strengths and weaknesses of each language and how they compare and where each language is best applied. The linked blog confirms my first impression that Go is well suited for networking applications and simplicity whereas Rust is well suited for OS/low-level applications. Does anyone here have any experience using Go and Rust? What are your thoughts on the two languages?
- mcqueenjordan 3y ago
  Yes, experience in both. I'm considerably more inclined to use Rust for any use case, but I'll freely admit that I'm biased because it appeals to my sensibilities more. IMO the most compelling use case to use Go over Rust is if you're writing a networked service or networking code.
  - sophacles 3y ago
    Having used both Go and Rust for networked and networking code, I'd say it's a pretty nuanced set of tradeoffs between the two. I've written a couple protocol handlers in rust (that is parse the packet and handle the logic before giving the data to some other bit of code) and I found Rust really nice for that - real enums, pattern matching, and the use of traits can make interacting with binary protocols (particularly of the state machine variety) really nice with ergonomic interfaces. Meanwhile go's net/IP type (etc) often leave me frustrated. On the other hand, goroutines and channels are such a nice way to handle a lot of message passing around a server application that I find myself reaching for mpsc channels and using tokio tasks like clunkier goroutines in rust often. I think I'd draw the line more along "how often I need to work with byte arrays that I want to turn into semantic types" - if I'm going down to the protocol level and not doing a ton of high level logic I'll reach for rust, if it's more high level handling of requests/responses (http for example) I'll often reach for go. (like the parent, I'm more inclined to reach for Rust over go if all else is equal, but I enjoy go too).
    - 3y ago
bluejekyll 3y ago
> By serializing to JSON, a format with robust support in both Go and Rust, we were able to minimize our FFI surface area, and avoid a whole class of cross-platform, cross-language bugs. There’s been a lot of discussion of the marshaling and unmarshaling costs of Json (and really any other text format). Should we be looking more strongly at binary formats, and if those have better outcomes than Json? I’m guessing they didn’t want to introduce more change to the Go code than necessary, but I’m wondering if people have any horror stories to share about any of the binary formats available?
- wrs 3y ago
  It depends on what you’re marshaling, and how much. Anyway, in my experience, the biggest practical disadvantage of JSON is not performance, it’s that people tend not to put a schema validation/migration layer on top of it, which leads over time to chaos. Many of the popular binary formats have schema validation built in.
  - throwawaymaths 3y ago
    Totally insane that people don't put validation since jsonschema is quite nice
  - packetlost 3y ago
    Text formats have innate issues when it comes to deserialization: you don't know the size of each field of your data until you scan the entire field. This makes deserializers generally more prone to DOS attacks. Many binary formats must necessarily store the size of fields as metadata, which means you might not need to deserialize the whole message to interpret parts of it. Further, JSON doesn't support streaming natively. These are mostly unique use-cases with marginal improvements, but it can matter.
- importantbrian 3y ago
  Isn't this the point of things like protobuf and thrift? It seems like everyone beyond a certain scale sends data around in some binary format rather than JSON.
progbits 3y ago
> since you can't statically link glibc due to licensing issues Sure I can. Maybe YOU can't, due to the license you have chosen, but just say so.
- bunga-bunga 3y ago
  That's pedantic. That's exactly what they wrote: due to licensing issues. "You" here is a generic pronoun, not "you, reader"
  - masklinn 3y ago
    Not only that, but beyond the licensing issues it's really not supported by glibc itself because glibc will dlopen NSS and iconv, which link to libc, which leads to glibc being dynamically linked anyway except now you have two copies of it, which can play havoc on shared resources (like standard streams). It's an issue from a licensing point of view, but it's also a rather poor idea from a technical point of view.
    - mike_hock 3y ago
      > but it's also a rather poor idea from a technical point of view on glibc's part.
pier25 3y ago
Why is Vercel investing so much into their own thing instead of a tool like EsBuild that already has widespread usage with Vite?
- reducesuffering 3y ago
  Isn't this because the vast majority of existing codebases and know-how are in webpack? They're trying to supplant webpack with a simple --turbo and keep existing momentum.
  - pier25 3y ago
    AFAIK the JS ecosystem is moving onto Vite which uses esbuild and Rollup under the hood.
    - reducesuffering 3y ago
      Moving on because webpack is bad and turbo repo is alpha, so yes Vite is the best of the bunch right now. But you asked what they were trying to do. Most codebases are still webpack, and only some new ones are using Vite. Significantly upgrading to a drop in webpack replacement, with performance and usability of Vite is more ideal, and that’s what they’re trying to do. Just migrate to Vite is like a version upgrade without backwards compatibility.
throwasdfjwe7 3y ago
I recommend using buildpacks.io or gcr.io/distroless or scratch or others of the ilk. Using alpine for achieving slim containers is a waste of time and frankly stupid.
jarjoura 3y ago
Can someone help me understand if I missed something here. Did they negate all the hypothetical performance gains made from Rust by crippling it with JSON IPC? If so, the mental overhead of Rust memory management doesn't seem worth it.
- masklinn 3y ago
  1. increasing performances is not one of the reasons they give for switching, although 2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow Also JSON is used when delegating to Go because the Rust binary does not handle the command anyway.
  - znkr 3y ago
    > 2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow It’s slow compared to a function call in C, but it’s still measured in nanoseconds. If you’re not calling C functions from a tight loop in Go it’s unlikely to be noticeable.
polyrand 3y ago
One detail I enjoy from this post is that sometimes you can just call a CLI[0]. It's easy to spend a lot of time figuring out how to expose some Rust/C code as a library for your language, but I like the simplicity of just compiling, shipping the binary and then calling it as a subprocess. Yes, there's overhead in starting a new process to "just call a function", but I think this approach is still underutilized. [0]: https://github.com/vercel/turbo/blob/c0ee0dea7388d1081512c934ea39a9b78137a1a3/crates/turborepo/src/main.rs#L14 https://github.com/vercel/turbo/blob/c0ee0dea7388d1081512c93...
- phamilton 3y ago
  I have a love/hate relationship with AWS's Kinesis Client Library. In one of our implementations, we use the `amazon_kcl_helper.py` script (python) to download and configure the KCL Multilang Daemon (a java application) which calls out to our application (a golang binary) via STDIN/STDOUT. And honestly, it's fine. The Multilang Daemon manages shard ownership, spins up one binary per shard, fetches records, passes them to the binary, updates checkpoints, etc. Our binary just focuses on our actual application logic. And the python script does a good job of setting everything up. It just feels wild to involve all three.

116 points by hardwaregeek 3y ago

This was referenced, which I thought was interesting: https://github.com/golang/go/issues/13492 https://github.com/golang/go/issues/13492 golang cannot be compiled as a C static library with musl.

deleted 3y ago
[deleted]

phamilton 3y ago

Ah, another tale that includes getting bit by static linking and alpine. Maybe I've just had a bad sample size, but I just haven't experienced a big enough win by using alpine to justify the weird scenarios that come up on occasion. The idea is nice. A small, stripped down container that will load quickly and have very few maintenance issues (due to basically zero dependencies). But my debian-slim images work well enough and when I do hit a problem there's more community around it and it's more straightforward to fix.

deusum 3y ago
People keep saying they need a light and fast distro - no systemd, no glibc - but it breaks a lot of other software. So, you need to be prepared to patch a lot of stuff, and build packages from source. Personally, I would love more of them to consider FreeBSD. It's got all of those features, a linux compatibility layer if necessary, a more permissive license, etc. I'd just love to see a lot more developers helping over there.
- packetlost 3y ago
  Except you can't run FreeBSD in Kubernetes really. Or really anywhere besides in VMs in the Linux monoculture.
- richardwhiuk 3y ago
  Moving to FreeBSD would require patching even more things.
- ASalazarMX 3y ago
  > a more permissive license I see this as a net disadvantage. The great insight of the GPL licences is forcing changes to be contributed back to the project, companies can't easily privatize a public effort.
  - szundi 3y ago
    And this makes GPL based software like Linux stronger, more popular, more community to help
dontlaugh 3y ago
It's also much easier to set up debugging tools when necessary.
klooney 3y ago
Glibc is very good. They should get plaudits somewhere by someone occasionally.
adameasterling 3y ago
Alpine is a trap! For a Python product, we tried to make it work for about a year before throwing our hands up and switching to a plain Ubuntu image. We’ve had no regrets.
returningfory2 3y ago
Small Docker image or small binary size is this decade's canonical premature optimization.
- didntcheck 3y ago
  Especially base images. The size of your own layers might be worth worrying about, but not the shared base that should only be stored once for arbitrarily many containers
rollcat 3y ago
I am concerned about the Linux/glibc monoculture, and its effect on the broader ecosystem. Whatever breaks on Linux/musl, is also likely to break on other operating systems. Whatever breaks loudly elsewhere, is also quite likely silently broken on Linux/glibc. If our default response is to double down on barely patching it enough to limp along, no wonder it keeps breaking.
- onlypositive 3y ago
  Most the issues with alpine Linux people face are down to package maintainers breaking backwards compatibility constantly. Package renames are a regular problem.
  - singron 3y ago
    One thing I love about NixOS so far is that when they do a rename, they can print a deprecation warning every time you use the old name while evaluating your config. When I get a moment, I go through and fix them. You could probably do something similar in a traditional distro by creating an alias package that prints a warning on install, but then you actually have to watch the install logs to see that.
  - throw16180339 3y ago
    Alpine has at least the following deficiencies: DNS TCP resolution doesn't work (https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-on-alpine-linux https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-o...). It's a feature that it doesn't retry truncated lookups or obey the relevant RFCs (https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296). Alpine has a 128k stack size (https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/ https://ariadne.space/2021/06/25/understanding-thread-stack-...), while most other OSes have at least a 512k stack.
    - rollcat 3y ago
      Just to be clear, I'm not blindly defending musl/Alpine, rather noting the detrimental effects of monoculture. Quite often fixing a build or a package on another system or architecture is a matter of a 3-5 line diff, and I've contributed quite a few over the years. It usually boils down to an incorrect assumption by the author/maintainer, stuff like "#ifdef __linux__" (when what you actually mean is: "any UNIX-like system with X11"), or hardcoding CC=gcc (where CC=cc just works).
auspex 3y ago
Alpine is also based on MUSCL rather than glibc which generally has fewer well known CVEs which helps from a security standpoint.
- danpalmer 3y ago
  Is that because it's got better security, or fewer eyes on it?
  - mike_hock 3y ago
    Both.

yndoendo 3y ago

Often people complain how shells in Unix/POSIX communicate via strings and praise PowerShell for communication with objects. Yet stringification still is the most simplistic means for communication between multiple domains as shown in this article.

cesarb 3y ago
In this case, the relevant aspect was not stringification, but serialization. It would have worked just as fine had they used a binary encoding like CBOR instead of JSON.
jwmullally 3y ago
Yes, its interesting to compare the de-facto standards around input and output parsing of CLI tools in Unix land. Input has been a solved problem since the 70s for simple use cases, it's output that was always a pain. Inputs are arbitrary string STDIN and tokenized command line args (argv/argv). Outputs are arbitrary string STDOUT/STDERR and well-formed simple uint8_t return codes. For input, most tools use the command line args for specifying tunable parameters. These are easy to adjust on the shell or in shell scripts, as opposed to deserializing & modifying & reserializing input files or STDIN. Most use the getopt convention (`myprog --input myfile -b 123`), which makes them well-formed enough to be a stable API and describable by simple grammars. As they are (usually) position independent key-value pairs (with some value arrays for multiple files etc), it makes it easy to add options without breaking existing callers. Look at the mountain of shell scripts out there that continue to run even as tools are updated. For output, nothing similar exists. It would be interesting if UNIX tools could also output argc/argv and shells had builtin functions to easily parse (getopt style) and index into those. Or even just have a flat key-value envvar style return list. (I guess you could have a convention of the last line of STDOUT being dedicated to that and piping it into some tool that sets a bunch of prefixed env vars). Would have made everyday output parsing from random tools a whole lot easier instead of grep/sed/awk and the dealing with changing output as people updated the tools ("ifconfig" versus "ip" well-formed grammar). Arrays and nested data structures are where everything gets complicated, and you need something like Powershell (powerful but clunky IMHO) or JSON and full programming languages. jshn is cool but verbose as its necessarily just a POSIX shell "extension".

uhtred 3y ago

Coming soon: How Turborepo is porting from Rust to Zig/whatever hottest language is to have on resume next year These blog posts make me laugh. Pretty sure they could achieve whatever they are doing with any language, heck even PHP would probably work. Just not cool though is it.

Georgelemental 3y ago
They explain their motivation here: https://vercel.com/blog/turborepo-migration-go-rust https://vercel.com/blog/turborepo-migration-go-rust
- uhtred 3y ago
  [flagged]
  - brodouevencode 3y ago
    Were you not satisfied with the justification? If not why not?
    - peppermint_gum 3y ago
      That article did little to convince me they aren't just chasing the latest trends. I mean, the only concrete example in that post is that Go's stdlib function for setting file permissions is bad. It's such a minor thing, it's ridiculous.
      - robot_no_421 3y ago
        Article was not written to convince skeptical contrarians, it was written to explain their decisions. I doubt they care that you aren't convinced of their choices when you have no skin in their game. If you read their article clearly laying our their reasons for their decision to migrate to Rust and your take away was "I disagree with their reasons, they really did it for a different reason", idk what to tell you. You're basically turning a statement they made into an argument they aren't making.

horeszko 3y ago

Their previous blog post (linked in the first sentence) about why the switch from Go to Rust is insightful. I'm currently learning Go but have had my eye on Rust as well and have been wondering about the strengths and weaknesses of each language and how they compare and where each language is best applied. The linked blog confirms my first impression that Go is well suited for networking applications and simplicity whereas Rust is well suited for OS/low-level applications. Does anyone here have any experience using Go and Rust? What are your thoughts on the two languages?

mcqueenjordan 3y ago
Yes, experience in both. I'm considerably more inclined to use Rust for any use case, but I'll freely admit that I'm biased because it appeals to my sensibilities more. IMO the most compelling use case to use Go over Rust is if you're writing a networked service or networking code.
- sophacles 3y ago
  Having used both Go and Rust for networked and networking code, I'd say it's a pretty nuanced set of tradeoffs between the two. I've written a couple protocol handlers in rust (that is parse the packet and handle the logic before giving the data to some other bit of code) and I found Rust really nice for that - real enums, pattern matching, and the use of traits can make interacting with binary protocols (particularly of the state machine variety) really nice with ergonomic interfaces. Meanwhile go's net/IP type (etc) often leave me frustrated. On the other hand, goroutines and channels are such a nice way to handle a lot of message passing around a server application that I find myself reaching for mpsc channels and using tokio tasks like clunkier goroutines in rust often. I think I'd draw the line more along "how often I need to work with byte arrays that I want to turn into semantic types" - if I'm going down to the protocol level and not doing a ton of high level logic I'll reach for rust, if it's more high level handling of requests/responses (http for example) I'll often reach for go. (like the parent, I'm more inclined to reach for Rust over go if all else is equal, but I enjoy go too).
  - 3y ago

bluejekyll 3y ago

> By serializing to JSON, a format with robust support in both Go and Rust, we were able to minimize our FFI surface area, and avoid a whole class of cross-platform, cross-language bugs. There’s been a lot of discussion of the marshaling and unmarshaling costs of Json (and really any other text format). Should we be looking more strongly at binary formats, and if those have better outcomes than Json? I’m guessing they didn’t want to introduce more change to the Go code than necessary, but I’m wondering if people have any horror stories to share about any of the binary formats available?

wrs 3y ago
It depends on what you’re marshaling, and how much. Anyway, in my experience, the biggest practical disadvantage of JSON is not performance, it’s that people tend not to put a schema validation/migration layer on top of it, which leads over time to chaos. Many of the popular binary formats have schema validation built in.
- throwawaymaths 3y ago
  Totally insane that people don't put validation since jsonschema is quite nice
- packetlost 3y ago
  Text formats have innate issues when it comes to deserialization: you don't know the size of each field of your data until you scan the entire field. This makes deserializers generally more prone to DOS attacks. Many binary formats must necessarily store the size of fields as metadata, which means you might not need to deserialize the whole message to interpret parts of it. Further, JSON doesn't support streaming natively. These are mostly unique use-cases with marginal improvements, but it can matter.
importantbrian 3y ago
Isn't this the point of things like protobuf and thrift? It seems like everyone beyond a certain scale sends data around in some binary format rather than JSON.

progbits 3y ago

> since you can't statically link glibc due to licensing issues Sure I can. Maybe YOU can't, due to the license you have chosen, but just say so.

bunga-bunga 3y ago
That's pedantic. That's exactly what they wrote: due to licensing issues. "You" here is a generic pronoun, not "you, reader"
- masklinn 3y ago
  Not only that, but beyond the licensing issues it's really not supported by glibc itself because glibc will dlopen NSS and iconv, which link to libc, which leads to glibc being dynamically linked anyway except now you have two copies of it, which can play havoc on shared resources (like standard streams). It's an issue from a licensing point of view, but it's also a rather poor idea from a technical point of view.
  - mike_hock 3y ago
    > but it's also a rather poor idea from a technical point of view on glibc's part.

pier25 3y ago

Why is Vercel investing so much into their own thing instead of a tool like EsBuild that already has widespread usage with Vite?

reducesuffering 3y ago
Isn't this because the vast majority of existing codebases and know-how are in webpack? They're trying to supplant webpack with a simple --turbo and keep existing momentum.
- pier25 3y ago
  AFAIK the JS ecosystem is moving onto Vite which uses esbuild and Rollup under the hood.
  - reducesuffering 3y ago
    Moving on because webpack is bad and turbo repo is alpha, so yes Vite is the best of the bunch right now. But you asked what they were trying to do. Most codebases are still webpack, and only some new ones are using Vite. Significantly upgrading to a drop in webpack replacement, with performance and usability of Vite is more ideal, and that’s what they’re trying to do. Just migrate to Vite is like a version upgrade without backwards compatibility.

throwasdfjwe7 3y ago

I recommend using buildpacks.io or gcr.io/distroless or scratch or others of the ilk. Using alpine for achieving slim containers is a waste of time and frankly stupid.

jarjoura 3y ago

Can someone help me understand if I missed something here. Did they negate all the hypothetical performance gains made from Rust by crippling it with JSON IPC? If so, the mental overhead of Rust memory management doesn't seem worth it.

masklinn 3y ago
1. increasing performances is not one of the reasons they give for switching, although 2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow Also JSON is used when delegating to Go because the Rust binary does not handle the command anyway.
- znkr 3y ago
  > 2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow It’s slow compared to a function call in C, but it’s still measured in nanoseconds. If you’re not calling C functions from a tight loop in Go it’s unlikely to be noticeable.

polyrand 3y ago

One detail I enjoy from this post is that sometimes you can just call a CLI[0]. It's easy to spend a lot of time figuring out how to expose some Rust/C code as a library for your language, but I like the simplicity of just compiling, shipping the binary and then calling it as a subprocess. Yes, there's overhead in starting a new process to "just call a function", but I think this approach is still underutilized. [0]: https://github.com/vercel/turbo/blob/c0ee0dea7388d1081512c934ea39a9b78137a1a3/crates/turborepo/src/main.rs#L14 https://github.com/vercel/turbo/blob/c0ee0dea7388d1081512c93...

phamilton 3y ago
I have a love/hate relationship with AWS's Kinesis Client Library. In one of our implementations, we use the `amazon_kcl_helper.py` script (python) to download and configure the KCL Multilang Daemon (a java application) which calls out to our application (a golang binary) via STDIN/STDOUT. And honestly, it's fine. The Multilang Daemon manages shard ownership, spins up one binary per shard, fetches records, passes them to the binary, updates checkpoints, etc. Our binary just focuses on our actual application logic. And the python script does a good job of setting everything up. It just feels wild to involve all three.