5 ms·

Well done and congrats to the Turborepo team on the launch as well as the Vercel merger, which I think is a great thing for the JS ecosystem! We now have a hea

Well done and congrats to the Turborepo team on the launch as well as the Vercel merger, which I think is a great thing for the JS ecosystem! We now have a healthy competition between several JS domain-specific build tools bit.dev, Nx, Turborepo, and Rush. This is in addition to plugins to general purpose monorepo tooling like rules_nodejs (Bazel). I'm looking forward to the seeing the new ideas that come out from the community as this field matures. However, I feel a bit sad at the state of general purpose build tools (Bazel/Pants/Buck, make, GitHub Actions, Maven, Nix) or cloud-based IDEs (Cloud9, Codespaces). These tools come off as too complex to operate and build on top of such that we, the JS community, seem to be choosing to build JS-specific tooling from scratch instead. There are a huge number of mostly non-JS-specific problems that monorepo tooling eventually needs to solve: distributed build artifact and test result caching, distributed action execution, sandboxing, resource management and queuing, observability, and integration with other CI tools to name a few. I wish somehow we could reorganize around a smaller set of primitives instead of what appears to be reinventing the wheel. Regardless, I think all of this effort and attention has lent credence to the monorepo thesis, and I'm very excited to see what's next.

palmdeezy 4y ago
> There are a huge number of mostly non-JS-specific problems that monorepo tooling eventually needs to solve: distributed build artifact and test result caching, distributed action execution, sandboxing, resource management and queuing, observability, and integration with other CI tools to name a few. Turborepo author/founder here.... I agree. I built Turborepo because existing tools weren’t meeting our needs. To solve these problems and still be flexible, many existing build tools end up with lots of configuration bloat. We’re trying to avoid that. We want to reimagine the developer experience of monorepo tooling and make it accessible for everyone.
- dfabulich 4y ago
  Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular? (Is Bazel designed in a way that make it impossible to do JS monorepos well?)
  - palmdeezy 4y ago
    Turborepo author/founder here.... Yes, it's possible. Databricks currently uses both Turborepo and Bazel together in their monorepo[1] [1] https://twitter.com/elado/status/1504216742393876483?s=20&t=GWrtgN2F5_E7v_0jwAWYbw https://twitter.com/elado/status/1504216742393876483?s=20&t=...
  - qbasic_forever 4y ago
    > Is Bazel designed in a way that make it impossible to do JS monorepos well? Not impossible, but you really need to go all in with it and follow its conventions and practices. See this for the main docs: https://github.com/bazelbuild/rules_nodejs https://github.com/bazelbuild/rules_nodejs One thing in particular that doesn't work well in the bazel world is doing your own stuff outside its BUILD.bazel files. If you're used to just npm install and jam some code in your package.json scripts... that doesn't usually work in the bazel world. If you have a lot of logic or tools in your build you'll likely need to go all in and make bazel starlark rules or macros that recreate that logic. Nothing is impossible, but expect to spend time getting up to speed and getting things working the bazel way.
  - lhorie 4y ago
    Yes, it's possible to use Bazel w/ JS tooling. At Uber we run a 1000+ package monorepo w/ Bazel and Yarn. Someone else mentioned rules_nodejs if you want to go with the popular option that is more or less in line with the Bazel philosophy[0]. We use a project called jazelle[1] that makes some different trade-offs that lets us do some interesting things like not busting the whole world's cache when lockfile changes, and source code generation (including automatic syncing of BUILD files to package.json) [0] https://news.ycombinator.com/item?id=30959893 https://news.ycombinator.com/item?id=30959893 [1] https://github.com/uber-web/jazelle https://github.com/uber-web/jazelle
  - steven-xu 4y ago
    > Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular? It's definitely possible, but I think the practical limitations would make it too complex to reason around and maintain. You'd end up creating two separate and overlapping systems to declare dependency graphs and input sources and manage caching and execution. I haven't yet seen a case where the two are actually interleaved. Currently at Databricks, we use Bazel to provide the correctness guarantees and interop needed for CI, and we use JS-specific tooling (non-Bazel) locally to meet our performance needs, where the usage profile is different and where we're willing to make correctness tradeoffs. > (Is Bazel designed in a way that make it impossible to do JS monorepos well?) There are limitations in Bazel that don't play nicely with modern JS conventions. For example, Bazel's standard sandbox is based on symlink farms, and Node.js and the ecosystem by default follow symlinks[1] to their real locations, effectively breaking sandboxing. A FUSE or custom filesystem (Google's version of Bazel takes advantage of one[2]) would be better but is not as portable. As another example, Bazel's action cache tries to watch or otherwise verify the integrity of every input file to an action, and when node_modules is 100k+ files, this gets expensive and is prone to non-determinism. Bazel does this for correctness, which is noble but results in practical performance problems. You need to do extra work to "trick" Bazel into not reading these 100k+ files each time. The problems feel solvable to me, but not easily without adding yet more configuration options to Bazel. The influx of new JS-specific tooling is a reset to this, building the minimum viable set of functionality that the JS ecosystem specifically needs, without the burdens of being a general purpose build system. [1] https://nodejs.org/api/cli.html#--preserve-symlinks https://nodejs.org/api/cli.html#--preserve-symlinks [2] https://dl.acm.org/doi/10.1145/2854146 https://dl.acm.org/doi/10.1145/2854146
- pleasegod123 4y ago
  Hey. Off topic but would you consider handing Formik and TSDX to a community member? Huge appreciation for turborepo and your work on these other libraries -- they have become key parts of the ecosystem and turborepo could follow. However, this is all the more reason why it could be super helpful to address the governance issue on those projects. Thanks and sorry for disturbing!
qbasic_forever 4y ago
I'm sad JS needs a build step. The best build system is none at all IMHO. I'd love to see native support everywhere for typescript or other things we typically depend on a build for today.
- krapp 4y ago
  >I'm sad JS needs a build step. The best build system is none at all IMHO. Good news: JS doesn't need a build step. Modern webdev wants a build step mostly because it wants Javascript to feel more like a "serious" language. There are technical benefits to compiling to Javascript, most of which can be served by other means, but the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill. >I'd love to see native support everywhere for typescript or other things we typically depend on a build for today. Typescript is part of the problem. You can literally just accept Javascript for what it is - dynamically typed - and write it like any other scripting language.
  - qbasic_forever 4y ago
    I'm with you in that a lot of TS use seems dogmatic or ritualistic today. I have a strong feeling in the near future we're going to see the bow string snap back and simple zero build, basic dynamic use of pure JS comes back in vogue.
    - krapp 4y ago
      It seems to be coming back as 'vanilla JS' so maybe there's hope.
    - ceeplusplus 4y ago
      TS is about readability and maintainability of code at scale. Having types helps immensely in understanding a new codebase and working in a large codebase with many other people.
      - krapp 4y ago
hyperupcall 4y ago
> However, I feel a bit sad at the state of general purpose build tools (Bazel/Pants/Buck, make, GitHub Actions, Maven, Nix) or cloud-based IDEs (Cloud9, Codespaces). These tools come off as too complex to operate and build on top of... I definitely agree, although I've found Please (please.build) to potentially be a solution in this area. It is a lot simpler, smaller, and more modern than Buck and Bazel, but shares similar properties as to be familiar (I.e. the buildfile syntax). I think it is supposed to be easier to extend with other languages, but I haven't tries that myself.
andrew_ 4y ago
bare-metal pnpm (what Rush is built upon) has never let me down. Handles rather large monorepos without issue.
darepublic 4y ago
I like github actions better than learning a new monorepo framework

palmdeezy 4y ago

> There are a huge number of mostly non-JS-specific problems that monorepo tooling eventually needs to solve: distributed build artifact and test result caching, distributed action execution, sandboxing, resource management and queuing, observability, and integration with other CI tools to name a few. Turborepo author/founder here.... I agree. I built Turborepo because existing tools weren’t meeting our needs. To solve these problems and still be flexible, many existing build tools end up with lots of configuration bloat. We’re trying to avoid that. We want to reimagine the developer experience of monorepo tooling and make it accessible for everyone.

dfabulich 4y ago
Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular? (Is Bazel designed in a way that make it impossible to do JS monorepos well?)
- palmdeezy 4y ago
  Turborepo author/founder here.... Yes, it's possible. Databricks currently uses both Turborepo and Bazel together in their monorepo[1] [1] https://twitter.com/elado/status/1504216742393876483?s=20&t=GWrtgN2F5_E7v_0jwAWYbw https://twitter.com/elado/status/1504216742393876483?s=20&t=...
- qbasic_forever 4y ago
  > Is Bazel designed in a way that make it impossible to do JS monorepos well? Not impossible, but you really need to go all in with it and follow its conventions and practices. See this for the main docs: https://github.com/bazelbuild/rules_nodejs https://github.com/bazelbuild/rules_nodejs One thing in particular that doesn't work well in the bazel world is doing your own stuff outside its BUILD.bazel files. If you're used to just npm install and jam some code in your package.json scripts... that doesn't usually work in the bazel world. If you have a lot of logic or tools in your build you'll likely need to go all in and make bazel starlark rules or macros that recreate that logic. Nothing is impossible, but expect to spend time getting up to speed and getting things working the bazel way.
- lhorie 4y ago
  Yes, it's possible to use Bazel w/ JS tooling. At Uber we run a 1000+ package monorepo w/ Bazel and Yarn. Someone else mentioned rules_nodejs if you want to go with the popular option that is more or less in line with the Bazel philosophy[0]. We use a project called jazelle[1] that makes some different trade-offs that lets us do some interesting things like not busting the whole world's cache when lockfile changes, and source code generation (including automatic syncing of BUILD files to package.json) [0] https://news.ycombinator.com/item?id=30959893 https://news.ycombinator.com/item?id=30959893 [1] https://github.com/uber-web/jazelle https://github.com/uber-web/jazelle
- steven-xu 4y ago
  > Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular? It's definitely possible, but I think the practical limitations would make it too complex to reason around and maintain. You'd end up creating two separate and overlapping systems to declare dependency graphs and input sources and manage caching and execution. I haven't yet seen a case where the two are actually interleaved. Currently at Databricks, we use Bazel to provide the correctness guarantees and interop needed for CI, and we use JS-specific tooling (non-Bazel) locally to meet our performance needs, where the usage profile is different and where we're willing to make correctness tradeoffs. > (Is Bazel designed in a way that make it impossible to do JS monorepos well?) There are limitations in Bazel that don't play nicely with modern JS conventions. For example, Bazel's standard sandbox is based on symlink farms, and Node.js and the ecosystem by default follow symlinks[1] to their real locations, effectively breaking sandboxing. A FUSE or custom filesystem (Google's version of Bazel takes advantage of one[2]) would be better but is not as portable. As another example, Bazel's action cache tries to watch or otherwise verify the integrity of every input file to an action, and when node_modules is 100k+ files, this gets expensive and is prone to non-determinism. Bazel does this for correctness, which is noble but results in practical performance problems. You need to do extra work to "trick" Bazel into not reading these 100k+ files each time. The problems feel solvable to me, but not easily without adding yet more configuration options to Bazel. The influx of new JS-specific tooling is a reset to this, building the minimum viable set of functionality that the JS ecosystem specifically needs, without the burdens of being a general purpose build system. [1] https://nodejs.org/api/cli.html#--preserve-symlinks https://nodejs.org/api/cli.html#--preserve-symlinks [2] https://dl.acm.org/doi/10.1145/2854146 https://dl.acm.org/doi/10.1145/2854146
pleasegod123 4y ago
Hey. Off topic but would you consider handing Formik and TSDX to a community member? Huge appreciation for turborepo and your work on these other libraries -- they have become key parts of the ecosystem and turborepo could follow. However, this is all the more reason why it could be super helpful to address the governance issue on those projects. Thanks and sorry for disturbing!

qbasic_forever 4y ago

I'm sad JS needs a build step. The best build system is none at all IMHO. I'd love to see native support everywhere for typescript or other things we typically depend on a build for today.

krapp 4y ago
>I'm sad JS needs a build step. The best build system is none at all IMHO. Good news: JS doesn't need a build step. Modern webdev wants a build step mostly because it wants Javascript to feel more like a "serious" language. There are technical benefits to compiling to Javascript, most of which can be served by other means, but the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill. >I'd love to see native support everywhere for typescript or other things we typically depend on a build for today. Typescript is part of the problem. You can literally just accept Javascript for what it is - dynamically typed - and write it like any other scripting language.
- qbasic_forever 4y ago
  I'm with you in that a lot of TS use seems dogmatic or ritualistic today. I have a strong feeling in the near future we're going to see the bow string snap back and simple zero build, basic dynamic use of pure JS comes back in vogue.
  - krapp 4y ago
    It seems to be coming back as 'vanilla JS' so maybe there's hope.
  - ceeplusplus 4y ago
    TS is about readability and maintainability of code at scale. Having types helps immensely in understanding a new codebase and working in a large codebase with many other people.
    - krapp 4y ago

hyperupcall 4y ago

> However, I feel a bit sad at the state of general purpose build tools (Bazel/Pants/Buck, make, GitHub Actions, Maven, Nix) or cloud-based IDEs (Cloud9, Codespaces). These tools come off as too complex to operate and build on top of... I definitely agree, although I've found Please (please.build) to potentially be a solution in this area. It is a lot simpler, smaller, and more modern than Buck and Bazel, but shares similar properties as to be familiar (I.e. the buildfile syntax). I think it is supposed to be easier to extend with other languages, but I haven't tries that myself.

andrew_ 4y ago

bare-metal pnpm (what Rush is built upon) has never let me down. Handles rather large monorepos without issue.

darepublic 4y ago

I like github actions better than learning a new monorepo framework