8 ms·

Building TypeScript Projects with Bazel

90 points by mgechev 8y ago

habitue 8y ago
I've used bazel pretty extensively and I have to say it's not fully baked unless you're using it the way Google does internally (vendor all dependencies) and you're using a language Google does internally. Additionally, it wants to take over your entire build process end to end. Get used to prefixing every shell command with `bazel run` or `bazel build` because everything goes into a little sandbox and bazel likes to be in the middle of everything. These tradeoffs make sense at Google scale (the goal of bazel is to have very repeatable builds), but in my opinion it's a huge drag with not much benefit for smaller shops (to be clear, "Oh you only have 200 developers?" Still a small shop by these standards)
- tonetheman 8y ago
  I am a minority I feel but everywhere I have ever worked I have always pushed to keep your dependencies close. There was never a way to reproduce a build without doing so. I know other smart people want badly to trust in outside sources ... but I am old and jaded. And never skipped a build because npm was down... or before that if I could get a jar from a remote source ... and so on.
  - lstamour 8y ago
    Bazel encourages you to list geographically diverse sources to download from for workspace setup. That said, you could run a local proxy with something like Squid or Varnish, turn on caching in a remote build server, or set up a mono-repo and never use npm (as Google does), each has advantages and tradeoffs... you could also archive and backup your Bazel folders to hermetically reproduce workspaces and builds if patching is necessary, though you’d have to also archive a copy of your host or build environment if you’re paranoid about host tools or configurations changing.
  - habitue 8y ago
    Vendoring is totally a thing you can do, but keeping them up to date with upstream (and keeping your bazel BUILD files current) is a full time job. You have to hire someone to do that work, it's not completely automatable.
    - vageli 8y ago
      Non-security upgrades probably _shouldn't_ be automated.
      - habitue 8y ago
        Absolutely agree. But I meant maintaining BUILD files for code you didn't write can be very time consuming. Even a security-only update can include new source files that you now have to tell Bazel about. It's real work
- xyzzy_plugh 8y ago
  > unless you're using a language Google does internally. If only. Go support is pretty horrific and painful.
- krona 8y ago
  Bazel, from the outset, wasn't designed for open source projects that tend to have dependencies with incompatible build systems. It isn't flexible enough.
- mark_l_watson 8y ago
  Thank you for those comments. For my own projects at home I mostly use Haskell, Python, and Java, but sometimes also Typescript, SBCL Common Lisp, Racket, Chez Scheme, and Scala. I was imagining getting set up to build everything with Bazel after reading the article but your comments are on point and made me reconsider. I used to vender locally Ruby, Merb, Rails, and most Java libraries I used - but I stopped doing that about ten years ago.
saagarjha 8y ago
Unrelated, but the GitHub logo in the corner covers a significant portion of the content for me on mobile. This isn’t too annoying because I can go into reader mode, but I thought I might mention it in the hopes that it gets fixed.
lstamour 8y ago
For more examples of how to use Bazel with TypeScript or JavaScript, have a look at the code samples I shared in this somewhat unrelated GitHub issue: https://github.com/bazelbuild/rules_typescript/issues/344 https://github.com/bazelbuild/rules_typescript/issues/344 (near the bottom I post a few more realistic ones as I experimented with getting webpack working, etc.) It’s a possibility that some of this post (the TypeScript-Rollup chaining) might have been inspired by this issue (maybe), but as I note in various places there, it’s been tricky to get these three separate projects to work together — rules_nodejs was recently updated to better work with npm with an escape hatch if your node or compiler code cannot be rewritten to allow Bazel to manage your npm modules - but while rules_typescript was rewritten to support rules_nodejs, the otherwise excellent rules_closure is not updated to support such npm modules. With few dependencies, and with caching of individual modules, your builds can go lightning fast so you don’t need the watch mode that webpack promotes so heavily. However, ibazel lets you watch your codebase for changes and rebuilds, it just doesn’t inject into your browser and host a web server for you the way webpack does. I’d say at this point Bazel deserves your experimentation, but until it supports all common JS build tools and scenarios, including linting and IDE integration (there’s some but currently it’s all maintained by groups at Google and not the IDE companies themselves the way other build tools are integrated),so it’ll be a hard sell for most companies to adopt it wholesale, unless, like Dropbox and others, you have a diverse set of programming languages and targets that have shared dependencies that, when changed, you want to trigger partial rebuilds of multiple projects. But the dependencies have to be supported by the rules you’re using and right now, there are some limits to what first-party rules support. The tools are at a state where a couple people on your team will need to learn Starlark, the Python-like language used in lower-level .bzl files (which can define rules and macros that the BUILD.bazel files that can then refer to) and especially the internals of Bazel including reading source code, a lot of it. This is because the debugger tools are there but not fully bug-free yet, they don’t provide the rich debugging experience you’ve grown to expect with other languages. It’s kind of like debugging and writing Go code early on — you knew once you found the right syntax that it would be obvious but until then it was a lot of experimentation and copying other examples. It’s obvious the teams at Google are pushing Bazel’s development hard, but the Angular team right now is most visible in trying to make things easier for third-party, non-Google development workflows using Bazel. And I’m sure the project will be one to watch in 2019.
akhilcacharya 8y ago
I always enjoy reading about build systems. I wish there was a run-down on how internal global build systems work across large companies (eg. Brazil vs. Blaze/Bazel vs. Pants vs. etc etc)
- jingwen 8y ago
  You may enjoy this article: https://ruudvanasseldonk.com/2018/09/03/build-system-insights https://ruudvanasseldonk.com/2018/09/03/build-system-insight...
solomatov 8y ago
Unrelated to typescript. I use bazel for about 1 year, and super happy about it. Builds are really fast and reproducible.
- erichocean 8y ago
  Same. Bazel is a breath of fresh air after CMake. Makes me actually want to use C++ for daily work.
  - floatboth 8y ago
    Have you tried Meson? It's also a breath of fresh air after CMake, but while Bazel is a "corporate" build system (wants to take everything over, assumes one consistent environment), Meson is a "very Unix" build system, it aims for portability, configurability and reproducibility.
    - solomatov 8y ago
      It doesn't take everything over. You can use repository rules to build dependencies in whatever way you want. Also, it's not corporate and not bloated. If you understand the system well enough, you can create support for a language yourself from scratch without spending huge amount of time on it.
netheril96 8y ago
As a Googler I am very satisfied with Blaze (the internal version of Bazel) at work. But I just don't want to manage such a complex beast for my own personal projects. Besides the large amount of dependencies to install for a build tool, it demands me to manually rewrite the build script for almost all third party libraries. At Google someone has most likely done that for me, but outside it, I have to do it myself.
- n-gatedotcom 8y ago
  Whoever doenvoted, can you please explain so I know what's wrong ? To an not-googler this seems a pretty well reasoned argument with not much that seems to be wrong-facts or stupid.
- habitue 8y ago
  This is the catch with Bazel. All of its integration with npm, pypi, crates.io, etc are hacks that essentially vendor the source of your dependencies and attempt to generate BUILD files for those deps with some heuristics. For a lot of packages the heuristics work, but not all. Ultimately bazel needs you to vendor and hand-maintain BUILD files to get a seamless experience.
n-gatedotcom 8y ago
I am not very familiar with build systems but how difficult would it be to make one that wraps the native ones whenever and provides modern features, unlike make.
logicchains 8y ago
Anyone had any luck embedding Bazel in another build system, like CMake? Tensorflow's C++ API seems to only support building with Bazel[1], which makes it painful to integrate into a standard C++ build pipeline (don't want the person maintaining the build to come after me with an axe). I'm seriously considering just rewriting to Torch for low latency deployment, which is nicely packaged in the standard manner of library + header file rather than trying to force users to incorporate a heavyweight, esoteric build system just to use the library. 1. https://www.tensorflow.org/guide/extend/cc https://www.tensorflow.org/guide/extend/cc - "The C++ API is only designed to work with TensorFlow bazel build. If you need a stand-alone option, use the C API" (and the C API page notes that it's "designed for simplicity and uniformity rather than convenience", making it unsuitable as a wholesale replacement for the C++ API).
- lstamour 8y ago
  Eventually they’ll refactor the C++ and Java hard-codes functionality into Starlark rules but until then, they don’t support exporting headers from Bazel C++ yet— https://github.com/bazelbuild/bazel/issues/5192 https://github.com/bazelbuild/bazel/issues/5192
fhrow4484 8y ago
Some features that I find elegant in Bazel, but probably don't have that much impact on small projects/teams is the whole concept of "visibility"[1]. For large projects, being able to control who can take a dependency on your code is really nice. This mixes really well with automatically adding Required reviewers in Pull Requests based on which folder gets modified [2][3]. [1] https://docs.bazel.build/versions/master/be/common-definitions.html#common-attributes https://docs.bazel.build/versions/master/be/common-definitio... [2] https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=vsts#automatically-include-code-reviewers https://docs.microsoft.com/en-us/azure/devops/repos/git/bran... [3] https://help.github.com/articles/about-codeowners/ https://help.github.com/articles/about-codeowners/
- joshuamorton 8y ago
  It's also incredibly nice for having a clean deprecation process. Make visibility private except to current blaze deps, and then deprecate or delete as normal. You can be certain no one new will depend on the old stuff.
jondubois 8y ago
I cannot believe how popular TypeScript has become. Is Microsoft running a huge ad campaign? I've used TypeScript for frontend work (Angular 2) for about 6 months. I did not renew my contract because I did not enjoy constantly fixing TypeScript configuration, versioning and source mapping issues. My new company switched to TypeScript for backend work and development across all teams has slowed down noticeably. I spend a lot of my time just discussing with colleagues what types should be called and whether or not we should merge two types together into one or whether or not it makes sense to have two different types to represent essentially the same data in order to enforce different constraints on the user of the data depending on the kinds of functionality that we want to expose. Then, as we build the project (in an agile way), we have to keep renaming and combining types ALL the time because otherwise we end up with too many different types that sound almost the same and have almost all the same properties but are slightly different semantically. Also, we spend a lot of time just thinking about what the intention of the user should be and all the ways in which we can impose contraints on them instead of correctly acknowledging that the user will always find some way to do whatever the heck they want and that all this strict typing adds no real value once you introduce the human factor. Ironically, the so called 'stupidity' which TypeScript aims to protect the code from will find even more compex, more creative and less obvious ways to express itself within that code.
- tekkk 8y ago
  What are you talking about? Lost productivity compared to what, JS? Merge types? You can just use `&` to combine them if need be? Discussing types? Does it matter that much, and I mean refactoring them isn't a big deal if need be. Two different types represent essentially the same data? You've tried extending interfaces or using generics? Seems like you lack an experienced TS developer in your team and are contributing your lack of experience as a fault of the language itself. The typing has saved my ass maany times and makes the code in our projects actually maintainable for us and for someone outside of our team's bubble. It might take a while longer to code the app, but it'll take less hours in the long run to keep it safe and working.
  - jondubois 8y ago

90 points by mgechev 8y ago

I've used bazel pretty extensively and I have to say it's not fully baked unless you're using it the way Google does internally (vendor all dependencies) and you're using a language Google does internally. Additionally, it wants to take over your entire build process end to end. Get used to prefixing every shell command with `bazel run` or `bazel build` because everything goes into a little sandbox and bazel likes to be in the middle of everything. These tradeoffs make sense at Google scale (the goal of bazel is to have very repeatable builds), but in my opinion it's a huge drag with not much benefit for smaller shops (to be clear, "Oh you only have 200 developers?" Still a small shop by these standards)

tonetheman 8y ago
I am a minority I feel but everywhere I have ever worked I have always pushed to keep your dependencies close. There was never a way to reproduce a build without doing so. I know other smart people want badly to trust in outside sources ... but I am old and jaded. And never skipped a build because npm was down... or before that if I could get a jar from a remote source ... and so on.
- lstamour 8y ago
  Bazel encourages you to list geographically diverse sources to download from for workspace setup. That said, you could run a local proxy with something like Squid or Varnish, turn on caching in a remote build server, or set up a mono-repo and never use npm (as Google does), each has advantages and tradeoffs... you could also archive and backup your Bazel folders to hermetically reproduce workspaces and builds if patching is necessary, though you’d have to also archive a copy of your host or build environment if you’re paranoid about host tools or configurations changing.
- habitue 8y ago
  Vendoring is totally a thing you can do, but keeping them up to date with upstream (and keeping your bazel BUILD files current) is a full time job. You have to hire someone to do that work, it's not completely automatable.
  - vageli 8y ago
    Non-security upgrades probably _shouldn't_ be automated.
    - habitue 8y ago
      Absolutely agree. But I meant maintaining BUILD files for code you didn't write can be very time consuming. Even a security-only update can include new source files that you now have to tell Bazel about. It's real work
xyzzy_plugh 8y ago
> unless you're using a language Google does internally. If only. Go support is pretty horrific and painful.
krona 8y ago
Bazel, from the outset, wasn't designed for open source projects that tend to have dependencies with incompatible build systems. It isn't flexible enough.
mark_l_watson 8y ago
Thank you for those comments. For my own projects at home I mostly use Haskell, Python, and Java, but sometimes also Typescript, SBCL Common Lisp, Racket, Chez Scheme, and Scala. I was imagining getting set up to build everything with Bazel after reading the article but your comments are on point and made me reconsider. I used to vender locally Ruby, Merb, Rails, and most Java libraries I used - but I stopped doing that about ten years ago.

saagarjha 8y ago

Unrelated, but the GitHub logo in the corner covers a significant portion of the content for me on mobile. This isn’t too annoying because I can go into reader mode, but I thought I might mention it in the hopes that it gets fixed.

lstamour 8y ago

For more examples of how to use Bazel with TypeScript or JavaScript, have a look at the code samples I shared in this somewhat unrelated GitHub issue: https://github.com/bazelbuild/rules_typescript/issues/344 https://github.com/bazelbuild/rules_typescript/issues/344 (near the bottom I post a few more realistic ones as I experimented with getting webpack working, etc.) It’s a possibility that some of this post (the TypeScript-Rollup chaining) might have been inspired by this issue (maybe), but as I note in various places there, it’s been tricky to get these three separate projects to work together — rules_nodejs was recently updated to better work with npm with an escape hatch if your node or compiler code cannot be rewritten to allow Bazel to manage your npm modules - but while rules_typescript was rewritten to support rules_nodejs, the otherwise excellent rules_closure is not updated to support such npm modules. With few dependencies, and with caching of individual modules, your builds can go lightning fast so you don’t need the watch mode that webpack promotes so heavily. However, ibazel lets you watch your codebase for changes and rebuilds, it just doesn’t inject into your browser and host a web server for you the way webpack does. I’d say at this point Bazel deserves your experimentation, but until it supports all common JS build tools and scenarios, including linting and IDE integration (there’s some but currently it’s all maintained by groups at Google and not the IDE companies themselves the way other build tools are integrated),so it’ll be a hard sell for most companies to adopt it wholesale, unless, like Dropbox and others, you have a diverse set of programming languages and targets that have shared dependencies that, when changed, you want to trigger partial rebuilds of multiple projects. But the dependencies have to be supported by the rules you’re using and right now, there are some limits to what first-party rules support. The tools are at a state where a couple people on your team will need to learn Starlark, the Python-like language used in lower-level .bzl files (which can define rules and macros that the BUILD.bazel files that can then refer to) and especially the internals of Bazel including reading source code, a lot of it. This is because the debugger tools are there but not fully bug-free yet, they don’t provide the rich debugging experience you’ve grown to expect with other languages. It’s kind of like debugging and writing Go code early on — you knew once you found the right syntax that it would be obvious but until then it was a lot of experimentation and copying other examples. It’s obvious the teams at Google are pushing Bazel’s development hard, but the Angular team right now is most visible in trying to make things easier for third-party, non-Google development workflows using Bazel. And I’m sure the project will be one to watch in 2019.

akhilcacharya 8y ago

I always enjoy reading about build systems. I wish there was a run-down on how internal global build systems work across large companies (eg. Brazil vs. Blaze/Bazel vs. Pants vs. etc etc)

jingwen 8y ago
You may enjoy this article: https://ruudvanasseldonk.com/2018/09/03/build-system-insights https://ruudvanasseldonk.com/2018/09/03/build-system-insight...

solomatov 8y ago

Unrelated to typescript. I use bazel for about 1 year, and super happy about it. Builds are really fast and reproducible.

erichocean 8y ago
Same. Bazel is a breath of fresh air after CMake. Makes me actually want to use C++ for daily work.
- floatboth 8y ago
  Have you tried Meson? It's also a breath of fresh air after CMake, but while Bazel is a "corporate" build system (wants to take everything over, assumes one consistent environment), Meson is a "very Unix" build system, it aims for portability, configurability and reproducibility.
  - solomatov 8y ago
    It doesn't take everything over. You can use repository rules to build dependencies in whatever way you want. Also, it's not corporate and not bloated. If you understand the system well enough, you can create support for a language yourself from scratch without spending huge amount of time on it.

netheril96 8y ago

As a Googler I am very satisfied with Blaze (the internal version of Bazel) at work. But I just don't want to manage such a complex beast for my own personal projects. Besides the large amount of dependencies to install for a build tool, it demands me to manually rewrite the build script for almost all third party libraries. At Google someone has most likely done that for me, but outside it, I have to do it myself.

n-gatedotcom 8y ago
Whoever doenvoted, can you please explain so I know what's wrong ? To an not-googler this seems a pretty well reasoned argument with not much that seems to be wrong-facts or stupid.
habitue 8y ago
This is the catch with Bazel. All of its integration with npm, pypi, crates.io, etc are hacks that essentially vendor the source of your dependencies and attempt to generate BUILD files for those deps with some heuristics. For a lot of packages the heuristics work, but not all. Ultimately bazel needs you to vendor and hand-maintain BUILD files to get a seamless experience.

n-gatedotcom 8y ago

I am not very familiar with build systems but how difficult would it be to make one that wraps the native ones whenever and provides modern features, unlike make.

logicchains 8y ago

Anyone had any luck embedding Bazel in another build system, like CMake? Tensorflow's C++ API seems to only support building with Bazel[1], which makes it painful to integrate into a standard C++ build pipeline (don't want the person maintaining the build to come after me with an axe). I'm seriously considering just rewriting to Torch for low latency deployment, which is nicely packaged in the standard manner of library + header file rather than trying to force users to incorporate a heavyweight, esoteric build system just to use the library. 1. https://www.tensorflow.org/guide/extend/cc https://www.tensorflow.org/guide/extend/cc - "The C++ API is only designed to work with TensorFlow bazel build. If you need a stand-alone option, use the C API" (and the C API page notes that it's "designed for simplicity and uniformity rather than convenience", making it unsuitable as a wholesale replacement for the C++ API).

lstamour 8y ago
Eventually they’ll refactor the C++ and Java hard-codes functionality into Starlark rules but until then, they don’t support exporting headers from Bazel C++ yet— https://github.com/bazelbuild/bazel/issues/5192 https://github.com/bazelbuild/bazel/issues/5192

fhrow4484 8y ago

Some features that I find elegant in Bazel, but probably don't have that much impact on small projects/teams is the whole concept of "visibility"[1]. For large projects, being able to control who can take a dependency on your code is really nice. This mixes really well with automatically adding Required reviewers in Pull Requests based on which folder gets modified [2][3]. [1] https://docs.bazel.build/versions/master/be/common-definitions.html#common-attributes https://docs.bazel.build/versions/master/be/common-definitio... [2] https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=vsts#automatically-include-code-reviewers https://docs.microsoft.com/en-us/azure/devops/repos/git/bran... [3] https://help.github.com/articles/about-codeowners/ https://help.github.com/articles/about-codeowners/

joshuamorton 8y ago
It's also incredibly nice for having a clean deprecation process. Make visibility private except to current blaze deps, and then deprecate or delete as normal. You can be certain no one new will depend on the old stuff.

jondubois 8y ago

I cannot believe how popular TypeScript has become. Is Microsoft running a huge ad campaign? I've used TypeScript for frontend work (Angular 2) for about 6 months. I did not renew my contract because I did not enjoy constantly fixing TypeScript configuration, versioning and source mapping issues. My new company switched to TypeScript for backend work and development across all teams has slowed down noticeably. I spend a lot of my time just discussing with colleagues what types should be called and whether or not we should merge two types together into one or whether or not it makes sense to have two different types to represent essentially the same data in order to enforce different constraints on the user of the data depending on the kinds of functionality that we want to expose. Then, as we build the project (in an agile way), we have to keep renaming and combining types ALL the time because otherwise we end up with too many different types that sound almost the same and have almost all the same properties but are slightly different semantically. Also, we spend a lot of time just thinking about what the intention of the user should be and all the ways in which we can impose contraints on them instead of correctly acknowledging that the user will always find some way to do whatever the heck they want and that all this strict typing adds no real value once you introduce the human factor. Ironically, the so called 'stupidity' which TypeScript aims to protect the code from will find even more compex, more creative and less obvious ways to express itself within that code.

tekkk 8y ago
What are you talking about? Lost productivity compared to what, JS? Merge types? You can just use `&` to combine them if need be? Discussing types? Does it matter that much, and I mean refactoring them isn't a big deal if need be. Two different types represent essentially the same data? You've tried extending interfaces or using generics? Seems like you lack an experienced TS developer in your team and are contributing your lack of experience as a fault of the language itself. The typing has saved my ass maany times and makes the code in our projects actually maintainable for us and for someone outside of our team's bubble. It might take a while longer to code the app, but it'll take less hours in the long run to keep it safe and working.
- jondubois 8y ago