14 ms·

Launch HN: Moonrepo (YC W23) – Open-source build system

200 points by mileswjohnson 4y ago

Hey HN, Miles and James here from Moonrepo (https://moonrepo.dev https://moonrepo.dev). Are you struggling with large codebases? Well look no further! We built Moonrepo to simplify repository management, project ownership, task running, and everyday developer and productivity workflows. If you’ve used Bazel (or another “enterprise” build system) in the past, you’re probably aware of how complex they can be to setup, configure, and use. Let alone the cognitive overhead required by developers on a day to day basis. After more than a decade in the industry, with many of those years working on infrastructure and developer tooling related products, we set out to build Moon, a language agnostic build system. Existing systems focused solely on runtime logistics (faster builds, concurrency), while we want to also focus on the developer experience. We do this by automating workflows as much as possible, in an effort to reduce manual work. We constantly sync and verify configuration, so that the repository stays in a healthy state. We also infer/detect as much as we can from the environment/repository/codebase, so pieces "just work". We wanted our system to be enjoyable to use and easy to understand, but also solve the same problems as existing systems. For example, configuration is in YAML, not a proprietary syntax. Tasks are defined and run as if you were running them in the terminal; no more abstractions like BUILD files. Unlike Bazel, we don’t hide or heavily rewrite terminal output, so the feedback loop is what you expect. We manage a toolchain, ensuring the correct version of languages is used (no more “works on my machine”). And lastly, our foundation is built on Rust and Tokio, so performance is first-class, the runtime is reliable, and memory safety is guaranteed. We follow the open core model. Moon is open source, but we’re also working on a few subscription-based services for monitoring and improving your continuous integration pipelines, a registry of project and code ownership, a continuous deployment/delivery board, auxiliary application systems, and more. We haven't finalized the subscription model yet, so there's no pricing information on the website. However, we do have a starter/free tier that everyone can use by registering on https://moonrepo.app https://moonrepo.app. In the future, we will offer on-prem as well. Although Moonrepo is relatively new, we’re already feature-packed, stable, and used in production. We’re big fans of honest feedback, and look forward to your comments!

joeswartz 4y ago
Congrats on the launch! Indeed interesting space. I wonder if it has some overlap with https://skaffold.dev/ https://skaffold.dev/. I would be curious to hear the differences.
- mileswjohnson 4y ago
  At a quick glance, I don't believe there's any overlap.
kodah 4y ago
I went to see if I could use moon and found this page: https://moonrepo.dev/docs#supported-languages https://moonrepo.dev/docs#supported-languages Does that mean most of the automated features you mentioned here only apply to JS?
beisner 4y ago
What's the motivation for using YAML instead of Starlark (Bazel, Buck) or something closer to Python (Pants, please.build)? Seems as though much of the other monorepo tools have (kinda) standardized on this. As I understand it, the primary reason these build systems leverage these Python-variants is so that the build rules, toolchains, constraints, and build definitions can all be written in the same language (since build rules often require some programmatic behavior). Perhaps with a future vision of them being totally interoperable across build systems.
- Denzel 4y ago
  Not to mention using an actual language aids readability, extensibility, and static analysis, unlike a data exchange format. Starlark is a benefit, not a con, so YAML feels like a major step backwards. I'm generally happy with Blaze/Bazel, so I'm not necessarily in the target market for Moonrepo, I guess. EDIT: This isn't really competing with Blaze/Bazel either when I look at the execution model. It goes back to imperatively defined tasks instead of declaratively defined dependencies, which feels more spirtually aligned with Make than Blaze/Bazel.
  - mileswjohnson 4y ago
    Yeah that's fair feedback. At this time, we consider ourselves more of a Bazel-lite than an actual Bazel replacement. Once we support more languages and features, this may change in the future.
    - Denzel 4y ago
      I'm not trying to be overly critical here, in fact, I want to share the type of empathetic feedback I'd hope to receive as a founder. Have you actually talked to Blaze/Bazel users to understand what frustrations (if any) they have with their current build system? Have these users asked for a Bazel-lite? If so, and you still want to position yourself as Bazel-lite, then you should include some of their direct feedback and write your messaging accordingly. As a daily Blaze/Bazel user, I don't have a desire for a Bazel-lite. I've worked at a midsize company that used Bazel, and I'm working at the company that created Blaze/Bazel. Disclaimer: Opinions expressed are my own; not representative of my employer.
QuiiBz 4y ago
Congrats on the launch! I've been following Moon since a few months, seems like an interesting project. Could you explain why any existing project using Turborepo/Nx should switch to Moonrepo? What are the advantages and disadvantages? The support for multiple languages seems like a big advantage.
- mileswjohnson 4y ago
  I can speak to both of these. I'll start with Turborepo. Turbo is primarily a task runner for `package.json` scripts with some caching... and that's basically it. If that's all you need, then great, but if you're looking for more functionality, that's where moon comes in. moon is more than just a task runner, we're aiming to be a repository management tool as a whole. This includes project/code ownership, direct CI support, future CD support, code generation, hooks management, constraints, release workflows, and much more. With that being said, we do have a comparison article against Turbo: https://moonrepo.dev/docs/comparison#turborepo https://moonrepo.dev/docs/comparison#turborepo As for Nx, they're more of a competitor than Turborepo. Nx and moon are aiming to solve the same problems, but go about it in different ways. Nx is Node.js based and requires heavy adoption of their ecosystem (@nrwl packages) and their executors pattern. In the long run, this becomes a heavy source of tech debt, as your dependencies are now tightly coupled to their packages and release timelines. With moon, we wanted to avoid this all together. There are no coupled dependencies, and tasks are ran as if you ran them yourself on the command line. No abstraction layer necessary. We also want to embrace a language's ecosystem as much as possible, so moon adoption should be rather simple and transparent (at most each project has a moon.yml file). But to your last point, we agree, multi-language support is a massive advantage. Having both backend and frontend code in the same repository, powered by the same build system, is a massive win in maintenance costs and developer time saved.
  - QuiiBz 4y ago
    Thanks for your detailed answer. > release workflows Looking forward for this, especially if that also means auto-publishing of NPM packages, Rust crates, etc.
pplonski86 4y ago
Congrats on launch! Do you have examples of monorepos, for example React with TypeScript and Django with Python?
- mileswjohnson 4y ago
  Thank you! We have an example monorepo (https://github.com/moonrepo/examples https://github.com/moonrepo/examples) but at this time it's only JavaScript. We're working on adding Go to this repo, but we personally don't have enough Python experience to add Python. Always open to contributions!
KMag 4y ago
I'm not a lawyer, but given that monorepo was in use in the versioning/build space long before launch, I would have been hesitant to launch with a name where it's going to be an uphill battle to enforce trademark. (I also worked for Google a while back, and they were very conscious about not using Google as a verb internally, losing a slow battle against the tide of trademark dilution.)
- avarun 4y ago
  It took me three readthroughs to notice as well, but the name is actually `moonrepo` not `monorepo`.
  - KMag 4y ago
    Oops. Thanks for the catch.
  - sirtimbly 4y ago
    try googling for it... spell check autocorrect is killing it.
    - avarun 4y ago
      Shows up fine for me
- tnorthcutt 4y ago
  Did you notice that this product is called `moonrepo`, and not `monorepo`?
bunged 4y ago
Why would I want to use this instead of, say, GitLab's CI/CD Pipelines or Azure Pipelines? It's a fair bit of effort to change one's already established code repository and build system, and I don't really understand from your pitch what you are offering that is sufficiently advantageous for a decision maker to choose to switch.
- mileswjohnson 4y ago
  So moon wouldn't replace your actual CI pipelines. It would be a tool that runs _in your pipeline_ to effectively run tasks as fast as possible. For example, in CI, tasks are only ran if they are affected by files changed in the pull request. No more running everything unnecessarily. We also support remote caching of artifacts, which helps to speed up CI even further, by avoiding long build times.
  - bunged 4y ago
    Interesting, so does it keep all the intermediates from previous builds and then does an incremental build on top of this? Like doing a local dev build but in the cloud?
    - mileswjohnson 4y ago
      Yeah at a high-level, that's how it works. However, the incremental caching part does require remote caching of artifacts, so that they can be shared across CI jobs/runs.
      - bunged 4y ago
        Thanks for explaining, that does sound compelling, and useful that it can be slotted into an existing system.
    - tunkafor 4y ago
      We've had a lot of success with Brisk (https://brisktest.com/ https://brisktest.com/) doing this exact thing. It keeps our environment running so we don't have to rebuild on every run, kind of like a dev build. It's really fast.
newaccount2021 4y ago
[dead]
jpgvm 4y ago
Bazel has a heavy focus on correctness (pretty much to the exclusion of everything else). Where does Moon fall on the correctness gradient? Does it enforce hermecity or deterministic builds or give me tools to accomplish it? In the same vein as those questions how does caching work? Is it content based like Bazel or mtime like Nx et al? If there is no sandboxing does it do input tracking or is there manual "cache key" shenanigans? If the configuration language is YAML how am I expected to implement new behavior? Is that in Rust? Is there a plugin architecture? Do I need to rebuild the tool itself and redistribute it to my build machines and developers? The main appeal of Starlark/Python in build systems is ability to create macros or in many cases define entirely new rulesets to support new tools and languages without needing to modify the complex and performance sensitive core. Sorry for the skeptisicm but build systems are very complex beasts and new entrants like Nx don't measure up to tools like Bazel very well.
- thundergolfer 4y ago
  To be clear, Bazel focuses on correctness because it's essential to acheiving performance at scale. If you don't have correct caching of intermediate build artifacts, a system can't handle the compile and test requirements of large codebases.
  - malkia 4y ago
    that +1000
- mileswjohnson 4y ago
  > Does it enforce hermecity or deterministic builds or give me tools to accomplish it? I wouldn't say moon is hermetic, nor are we trying to be. We don't use the sandbox approach for tasks, and run against the original files. For the languages we support, this works best. As for deterministic, we try to be. We have a "toolchain" where we download languages/tools in the background, and run tasks using these tools. This ensures, at minimum, that the same version/variant of a tool is used across machines. > In the same vein as those questions how does caching work? It's content based. We also hash other kinds of inputs depending on the language that is running. > If the configuration language is YAML how am I expected to implement new behavior? (and other questions) Our focus right now is on non-compiled languages, primarily web languages, where custom behavior (like Starlark) is not necessary. In the future, this may change.
pdeva1 4y ago
one of the reasons Bazel needs BUILD files with explicit inputs /outputs defined per file is to do fine grained incremental builds and test runs. so if i change say foo.c, i only need to recompile foo.obj and run ‘foo-tests’. Moon seems to take globs as input. Thus modifying even a single file inside ‘src’ dir will trigger rebuild/retest of the entire ‘project’
- mileswjohnson 4y ago
  Our configuration is using globs but under the hood we content hash all the files that match the glob, and only run/cache if the aggregated hash has changed. For the languages we currently support, this is more than enough. Once we dive deeper into compiled languages (probably starting with Rust), we'll look into more granular reactivity and possible use something like sccache.
  - thundergolfer 4y ago
    Bazel's glob does the same thing. I'm not sure what the OP means, Bazel's incrementality is aided by fine-grained build input specification, but it's incrementaility is at core a combination of deterministic build rules, storage of build rule execution history, and early-stopping.
  - kccqzy 4y ago
    I don't see how that solves the problem mentioned by OP. If the build rule mentions foo.c, we only need to recompile that one object and relink. When you are using globs, changing one file changes the aggregated hash and then would necessitate recompiling every object file.
Iv 4y ago
That sounds like a "there are too many competing standards, let's make standard to unify them!" situation.
- mileswjohnson 4y ago
  insert xkcd comic
debacle 4y ago
How are you going to deal with the mass diaspora when your runway runs short and VCs want to make money? This seems to be a cyclical thing with build tools. Things are great until you expect customers to pay.
IceWreck 4y ago
You're building something like buck/blaze and replacing starlark with YAML ? I want a full programming language when defining complex rules. Build systems do a lot more than execute commands. You lost me there. Buck/Blaze aren't perfect but configuration was never an issue. Moon like it is currently is a glorified task runner.
- quickthrower2 4y ago
  We started using https://nuke.build/ https://nuke.build/. Early days so can’t comment too much but it seems good so far.
jedberg 4y ago
My advice to anyone making a new build system: Most likely you did this because you felt all the other ones are too complicated. But the reason the “enterprise” ones are so complicated is to serve their enterprise customers, who need “just this one feature” so they can use it. But those customers pay the bills. So basically you have to choose complexity and profit or simplicity and less (or no) profit. Good luck! But make sure you’re ready to make that choice.
- nitsky 4y ago
  Are you sure you can't have your cake and eat it too? You can have many configuration options, but give each one a sane default.
  - jez 4y ago
    In my experience (Bazel, sample size of 2 projects), the complexity doesn't come from configuration options that have defaults, but from how well the "mental model" of the build system fits the preconceived notions of how to structure, organize, and depend on code in an existing project. Almost none of the complexity comes from what configuration options I've registered ahead of time. It comes almost entirely from, "Well darn, this code depends on this completely unrelated part of the project. I wish it didn't, but now the build tool either needs to sometimes fail to rebuild something correctly, or it needs to build way too much to run quickly."
    - jpgvm 4y ago
      Ironically this means the best time to adopt Bazel is from the very start. Despite the fact Bazel doesn't add much at that point in time it's when the cost is lowest and it prevents impedance mismatch from being introduced. This runs counter to how most people think of things like Bazel which are tools you should only reach for when the situation has already grown out of control.
djstein 4y ago
congrats on the launch! can you share any information on how the system is currently deployed? or how it will be deployed for on premises solutions? also: are you looking for any other founders?
- mileswjohnson 4y ago
  We haven't built the on-prem solution yet, but we're leaning towards using Helm charts + Kubernetes. At minimum, it will all be Dockerfile based.
duped 4y ago
I'm supremely disappointed to see another service using YAML to configure task running. I do in fact need a real programming language to do this, and copying others in this vein is inheriting mistakes and not picking a battle-tested solution. What you will find is the vast majority of your configurations will invoke a make.sh script that does everything that you want to support in your system.
- mileswjohnson 4y ago
  Can you speak to what kind of "functionality" you need a language for? Are you referring to Starlark-like files?
  - duped 4y ago
    Turing completeness. No, I'm referring to using Python (waf, conan, etc), Javascript (nodejs scripts, esbuild, etc), and other "real" programming languages. It's incredibly frustrating to have to configure a build with something as shitty as YAML. There's a tangible amount of money I have wasted in organizations on it. A build is not actually a static configuration of another system. It's a program and deserves everything we need from programming languages.
    - nsm 4y ago
      You actually don't want a real programming language because their unbounded loops and standard libraries are sources of non-determinism that can introduce incorrectness in your build. Bazel's correctness is highly contingent on the same set of inputs producing the same outputs, and granularly tracking dependencies, running commands in sandboxes AND having a restricted build language is a key part of it. That said, Starlark is way closer to a real language (Python) than YAML.
      - School-Cotton 4y ago
        Unbounded loops have nothing to do with determinism. You can be deterministic and still Turing-complete.
lhorie 4y ago
IMHO, if you're targeting the Javascript ecosystem, this area is already fairly crowded, with Turborepo, Nx and various open source tools providing various degrees of functionality (Bazel, Pants, Lerna, etc) already competing in the space. I'm a tech lead for the web monorepo at Uber. We talked to the Turborepo guy a few years ago, and he admitted that he wasn't sure if it could handle our scale in terms of all the bells and whistles that a repo of our size leverages - and his is one of the more feature packed commercial offerings in this space. As a random example: we see thousands of commits a week, so invalidating the whole build graph when the lockfile changes is a complete non-starter, yet most turn-key solutions target small teams and are not equipped to cope with this problem. Phantom dependencies[0] are a problem with disjointed build graphs. Etc. As someone who's evaluated a ton of these systems, I would love to see where your experience in this space is coming from. A new kid in the block will have a lot to prove. [0] https://rushjs.io/pages/advanced/phantom_deps/ https://rushjs.io/pages/advanced/phantom_deps/
- palmdeezy 4y ago
  Turborepo author here… We do not invalidate the whole graph anymore for a lockfile change. We now have a sophisticated parser that can calculate if a change within the lockfile should actually alter the hash of a given target. In addition to higher cache hit rates, this is what powers our `turbo prune` command which allows teams to create slices of their monorepo for a target and its dependencies…useful for those building in Docker. Prune docs: https://turbo.build/repo/docs/reference/command-line-reference https://turbo.build/repo/docs/reference/command-line-referen... Turborepo is much more scalable now than when we spoke pre-Vercel acquisition. It now powers the core web codebases at Netflix, Snap, Disney Streaming, Hearst, Plex and thousands of other high-performance teams. You can see a full list of users here: https://turbo.build/showcase https://turbo.build/showcase Would be happy to reconnect about Uber’s web monorepo sometime.
  - renke1 4y ago
    Can Prune be used to build a bundle (as in a zip) for, say AWS Lambda, which includes only the dependencies (and not dev dependencies)? I've played around with pnpm's deploy but it felt a bit lackluster. Especially talking about situation where one has a backend package and some shared package. The bundle should contain all dependencies (but not dev dependencies) of the backend package and shared package and of course the built shared package should also be include in the bundle's node_modules.
bovermyer 4y ago
Would this be overkill for a personal project that is currently two repos, but I'm having to copy-paste code from one to the other whenever I make a change to a particular section?
- mileswjohnson 4y ago
  Can you talk a bit more about what kind of code is being copied?
  - bovermyer 4y ago
    The main repository is a single page app that's mostly Typescript compiled via Svelte down to a JavaScript bundle. Most of that repository is a series of modules in their own directories (e.g., "src/modules/character"). The second repository is a Node.js API. The majority of its functionality comes from one of the modules from the first repository that I copy in its entirety to the other repository whenever I change it.
    - mileswjohnson 4y ago
      Gotcha. While moon doesn't solve this directly, you do have a few options here: - Use git submodules. Have the node repo have a submodule on the app repo. - Publish the shared code to a private registry (like github's package registry), and pull it in as an npm package.
    - iskela 4y ago
      You should check git submodules out. Basically you could have your api-repo checkout certain commit from the spa-repo to a folder in the project root and reference the module reltive to that path.
    - Aeolun 4y ago
      Could you add the character module as a new package, the import it in the server with something like “char-package”: “file:../../client/modules/character”?
arnorhs 4y ago
Awesome, congrats. I've been an early trier of moon repo and I really fell for the slickness of the website and the name, when evaluating build tools. I've primarily worked in typescript codebases and have used raw yarn workspaces, lerna, nx and recently evaluated moon and turbo. The funky part is I eventually simply went with nx, not just because I've used it before, but also because I felt like the configuration is just simpler and more lightweight. Esp. since you can pretty much roll with it without defining much more than single simple config file - while moon required some 2-3 separate config files, plus config files in each project (I understand the per-project config is not required, I don't remember why I needed it - something to do with my build tasks).. In any case I intend to give it another shot soon. As for the whole config in yaml vs json vs toml or whatever, not a deal breaker since most of the time these sorts of configs are perhaps not something you end up interacting with programmatically - I know a lot of people enjoy editing yaml more than json files
- mileswjohnson 4y ago
  Thanks for the feedback. Too much configuration is something we keep thinking about, and are working to streamline. It's a bit involved since we support multiple languages.
- matesz 4y ago
  I burned few months evaluating and for trying out some monorepo options, for a cross-platform typescript project. Eventually I picked Nx, the main reason being having single package.json. Besides, yes, nx comes with a single configuration file for each project, but alongside of it are jest, eslint, babel and app/lib/spec/ide tsconfig configs - that’s a lot! All of this shouldn’t be visible to the developer. Initially, when trying to find myself in this mess, I thought that the solution lies in autogenerated ide workspaces - for vscode and sublime. But in practice it wasn’t that helpful, because there is always something which is not handled by autogenerate multiroot structure but is needed, so one needs to have multiple windows open anyways. Hopefully typescript 5.0 is going to help reduce some of this boilerplate with multiple tsconfig base classes, so lib/app/spec/ide tsconfigs will be able to extend from common base, which currently is not possible. The worst part of Nx is that there is lack of ssr support, so I had to patch nx with patch-package to generate obfuscated css classes in prod, because currently it’s not possible with their webpack executor! Recently they tried to simplify it a little bit, reducing this webpack boilerplate, which introduced few bugs but it's better than not doing anything! However, saying all of that and as someone mentioned in previous comments, dx is not primarily improved by lack of config boilerplate, but by having semantic structure of the actual code, clearly knowing what depends on what and where one can find it and put it. Right now we are on our own, because there is lack of information on how to structure cross platform codebase properly. I strongly believe it will change though. Best of luck to the people at moon! Ps. Nx has a nice blog post why they do t use bazel under the hood https://blog.nrwl.io/on-bazel-support-6be3b3ceba29 https://blog.nrwl.io/on-bazel-support-6be3b3ceba29
aidanhs 4y ago
(for context - I'm not interested in first class node support) This seems pretty cool. I particularly like how 'gradual' it seems to be relative to things like Bazel, i.e. you can take some shell scripts and migrate things over. I did have a play and hit an initial problem around project caching I think, which I raised at [0]. One comment, from the paranoid point of view of someone who has built distributed caching build systems before is that your caching is very pessimistic! I understand why you hash outputs by default (as well as inputs), but I think that will massively reduce hit rate a lot of the time when it may not be necessary? I raised [1]. Edit: for any future readers, I spotted an additional issue around the cache not being pessimistic enough [3] As an aside, I do wish build systems moved beyond the 'file-based' approach to inputs/outputs to something more abstract/extensible. For example, when creating docker images I'd prefer to define an extension that informs the build system of the docker image hash, rather than create marker files on disk (the same is true of initiating rebuilds on environment variable change, which I see moon has some limited support for). It just feels like language agnostic build systems saw the file-based nature of Make and said 'good enough for us' (honorable mention to Shake, which is an exception [2]). [0] https://github.com/moonrepo/moon/issues/637 https://github.com/moonrepo/moon/issues/637 [1] https://github.com/moonrepo/moon/issues/638 https://github.com/moonrepo/moon/issues/638 [2] https://shakebuild.com/why#expresses-many-types-of-build-rule https://shakebuild.com/why#expresses-many-types-of-build-rul... [3] https://github.com/moonrepo/moon/issues/640 https://github.com/moonrepo/moon/issues/640
- mileswjohnson 4y ago
  Thanks for the feedback and prototyping with it immediately! Always appreciated to get hands on feedback.
haberman 4y ago
> Tasks are defined and run as if you were running them in the terminal; no more abstractions like BUILD files. But abstractions are a good thing! They let you separate a system into layers, so that things programmed at a higher layer don't need to know all the details of the lower layers. It's the way we separate interface from implementation. Let me give an example from Bazel. Suppose you are compiling some C or C++, and you need to define a preprocessor symbol. There are two ways of doing this, "defines" and "local_defines": cc_library( name = "my_lib", srcs = ["my_lib.cc"], hdrs = ["my_lib.h"], # This will pass -DFOO when compiling this library # *or* any library that depends on it. defines = ["FOO"], ) cc_library( name = "my_lib", srcs = ["my_lib.cc"], hdrs = ["my_lib.h"], # This will pass -DFOO when compiling this library # only. Libraries that depend on us will *not* get # the symbol. local_defines = ["FOO"], ) You can use "defines" when you have #ifdef statements in headers and "local_defines" when you only test the symbols in your .cc files. I did not have to think about how defines will propagate to the libraries that depend on me, I just specified what I need and the build system handles the rest. I did not have to define my own CXXFLAGS variable and handle concatenating all of the right things together. The lower layer takes care of that. What Bazel lets you do is create abstraction boundaries within the build system. People can write rules that define a self-contained set of build logic, then other people can use those abstractions without knowing all the details of how they are implemented. Bazel is not perfect, and I have found myself banging my head against the wall many times. But overall the ability to define abstractions in a build system is, I think, one of the biggest things it gets right. Disclosure: I work for Google, but not on Bazel.
- theamk 4y ago
  Look at https://moonrepo.dev/docs#supported-languages https://moonrepo.dev/docs#supported-languages They only fully support Javascript. The complex stuff, like defines, C++ toolchain, dynamic libs, etc.. is all out of scope.
  - eisbaw 4y ago
    Wow true. Claiming to be a build system yet being limited to JS... Apparently, the legions of JS developers think the web is the only thing that exists today.
ctvo 4y ago
How do you compare with BuildBuddy? BuildBuddy embraces Bazel, but makes it easier to setup and operate Bazel when you don't have Google infrastructure. - https://www.buildbuddy.io/ https://www.buildbuddy.io/ - https://github.com/buildbuddy-io/buildbuddy https://github.com/buildbuddy-io/buildbuddy
- mileswjohnson 4y ago
  We have a long ways to go, but our end goal for moonbase is basically buildbuddy, but for non-bazel users. Right now moonbase requires moon, but we're looking to decouple it.
quickthrower2 4y ago
Apologies if this sounds reductive but can I think of it as an open source CircleCI? The thing that annoys me most about CircleCI, TravisCI, Github Actions and Appveyor is that there is no simple way to run the same thing locally to debug or test workflows without creating either git history mess or temporary hacks like taking off branch restrictions in the yaml.
- angio 4y ago
  I'm a big fan of using nix to achieve that. https://determinate.systems/posts/nix-github-actions https://determinate.systems/posts/nix-github-actions
- davistreybig 4y ago
  https://earthly.dev/ https://earthly.dev/ is an interesting way to run CI workflows locally
jcalder 4y ago
> We wanted our system to be enjoyable to use and easy to understand, but also solve the same problems as existing systems. For example, configuration is in YAML, not a proprietary syntax I'm incredibly skeptical of this. I'm ex-meta and have worked a lot with the enterprise solutions you're talking about and the choice of starlark (originally python) as the build definition language is one of the killer features of the systems. People want to create macros, target generators, etc. It's a common use case for a lot of engineers and IMO is a pretty killer feature. Being able to say "This is an X86 Python Binary", "This is an M1 python binary" and then bundle those into "this is how you build either of those binaries based on inputs" without ever touching the internals or anything other than (more or less) a blob of python is why those tools scale organizationally. It allows the teams that need to do weird stuff to unblock themselves without drowning the tools org. Sure, it has draw backs. Super deep macro layers are kinda a crime against humanity and debugging/evolving them can be quite expensive, but I think that's just the cost of software. If that logic isn't in the build definitions it'll expand into a meta layer that generates configuration (I've seen giant "Translate this definition into 30 configs to run stuff" systems time and time again). I may just be super biased from past mistakes and wins, but I think what you're doing is just moving the complexity out of your tool into neighboring tools and selling it as a win isn't really true, it's shuffling complexity around not removing it.
- mileswjohnson 4y ago
  Based on everyone's feedback about YAML (we didn't expect this much), we'll probably reconsider this!
  - gen220 4y ago
    Take a look at Apache Aurora [1] if you want some inspiration on how to mold python into a config language. I used this system for a few years and mainly agree with the person you're replying to – having a proper programming language to define config is a very nice feature that I miss. [1]: https://aurora.apache.org/ https://aurora.apache.org/
    - 4y ago
kkarimi 4y ago
I reviewed the market for JS monorepo tools a few months back and found NX to be a strong choice. Looking at Moon it seems like the syntax is quite nice but I don’t see graph feature of NX? Can moon only run on affected packages?
- mileswjohnson 4y ago
  Yes it can! That's pretty much the only way it works. The `moon run` command will only run if affected by changed files, and `moon ci` will only run affected tasks/projects in CI pipelines.
phist_mcgee 4y ago
Gonna plug our setup which is Justfiles[1] and turborepo. Just is a task runner (with full shell integration) that calls our turborepo tasks. We define all of our tasks in one justfile (things like repo setup, syncing env vars, and compiling dependencies) and then link them to turbo processes which cache the result. Massively reduced our cognitive load working with our monorepo, and is lightning fast. If we ever want to change it will be simple to remove both, so we're not tied to the ecosystem. [1]https://github.com/casey/just https://github.com/casey/just
moffkalast 4y ago
It's built in Rust and does not have full Rust language support? ... how?
- intelVISA 4y ago
  panic!()
masukomi 4y ago
A home page that literally just has company name, log in with github, and "learn more about company name" doesn't seem a very convincing pitch. I'm not going to click "learn more" because you've given me absolutely no reason to. The page may as well just be "log in with github" and nothing else. That'd serve the same functionality, and have an equally convincing pitch to prospective customers.
eric_khun 4y ago
We've started using Rush [1] at Buffer. Teams have been slowly migrating, and it has been great so far. We had (too many) repos with their own workflow and way too many different ways to build services, which has been annoying to maintain. I know teams use Rush at scale: Tiktok has ~450 projects, and Microsoft said they have ~700 projects. To deploy the services locally, we use Tilt[2] (K8s for local). We want to be able to reproduce production as much as possible and remove developer overhead on how things work locally and in production. Then come the issues with Docker and large node code bases: 1 challenge with large monorepos is the huge node_modules folder (rush among other tools put packages into a single large node_modules folder, and symlink every dependency there. It can contain millions of files and GBs, depending on how many 3rd-party libraries you use). On Linux, you can mount it without issues, but Mac has performance issues[3] with large folders. We pre-build that huge node_modules into a "base" image, and each service in the monorepo pulls that base image and only mounts what's necessary (a few mb). So we can save that npm build time and don't need to copy all those files inside the containers. It is fine because package.json does not change that often. You need to do this pre-build locally then in your CI/CD -> dockerhub, so everyone can get it. Another challenge is that you need to "watch" files to rebuild them. Watching all your files inside the monorepo isn't really viable. We use a dependency graph to know what services to watch and then copy the built files inside the Tilt containers. Hope this can help people. [1] https://rushjs.io https://rushjs.io [2] https://tilt.dev/ https://tilt.dev/ [3] https://github.com/docker/roadmap/issues/7 https://github.com/docker/roadmap/issues/7
- jpgvm 4y ago
  This is where Bazel + rules_js ecosystem shines. First of all it lazily fetches dependencies which means if you are only building 1 out of X projects you are only getting your projects dependencies from npm, then it's only providing just those dependencies to the sandbox when you are doing node.js things like Typescript compilation. Finally you can assemble Docker/OCI images using rules_docker and js_image_layer which prevents the need for Dockerfiles at all, creates images with just exactly the dependencies needed (instead of the huge node_modules base layer which would need re-generating and downloading anytime a single dep changed) and better yet can be built pretty much instantly in most cases because it doesn't run Docker to do it.
yewenjie 4y ago
A few months ago I tried moonrepo and couldn't get it to work. IIRC, it tries to bring its own nodejs and has no option for using the system provided one - this breaks entirely in NixOS as Nix's nodejs is patched for the lack of FHS.
malkia 4y ago
Instead of yaml have you looked into CUE?
tbezman 4y ago
Came in here to say that Miles is an absolute fucking unit of an engineer / thinker. We just adopted Moonrepo at Gallery and it's been excellent. He's felt the pain of other tools (bazel, nx, turborepo). So happy to see this launch on the front page
- mileswjohnson 4y ago
  <3
astral303 4y ago
Configuration is code, so the YAML written in your tool is still code, code that developers are consuming and versioning. Making it YAML doesn’t make it not code. Your YAML configuration files are proprietary syntax, it is a proprietary code syntax written in a language (YAML) lacks basic abilities for logic or control flow. In a system of any worthwhile complexity, inevitably custom build logic will have to be expressed, and I can’t imagine it being good when it is being cooked up in a proprietary configuration language based on YAML. How easy will it be for layers of abstractions to emerge in this system? That’s the ultimate limitation of markup-language-based / configuration-based build systems. Look up the difference between external DSLs and internal DSLs. The YAML config file is an external DSL. I prefer to express configuration using internal DSLs.
- SCLeo 4y ago
  Why? Why YAML? Why? YAML is absolutely terrible! Aside from the classic `country_code: NO`, the other day I ran into issues with scientific notations. Now, guess, which of the following are strings, and which are numbers: - 1e+10 - 1.e+10 - 1.0e10
  - speps 4y ago
    https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...
pharmakom 4y ago
Sounds great but why on earth YAML? It's too ambiguous and lacks expressive power. Starlark is actually one of the good things about Bazel. Alternatively, why not Dhall?
moltar 4y ago
Is there a way to use our own cache server? E.g. with Turborepo the cache server spec is somewhat known and there are plenty of various implementations in the wild.
throwawaaarrgh 4y ago
> For example, configuration is in YAML, not a proprietary syntax. This makes me sad. A product designed to be developer friendly and automate things should create a parser and custom config language, or at least borrow one. The whole reason there are so many is they are built for purpose to make a specific task easier. If you need features like templating and logic, you can slap in existing solutions like Jinja2 and Go templates. YAML is not a configuration format, it's a data serialization format. If you don't want to write a parser, use TOML. My final note is: don't reinvent the wheel. There is a fully open source alternative to Drone.io (I can't remember the name) that forked a while ago. Drone is the best CI system for developers, hands down. Build on the open source codebase, build some proprietary extensions and a new UI, sell your product there. It's proven (because Harness now owns it) and you can help build the existing open source product at the same time. Otherwise, I have no need for yet another build system. All build systems are pretty much the same to me (except Jenkins, which is pretty much the tenth circle of Hell) and I'd much rather use a fully open source one that I could pay for support + extensions for at work. Maybe you could make a better Jenkins?
tanepiper 4y ago
I've been using Moonrepo since it was in single-digit versions, and Miles has been very responsive when reporting bugs. I've used a few other monorepo tools and find it one of the easier to use and configure.
gandharvgarg 4y ago
Dark mode on the website is funny
one_thawt 4y ago
Your logo and especially your favicon look very similar to the Lua programming language imagery.
- oblio 4y ago
  Also: https://moonscript.org/ https://moonscript.org/ :-)
  - one_thawt 4y ago
    For sure! Reading the title, I thought moonrepo and moonbase were projects for MoonScript.

200 points by mileswjohnson 4y ago

joeswartz 4y ago

Congrats on the launch! Indeed interesting space. I wonder if it has some overlap with https://skaffold.dev/ https://skaffold.dev/. I would be curious to hear the differences.

mileswjohnson 4y ago
At a quick glance, I don't believe there's any overlap.

kodah 4y ago

I went to see if I could use moon and found this page: https://moonrepo.dev/docs#supported-languages https://moonrepo.dev/docs#supported-languages Does that mean most of the automated features you mentioned here only apply to JS?

beisner 4y ago

What's the motivation for using YAML instead of Starlark (Bazel, Buck) or something closer to Python (Pants, please.build)? Seems as though much of the other monorepo tools have (kinda) standardized on this. As I understand it, the primary reason these build systems leverage these Python-variants is so that the build rules, toolchains, constraints, and build definitions can all be written in the same language (since build rules often require some programmatic behavior). Perhaps with a future vision of them being totally interoperable across build systems.

Denzel 4y ago
Not to mention using an actual language aids readability, extensibility, and static analysis, unlike a data exchange format. Starlark is a benefit, not a con, so YAML feels like a major step backwards. I'm generally happy with Blaze/Bazel, so I'm not necessarily in the target market for Moonrepo, I guess. EDIT: This isn't really competing with Blaze/Bazel either when I look at the execution model. It goes back to imperatively defined tasks instead of declaratively defined dependencies, which feels more spirtually aligned with Make than Blaze/Bazel.
- mileswjohnson 4y ago
  Yeah that's fair feedback. At this time, we consider ourselves more of a Bazel-lite than an actual Bazel replacement. Once we support more languages and features, this may change in the future.
  - Denzel 4y ago
    I'm not trying to be overly critical here, in fact, I want to share the type of empathetic feedback I'd hope to receive as a founder. Have you actually talked to Blaze/Bazel users to understand what frustrations (if any) they have with their current build system? Have these users asked for a Bazel-lite? If so, and you still want to position yourself as Bazel-lite, then you should include some of their direct feedback and write your messaging accordingly. As a daily Blaze/Bazel user, I don't have a desire for a Bazel-lite. I've worked at a midsize company that used Bazel, and I'm working at the company that created Blaze/Bazel. Disclaimer: Opinions expressed are my own; not representative of my employer.

QuiiBz 4y ago

Congrats on the launch! I've been following Moon since a few months, seems like an interesting project. Could you explain why any existing project using Turborepo/Nx should switch to Moonrepo? What are the advantages and disadvantages? The support for multiple languages seems like a big advantage.

mileswjohnson 4y ago
I can speak to both of these. I'll start with Turborepo. Turbo is primarily a task runner for `package.json` scripts with some caching... and that's basically it. If that's all you need, then great, but if you're looking for more functionality, that's where moon comes in. moon is more than just a task runner, we're aiming to be a repository management tool as a whole. This includes project/code ownership, direct CI support, future CD support, code generation, hooks management, constraints, release workflows, and much more. With that being said, we do have a comparison article against Turbo: https://moonrepo.dev/docs/comparison#turborepo https://moonrepo.dev/docs/comparison#turborepo As for Nx, they're more of a competitor than Turborepo. Nx and moon are aiming to solve the same problems, but go about it in different ways. Nx is Node.js based and requires heavy adoption of their ecosystem (@nrwl packages) and their executors pattern. In the long run, this becomes a heavy source of tech debt, as your dependencies are now tightly coupled to their packages and release timelines. With moon, we wanted to avoid this all together. There are no coupled dependencies, and tasks are ran as if you ran them yourself on the command line. No abstraction layer necessary. We also want to embrace a language's ecosystem as much as possible, so moon adoption should be rather simple and transparent (at most each project has a moon.yml file). But to your last point, we agree, multi-language support is a massive advantage. Having both backend and frontend code in the same repository, powered by the same build system, is a massive win in maintenance costs and developer time saved.
- QuiiBz 4y ago
  Thanks for your detailed answer. > release workflows Looking forward for this, especially if that also means auto-publishing of NPM packages, Rust crates, etc.

pplonski86 4y ago

Congrats on launch! Do you have examples of monorepos, for example React with TypeScript and Django with Python?

mileswjohnson 4y ago
Thank you! We have an example monorepo (https://github.com/moonrepo/examples https://github.com/moonrepo/examples) but at this time it's only JavaScript. We're working on adding Go to this repo, but we personally don't have enough Python experience to add Python. Always open to contributions!

KMag 4y ago

I'm not a lawyer, but given that monorepo was in use in the versioning/build space long before launch, I would have been hesitant to launch with a name where it's going to be an uphill battle to enforce trademark. (I also worked for Google a while back, and they were very conscious about not using Google as a verb internally, losing a slow battle against the tide of trademark dilution.)

avarun 4y ago
It took me three readthroughs to notice as well, but the name is actually `moonrepo` not `monorepo`.
- KMag 4y ago
  Oops. Thanks for the catch.
- sirtimbly 4y ago
  try googling for it... spell check autocorrect is killing it.
  - avarun 4y ago
    Shows up fine for me
tnorthcutt 4y ago
Did you notice that this product is called `moonrepo`, and not `monorepo`?

bunged 4y ago

Why would I want to use this instead of, say, GitLab's CI/CD Pipelines or Azure Pipelines? It's a fair bit of effort to change one's already established code repository and build system, and I don't really understand from your pitch what you are offering that is sufficiently advantageous for a decision maker to choose to switch.

mileswjohnson 4y ago
So moon wouldn't replace your actual CI pipelines. It would be a tool that runs _in your pipeline_ to effectively run tasks as fast as possible. For example, in CI, tasks are only ran if they are affected by files changed in the pull request. No more running everything unnecessarily. We also support remote caching of artifacts, which helps to speed up CI even further, by avoiding long build times.
- bunged 4y ago
  Interesting, so does it keep all the intermediates from previous builds and then does an incremental build on top of this? Like doing a local dev build but in the cloud?
  - mileswjohnson 4y ago
    Yeah at a high-level, that's how it works. However, the incremental caching part does require remote caching of artifacts, so that they can be shared across CI jobs/runs.
    - bunged 4y ago
      Thanks for explaining, that does sound compelling, and useful that it can be slotted into an existing system.
  - tunkafor 4y ago
    We've had a lot of success with Brisk (https://brisktest.com/ https://brisktest.com/) doing this exact thing. It keeps our environment running so we don't have to rebuild on every run, kind of like a dev build. It's really fast.

newaccount2021 4y ago

[dead]

jpgvm 4y ago

Bazel has a heavy focus on correctness (pretty much to the exclusion of everything else). Where does Moon fall on the correctness gradient? Does it enforce hermecity or deterministic builds or give me tools to accomplish it? In the same vein as those questions how does caching work? Is it content based like Bazel or mtime like Nx et al? If there is no sandboxing does it do input tracking or is there manual "cache key" shenanigans? If the configuration language is YAML how am I expected to implement new behavior? Is that in Rust? Is there a plugin architecture? Do I need to rebuild the tool itself and redistribute it to my build machines and developers? The main appeal of Starlark/Python in build systems is ability to create macros or in many cases define entirely new rulesets to support new tools and languages without needing to modify the complex and performance sensitive core. Sorry for the skeptisicm but build systems are very complex beasts and new entrants like Nx don't measure up to tools like Bazel very well.

thundergolfer 4y ago
To be clear, Bazel focuses on correctness because it's essential to acheiving performance at scale. If you don't have correct caching of intermediate build artifacts, a system can't handle the compile and test requirements of large codebases.
- malkia 4y ago
  that +1000
mileswjohnson 4y ago
> Does it enforce hermecity or deterministic builds or give me tools to accomplish it? I wouldn't say moon is hermetic, nor are we trying to be. We don't use the sandbox approach for tasks, and run against the original files. For the languages we support, this works best. As for deterministic, we try to be. We have a "toolchain" where we download languages/tools in the background, and run tasks using these tools. This ensures, at minimum, that the same version/variant of a tool is used across machines. > In the same vein as those questions how does caching work? It's content based. We also hash other kinds of inputs depending on the language that is running. > If the configuration language is YAML how am I expected to implement new behavior? (and other questions) Our focus right now is on non-compiled languages, primarily web languages, where custom behavior (like Starlark) is not necessary. In the future, this may change.

pdeva1 4y ago

one of the reasons Bazel needs BUILD files with explicit inputs /outputs defined per file is to do fine grained incremental builds and test runs. so if i change say foo.c, i only need to recompile foo.obj and run ‘foo-tests’. Moon seems to take globs as input. Thus modifying even a single file inside ‘src’ dir will trigger rebuild/retest of the entire ‘project’

mileswjohnson 4y ago
Our configuration is using globs but under the hood we content hash all the files that match the glob, and only run/cache if the aggregated hash has changed. For the languages we currently support, this is more than enough. Once we dive deeper into compiled languages (probably starting with Rust), we'll look into more granular reactivity and possible use something like sccache.
- thundergolfer 4y ago
  Bazel's glob does the same thing. I'm not sure what the OP means, Bazel's incrementality is aided by fine-grained build input specification, but it's incrementaility is at core a combination of deterministic build rules, storage of build rule execution history, and early-stopping.
- kccqzy 4y ago
  I don't see how that solves the problem mentioned by OP. If the build rule mentions foo.c, we only need to recompile that one object and relink. When you are using globs, changing one file changes the aggregated hash and then would necessitate recompiling every object file.

Iv 4y ago

That sounds like a "there are too many competing standards, let's make standard to unify them!" situation.

mileswjohnson 4y ago
insert xkcd comic

debacle 4y ago

How are you going to deal with the mass diaspora when your runway runs short and VCs want to make money? This seems to be a cyclical thing with build tools. Things are great until you expect customers to pay.

IceWreck 4y ago

You're building something like buck/blaze and replacing starlark with YAML ? I want a full programming language when defining complex rules. Build systems do a lot more than execute commands. You lost me there. Buck/Blaze aren't perfect but configuration was never an issue. Moon like it is currently is a glorified task runner.

quickthrower2 4y ago
We started using https://nuke.build/ https://nuke.build/. Early days so can’t comment too much but it seems good so far.

jedberg 4y ago

My advice to anyone making a new build system: Most likely you did this because you felt all the other ones are too complicated. But the reason the “enterprise” ones are so complicated is to serve their enterprise customers, who need “just this one feature” so they can use it. But those customers pay the bills. So basically you have to choose complexity and profit or simplicity and less (or no) profit. Good luck! But make sure you’re ready to make that choice.

nitsky 4y ago
Are you sure you can't have your cake and eat it too? You can have many configuration options, but give each one a sane default.
- jez 4y ago
  In my experience (Bazel, sample size of 2 projects), the complexity doesn't come from configuration options that have defaults, but from how well the "mental model" of the build system fits the preconceived notions of how to structure, organize, and depend on code in an existing project. Almost none of the complexity comes from what configuration options I've registered ahead of time. It comes almost entirely from, "Well darn, this code depends on this completely unrelated part of the project. I wish it didn't, but now the build tool either needs to sometimes fail to rebuild something correctly, or it needs to build way too much to run quickly."
  - jpgvm 4y ago
    Ironically this means the best time to adopt Bazel is from the very start. Despite the fact Bazel doesn't add much at that point in time it's when the cost is lowest and it prevents impedance mismatch from being introduced. This runs counter to how most people think of things like Bazel which are tools you should only reach for when the situation has already grown out of control.

djstein 4y ago

congrats on the launch! can you share any information on how the system is currently deployed? or how it will be deployed for on premises solutions? also: are you looking for any other founders?

mileswjohnson 4y ago
We haven't built the on-prem solution yet, but we're leaning towards using Helm charts + Kubernetes. At minimum, it will all be Dockerfile based.

duped 4y ago

I'm supremely disappointed to see another service using YAML to configure task running. I do in fact need a real programming language to do this, and copying others in this vein is inheriting mistakes and not picking a battle-tested solution. What you will find is the vast majority of your configurations will invoke a make.sh script that does everything that you want to support in your system.

mileswjohnson 4y ago
Can you speak to what kind of "functionality" you need a language for? Are you referring to Starlark-like files?
- duped 4y ago
  Turing completeness. No, I'm referring to using Python (waf, conan, etc), Javascript (nodejs scripts, esbuild, etc), and other "real" programming languages. It's incredibly frustrating to have to configure a build with something as shitty as YAML. There's a tangible amount of money I have wasted in organizations on it. A build is not actually a static configuration of another system. It's a program and deserves everything we need from programming languages.
  - nsm 4y ago
    You actually don't want a real programming language because their unbounded loops and standard libraries are sources of non-determinism that can introduce incorrectness in your build. Bazel's correctness is highly contingent on the same set of inputs producing the same outputs, and granularly tracking dependencies, running commands in sandboxes AND having a restricted build language is a key part of it. That said, Starlark is way closer to a real language (Python) than YAML.
    - School-Cotton 4y ago
      Unbounded loops have nothing to do with determinism. You can be deterministic and still Turing-complete.

lhorie 4y ago

IMHO, if you're targeting the Javascript ecosystem, this area is already fairly crowded, with Turborepo, Nx and various open source tools providing various degrees of functionality (Bazel, Pants, Lerna, etc) already competing in the space. I'm a tech lead for the web monorepo at Uber. We talked to the Turborepo guy a few years ago, and he admitted that he wasn't sure if it could handle our scale in terms of all the bells and whistles that a repo of our size leverages - and his is one of the more feature packed commercial offerings in this space. As a random example: we see thousands of commits a week, so invalidating the whole build graph when the lockfile changes is a complete non-starter, yet most turn-key solutions target small teams and are not equipped to cope with this problem. Phantom dependencies[0] are a problem with disjointed build graphs. Etc. As someone who's evaluated a ton of these systems, I would love to see where your experience in this space is coming from. A new kid in the block will have a lot to prove. [0] https://rushjs.io/pages/advanced/phantom_deps/ https://rushjs.io/pages/advanced/phantom_deps/

palmdeezy 4y ago
Turborepo author here… We do not invalidate the whole graph anymore for a lockfile change. We now have a sophisticated parser that can calculate if a change within the lockfile should actually alter the hash of a given target. In addition to higher cache hit rates, this is what powers our `turbo prune` command which allows teams to create slices of their monorepo for a target and its dependencies…useful for those building in Docker. Prune docs: https://turbo.build/repo/docs/reference/command-line-reference https://turbo.build/repo/docs/reference/command-line-referen... Turborepo is much more scalable now than when we spoke pre-Vercel acquisition. It now powers the core web codebases at Netflix, Snap, Disney Streaming, Hearst, Plex and thousands of other high-performance teams. You can see a full list of users here: https://turbo.build/showcase https://turbo.build/showcase Would be happy to reconnect about Uber’s web monorepo sometime.
- renke1 4y ago
  Can Prune be used to build a bundle (as in a zip) for, say AWS Lambda, which includes only the dependencies (and not dev dependencies)? I've played around with pnpm's deploy but it felt a bit lackluster. Especially talking about situation where one has a backend package and some shared package. The bundle should contain all dependencies (but not dev dependencies) of the backend package and shared package and of course the built shared package should also be include in the bundle's node_modules.

bovermyer 4y ago

Would this be overkill for a personal project that is currently two repos, but I'm having to copy-paste code from one to the other whenever I make a change to a particular section?

mileswjohnson 4y ago
Can you talk a bit more about what kind of code is being copied?
- bovermyer 4y ago
  The main repository is a single page app that's mostly Typescript compiled via Svelte down to a JavaScript bundle. Most of that repository is a series of modules in their own directories (e.g., "src/modules/character"). The second repository is a Node.js API. The majority of its functionality comes from one of the modules from the first repository that I copy in its entirety to the other repository whenever I change it.
  - mileswjohnson 4y ago
    Gotcha. While moon doesn't solve this directly, you do have a few options here: - Use git submodules. Have the node repo have a submodule on the app repo. - Publish the shared code to a private registry (like github's package registry), and pull it in as an npm package.
  - iskela 4y ago
    You should check git submodules out. Basically you could have your api-repo checkout certain commit from the spa-repo to a folder in the project root and reference the module reltive to that path.
  - Aeolun 4y ago
    Could you add the character module as a new package, the import it in the server with something like “char-package”: “file:../../client/modules/character”?

arnorhs 4y ago

Awesome, congrats. I've been an early trier of moon repo and I really fell for the slickness of the website and the name, when evaluating build tools. I've primarily worked in typescript codebases and have used raw yarn workspaces, lerna, nx and recently evaluated moon and turbo. The funky part is I eventually simply went with nx, not just because I've used it before, but also because I felt like the configuration is just simpler and more lightweight. Esp. since you can pretty much roll with it without defining much more than single simple config file - while moon required some 2-3 separate config files, plus config files in each project (I understand the per-project config is not required, I don't remember why I needed it - something to do with my build tasks).. In any case I intend to give it another shot soon. As for the whole config in yaml vs json vs toml or whatever, not a deal breaker since most of the time these sorts of configs are perhaps not something you end up interacting with programmatically - I know a lot of people enjoy editing yaml more than json files

mileswjohnson 4y ago
Thanks for the feedback. Too much configuration is something we keep thinking about, and are working to streamline. It's a bit involved since we support multiple languages.
matesz 4y ago
I burned few months evaluating and for trying out some monorepo options, for a cross-platform typescript project. Eventually I picked Nx, the main reason being having single package.json. Besides, yes, nx comes with a single configuration file for each project, but alongside of it are jest, eslint, babel and app/lib/spec/ide tsconfig configs - that’s a lot! All of this shouldn’t be visible to the developer. Initially, when trying to find myself in this mess, I thought that the solution lies in autogenerated ide workspaces - for vscode and sublime. But in practice it wasn’t that helpful, because there is always something which is not handled by autogenerate multiroot structure but is needed, so one needs to have multiple windows open anyways. Hopefully typescript 5.0 is going to help reduce some of this boilerplate with multiple tsconfig base classes, so lib/app/spec/ide tsconfigs will be able to extend from common base, which currently is not possible. The worst part of Nx is that there is lack of ssr support, so I had to patch nx with patch-package to generate obfuscated css classes in prod, because currently it’s not possible with their webpack executor! Recently they tried to simplify it a little bit, reducing this webpack boilerplate, which introduced few bugs but it's better than not doing anything! However, saying all of that and as someone mentioned in previous comments, dx is not primarily improved by lack of config boilerplate, but by having semantic structure of the actual code, clearly knowing what depends on what and where one can find it and put it. Right now we are on our own, because there is lack of information on how to structure cross platform codebase properly. I strongly believe it will change though. Best of luck to the people at moon! Ps. Nx has a nice blog post why they do t use bazel under the hood https://blog.nrwl.io/on-bazel-support-6be3b3ceba29 https://blog.nrwl.io/on-bazel-support-6be3b3ceba29

aidanhs 4y ago

(for context - I'm not interested in first class node support) This seems pretty cool. I particularly like how 'gradual' it seems to be relative to things like Bazel, i.e. you can take some shell scripts and migrate things over. I did have a play and hit an initial problem around project caching I think, which I raised at [0]. One comment, from the paranoid point of view of someone who has built distributed caching build systems before is that your caching is very pessimistic! I understand why you hash outputs by default (as well as inputs), but I think that will massively reduce hit rate a lot of the time when it may not be necessary? I raised [1]. Edit: for any future readers, I spotted an additional issue around the cache not being pessimistic enough [3] As an aside, I do wish build systems moved beyond the 'file-based' approach to inputs/outputs to something more abstract/extensible. For example, when creating docker images I'd prefer to define an extension that informs the build system of the docker image hash, rather than create marker files on disk (the same is true of initiating rebuilds on environment variable change, which I see moon has some limited support for). It just feels like language agnostic build systems saw the file-based nature of Make and said 'good enough for us' (honorable mention to Shake, which is an exception [2]). [0] https://github.com/moonrepo/moon/issues/637 https://github.com/moonrepo/moon/issues/637 [1] https://github.com/moonrepo/moon/issues/638 https://github.com/moonrepo/moon/issues/638 [2] https://shakebuild.com/why#expresses-many-types-of-build-rule https://shakebuild.com/why#expresses-many-types-of-build-rul... [3] https://github.com/moonrepo/moon/issues/640 https://github.com/moonrepo/moon/issues/640

mileswjohnson 4y ago
Thanks for the feedback and prototyping with it immediately! Always appreciated to get hands on feedback.

haberman 4y ago

> Tasks are defined and run as if you were running them in the terminal; no more abstractions like BUILD files. But abstractions are a good thing! They let you separate a system into layers, so that things programmed at a higher layer don't need to know all the details of the lower layers. It's the way we separate interface from implementation. Let me give an example from Bazel. Suppose you are compiling some C or C++, and you need to define a preprocessor symbol. There are two ways of doing this, "defines" and "local_defines": cc_library( name = "my_lib", srcs = ["my_lib.cc"], hdrs = ["my_lib.h"], # This will pass -DFOO when compiling this library # *or* any library that depends on it. defines = ["FOO"], ) cc_library( name = "my_lib", srcs = ["my_lib.cc"], hdrs = ["my_lib.h"], # This will pass -DFOO when compiling this library # only. Libraries that depend on us will *not* get # the symbol. local_defines = ["FOO"], ) You can use "defines" when you have #ifdef statements in headers and "local_defines" when you only test the symbols in your .cc files. I did not have to think about how defines will propagate to the libraries that depend on me, I just specified what I need and the build system handles the rest. I did not have to define my own CXXFLAGS variable and handle concatenating all of the right things together. The lower layer takes care of that. What Bazel lets you do is create abstraction boundaries within the build system. People can write rules that define a self-contained set of build logic, then other people can use those abstractions without knowing all the details of how they are implemented. Bazel is not perfect, and I have found myself banging my head against the wall many times. But overall the ability to define abstractions in a build system is, I think, one of the biggest things it gets right. Disclosure: I work for Google, but not on Bazel.

theamk 4y ago
Look at https://moonrepo.dev/docs#supported-languages https://moonrepo.dev/docs#supported-languages They only fully support Javascript. The complex stuff, like defines, C++ toolchain, dynamic libs, etc.. is all out of scope.
- eisbaw 4y ago
  Wow true. Claiming to be a build system yet being limited to JS... Apparently, the legions of JS developers think the web is the only thing that exists today.

ctvo 4y ago

How do you compare with BuildBuddy? BuildBuddy embraces Bazel, but makes it easier to setup and operate Bazel when you don't have Google infrastructure. - https://www.buildbuddy.io/ https://www.buildbuddy.io/ - https://github.com/buildbuddy-io/buildbuddy https://github.com/buildbuddy-io/buildbuddy

mileswjohnson 4y ago
We have a long ways to go, but our end goal for moonbase is basically buildbuddy, but for non-bazel users. Right now moonbase requires moon, but we're looking to decouple it.

quickthrower2 4y ago

Apologies if this sounds reductive but can I think of it as an open source CircleCI? The thing that annoys me most about CircleCI, TravisCI, Github Actions and Appveyor is that there is no simple way to run the same thing locally to debug or test workflows without creating either git history mess or temporary hacks like taking off branch restrictions in the yaml.

angio 4y ago
I'm a big fan of using nix to achieve that. https://determinate.systems/posts/nix-github-actions https://determinate.systems/posts/nix-github-actions
davistreybig 4y ago
https://earthly.dev/ https://earthly.dev/ is an interesting way to run CI workflows locally

jcalder 4y ago

> We wanted our system to be enjoyable to use and easy to understand, but also solve the same problems as existing systems. For example, configuration is in YAML, not a proprietary syntax I'm incredibly skeptical of this. I'm ex-meta and have worked a lot with the enterprise solutions you're talking about and the choice of starlark (originally python) as the build definition language is one of the killer features of the systems. People want to create macros, target generators, etc. It's a common use case for a lot of engineers and IMO is a pretty killer feature. Being able to say "This is an X86 Python Binary", "This is an M1 python binary" and then bundle those into "this is how you build either of those binaries based on inputs" without ever touching the internals or anything other than (more or less) a blob of python is why those tools scale organizationally. It allows the teams that need to do weird stuff to unblock themselves without drowning the tools org. Sure, it has draw backs. Super deep macro layers are kinda a crime against humanity and debugging/evolving them can be quite expensive, but I think that's just the cost of software. If that logic isn't in the build definitions it'll expand into a meta layer that generates configuration (I've seen giant "Translate this definition into 30 configs to run stuff" systems time and time again). I may just be super biased from past mistakes and wins, but I think what you're doing is just moving the complexity out of your tool into neighboring tools and selling it as a win isn't really true, it's shuffling complexity around not removing it.

mileswjohnson 4y ago
Based on everyone's feedback about YAML (we didn't expect this much), we'll probably reconsider this!
- gen220 4y ago
  Take a look at Apache Aurora [1] if you want some inspiration on how to mold python into a config language. I used this system for a few years and mainly agree with the person you're replying to – having a proper programming language to define config is a very nice feature that I miss. [1]: https://aurora.apache.org/ https://aurora.apache.org/
  - 4y ago

kkarimi 4y ago

I reviewed the market for JS monorepo tools a few months back and found NX to be a strong choice. Looking at Moon it seems like the syntax is quite nice but I don’t see graph feature of NX? Can moon only run on affected packages?

mileswjohnson 4y ago
Yes it can! That's pretty much the only way it works. The `moon run` command will only run if affected by changed files, and `moon ci` will only run affected tasks/projects in CI pipelines.

phist_mcgee 4y ago

Gonna plug our setup which is Justfiles[1] and turborepo. Just is a task runner (with full shell integration) that calls our turborepo tasks. We define all of our tasks in one justfile (things like repo setup, syncing env vars, and compiling dependencies) and then link them to turbo processes which cache the result. Massively reduced our cognitive load working with our monorepo, and is lightning fast. If we ever want to change it will be simple to remove both, so we're not tied to the ecosystem. [1]https://github.com/casey/just https://github.com/casey/just

moffkalast 4y ago

It's built in Rust and does not have full Rust language support? ... how?

intelVISA 4y ago
panic!()

masukomi 4y ago

A home page that literally just has company name, log in with github, and "learn more about company name" doesn't seem a very convincing pitch. I'm not going to click "learn more" because you've given me absolutely no reason to. The page may as well just be "log in with github" and nothing else. That'd serve the same functionality, and have an equally convincing pitch to prospective customers.

eric_khun 4y ago

We've started using Rush [1] at Buffer. Teams have been slowly migrating, and it has been great so far. We had (too many) repos with their own workflow and way too many different ways to build services, which has been annoying to maintain. I know teams use Rush at scale: Tiktok has ~450 projects, and Microsoft said they have ~700 projects. To deploy the services locally, we use Tilt[2] (K8s for local). We want to be able to reproduce production as much as possible and remove developer overhead on how things work locally and in production. Then come the issues with Docker and large node code bases: 1 challenge with large monorepos is the huge node_modules folder (rush among other tools put packages into a single large node_modules folder, and symlink every dependency there. It can contain millions of files and GBs, depending on how many 3rd-party libraries you use). On Linux, you can mount it without issues, but Mac has performance issues[3] with large folders. We pre-build that huge node_modules into a "base" image, and each service in the monorepo pulls that base image and only mounts what's necessary (a few mb). So we can save that npm build time and don't need to copy all those files inside the containers. It is fine because package.json does not change that often. You need to do this pre-build locally then in your CI/CD -> dockerhub, so everyone can get it. Another challenge is that you need to "watch" files to rebuild them. Watching all your files inside the monorepo isn't really viable. We use a dependency graph to know what services to watch and then copy the built files inside the Tilt containers. Hope this can help people. [1] https://rushjs.io https://rushjs.io [2] https://tilt.dev/ https://tilt.dev/ [3] https://github.com/docker/roadmap/issues/7 https://github.com/docker/roadmap/issues/7

jpgvm 4y ago
This is where Bazel + rules_js ecosystem shines. First of all it lazily fetches dependencies which means if you are only building 1 out of X projects you are only getting your projects dependencies from npm, then it's only providing just those dependencies to the sandbox when you are doing node.js things like Typescript compilation. Finally you can assemble Docker/OCI images using rules_docker and js_image_layer which prevents the need for Dockerfiles at all, creates images with just exactly the dependencies needed (instead of the huge node_modules base layer which would need re-generating and downloading anytime a single dep changed) and better yet can be built pretty much instantly in most cases because it doesn't run Docker to do it.

yewenjie 4y ago

A few months ago I tried moonrepo and couldn't get it to work. IIRC, it tries to bring its own nodejs and has no option for using the system provided one - this breaks entirely in NixOS as Nix's nodejs is patched for the lack of FHS.

malkia 4y ago

Instead of yaml have you looked into CUE?

tbezman 4y ago

Came in here to say that Miles is an absolute fucking unit of an engineer / thinker. We just adopted Moonrepo at Gallery and it's been excellent. He's felt the pain of other tools (bazel, nx, turborepo). So happy to see this launch on the front page

mileswjohnson 4y ago
<3

astral303 4y ago

Configuration is code, so the YAML written in your tool is still code, code that developers are consuming and versioning. Making it YAML doesn’t make it not code. Your YAML configuration files are proprietary syntax, it is a proprietary code syntax written in a language (YAML) lacks basic abilities for logic or control flow. In a system of any worthwhile complexity, inevitably custom build logic will have to be expressed, and I can’t imagine it being good when it is being cooked up in a proprietary configuration language based on YAML. How easy will it be for layers of abstractions to emerge in this system? That’s the ultimate limitation of markup-language-based / configuration-based build systems. Look up the difference between external DSLs and internal DSLs. The YAML config file is an external DSL. I prefer to express configuration using internal DSLs.

SCLeo 4y ago
Why? Why YAML? Why? YAML is absolutely terrible! Aside from the classic `country_code: NO`, the other day I ran into issues with scientific notations. Now, guess, which of the following are strings, and which are numbers: - 1e+10 - 1.e+10 - 1.0e10
- speps 4y ago
  https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...

pharmakom 4y ago

Sounds great but why on earth YAML? It's too ambiguous and lacks expressive power. Starlark is actually one of the good things about Bazel. Alternatively, why not Dhall?

moltar 4y ago

Is there a way to use our own cache server? E.g. with Turborepo the cache server spec is somewhat known and there are plenty of various implementations in the wild.

throwawaaarrgh 4y ago

> For example, configuration is in YAML, not a proprietary syntax. This makes me sad. A product designed to be developer friendly and automate things should create a parser and custom config language, or at least borrow one. The whole reason there are so many is they are built for purpose to make a specific task easier. If you need features like templating and logic, you can slap in existing solutions like Jinja2 and Go templates. YAML is not a configuration format, it's a data serialization format. If you don't want to write a parser, use TOML. My final note is: don't reinvent the wheel. There is a fully open source alternative to Drone.io (I can't remember the name) that forked a while ago. Drone is the best CI system for developers, hands down. Build on the open source codebase, build some proprietary extensions and a new UI, sell your product there. It's proven (because Harness now owns it) and you can help build the existing open source product at the same time. Otherwise, I have no need for yet another build system. All build systems are pretty much the same to me (except Jenkins, which is pretty much the tenth circle of Hell) and I'd much rather use a fully open source one that I could pay for support + extensions for at work. Maybe you could make a better Jenkins?

tanepiper 4y ago

I've been using Moonrepo since it was in single-digit versions, and Miles has been very responsive when reporting bugs. I've used a few other monorepo tools and find it one of the easier to use and configure.

gandharvgarg 4y ago

Dark mode on the website is funny

one_thawt 4y ago

Your logo and especially your favicon look very similar to the Lua programming language imagery.

oblio 4y ago
Also: https://moonscript.org/ https://moonscript.org/ :-)
- one_thawt 4y ago
  For sure! Reading the title, I thought moonrepo and moonbase were projects for MoonScript.