3 ms·

compute system

22 points by pandiculation 2y ago

I've been tired with the current options on the market for awhile and decided to do something about it after the running into the disaster that is MLOps at my last two startups and having to manage a multiple operation platforms both for my fellow ML engineers, the general application CI/CD and orchestration layers while simultaneously building the application itself. Its still extremely early for the product but its functioning and is well on its way. I'd love feedback on the approach and peoples thoughts on the problem space. Personally my irritations have been in the poor tooling, slow iteration cycles, an the amount of manual setup that's needed just for a basic system with reasonable performance. I also wanted something that had a price point that is reasonable for solo projects, as I believe that's where most of the magic happens in our industry and I wanted something that supported those of us in that space while making it easier to contribute to projects by making it basically a single command to spin up the development environment. I've been pretty happy with the outcome so far, there are still a bunch of things we need to fix in upstream systems to really make it shine. But those issues are fairly straight forward technically just will take some time to get through the human portion of those issues.

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
LegNeato 2y ago
As someone who lead this space at multiple FAANGs, just use Bazel or Buck2 and have whatever CI call it. CI should be one line(-ish) calling a local tool. "Fat" CI is an anti-pattern. The problem is not the CI tools you mention are bad and the only thing available, the problem is that newer tools and methods solving these issues are not known or adopted. And if those don't work for you, build custom on something like Temporal.
- pandiculation 2y ago
  bazels for building. and by all means use bazel if thats your jam, not what EG is about directly. Personally, I never saw bazel as terribly useful with the languages I use since they tend to have decent build/dependency management built in. (aka: it was a solution made for c/c++ and python which have terrible build systems) Nor did I find it useful for running test suites when I needed to include postgresql/redis/what have you into the environment. EG is about integrating the entire lifecycle from local development to deploying into production to managing SOP for deployed environments using virtualization. You can think of it as codespaces/temporal combined without ever needing to touch a cloud provided system. My main problem with those systems is essentially: not local first, priced for large companies, and usually require a team of engineers to maintain for the company. essentially I find those to be the antipatterns in software today. (I dont consider EG a CI/CD tool to be honest, just a problem it handles really well)
  - LegNeato 2y ago
    You are mistaken on many accounts: 1. Bazel/Buck is not just for building, it is a task runner with built in distributed execution and caching. Which is, uh, what CI is. 2. You not finding it useful might be because you want to continue to use the same methods and tools yet get different results. Bazel/Buck supports Go and Rust and those just fine (JS is steadily improving as well, I know multiple public companies using it). Now, the DX may be different or non-standard but that's the point...different methods for different results. Of course there are tradeoffs to it not being the standard way to do things for a particular language! 3. You can run postgresql/redis/what have you into the environment with Bazel/Buck, all the companies using these tools do it. Almost everyone wires it up to https://testcontainers.com/ https://testcontainers.com/, see https://www.docker.com/blog/revolutionize-your-ci-cd-pipeline-integrating-testcontainers-and-bazel/ https://www.docker.com/blog/revolutionize-your-ci-cd-pipelin.... 4. For integrating the entire lifecycle, this is how it is set up at most companies...use Bazel/Buck for building the environment, use Bazel/Buck for building, use Bazel/Buck to trigger a deploy, etc. For example, see https://github.com/adobe/rules_gitops https://github.com/adobe/rules_gitops Again, not trying to be rude, just drop info. Your list of pain points in the post is real, but only with certain / older systems or ways of using them. If you look into the newer systems, you'll see that they were largely created to solve those issues!
nrclark 2y ago
It's always interesting to see different tools in this space. I'm glad that people are trying to innovate on CI/CD workflows. @pandiculation, have you checked out Dagger? How does EG compare with it?

22 points by pandiculation 2y ago

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

LegNeato 2y ago

As someone who lead this space at multiple FAANGs, just use Bazel or Buck2 and have whatever CI call it. CI should be one line(-ish) calling a local tool. "Fat" CI is an anti-pattern. The problem is not the CI tools you mention are bad and the only thing available, the problem is that newer tools and methods solving these issues are not known or adopted. And if those don't work for you, build custom on something like Temporal.

pandiculation 2y ago
bazels for building. and by all means use bazel if thats your jam, not what EG is about directly. Personally, I never saw bazel as terribly useful with the languages I use since they tend to have decent build/dependency management built in. (aka: it was a solution made for c/c++ and python which have terrible build systems) Nor did I find it useful for running test suites when I needed to include postgresql/redis/what have you into the environment. EG is about integrating the entire lifecycle from local development to deploying into production to managing SOP for deployed environments using virtualization. You can think of it as codespaces/temporal combined without ever needing to touch a cloud provided system. My main problem with those systems is essentially: not local first, priced for large companies, and usually require a team of engineers to maintain for the company. essentially I find those to be the antipatterns in software today. (I dont consider EG a CI/CD tool to be honest, just a problem it handles really well)
- LegNeato 2y ago
  You are mistaken on many accounts: 1. Bazel/Buck is not just for building, it is a task runner with built in distributed execution and caching. Which is, uh, what CI is. 2. You not finding it useful might be because you want to continue to use the same methods and tools yet get different results. Bazel/Buck supports Go and Rust and those just fine (JS is steadily improving as well, I know multiple public companies using it). Now, the DX may be different or non-standard but that's the point...different methods for different results. Of course there are tradeoffs to it not being the standard way to do things for a particular language! 3. You can run postgresql/redis/what have you into the environment with Bazel/Buck, all the companies using these tools do it. Almost everyone wires it up to https://testcontainers.com/ https://testcontainers.com/, see https://www.docker.com/blog/revolutionize-your-ci-cd-pipeline-integrating-testcontainers-and-bazel/ https://www.docker.com/blog/revolutionize-your-ci-cd-pipelin.... 4. For integrating the entire lifecycle, this is how it is set up at most companies...use Bazel/Buck for building the environment, use Bazel/Buck for building, use Bazel/Buck to trigger a deploy, etc. For example, see https://github.com/adobe/rules_gitops https://github.com/adobe/rules_gitops Again, not trying to be rude, just drop info. Your list of pain points in the post is real, but only with certain / older systems or ways of using them. If you look into the newer systems, you'll see that they were largely created to solve those issues!

nrclark 2y ago

It's always interesting to see different tools in this space. I'm glad that people are trying to innovate on CI/CD workflows. @pandiculation, have you checked out Dagger? How does EG compare with it?