8 ms·

Dagger: A shell for the container age

199 points by gk1 2y ago

benatkin 2y ago
I tried using nushell and had it as my default myshell for weeks but eventually it had too little familiarity and I settled on fish. So I’m probably not going to try this shell as my default shell. I think it’s going to be an uphill battle for them to get much adoption. Maybe the sandboxing will be nice and they can provide it in a way that isn’t tied to a particular shell. However sandboxing on a shell command level is too coarse grained for some things.
- shykes 2y ago
  Hi Ben! You should definitely not use Dagger Shell as your default shell. It's meant to complement it rather than replace it. From the post: > Dagger Shell isn’t meant to replace your system shell, but to complement it. When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
  - benatkin 2y ago
    So part of the idea would be to keep my secrets out of my default shell? I’ve been looking for ways to do that but I would want specific commands to have access to them. For instance I would just want git and gh to have access to my GitHub credentials and for another program not to be able to spawn git or gh with these credentials. It would also be important not to be able to accidentally run one of these by copy and pasting something. It seems that it would need to at least be partly taken care of by my default shell for it to be usable though.
    - Imustaskforhelp 2y ago
      Hmm. Sounds as an interesting problem. But how can you really differentiate b/w a user opening git and some other program running git. I think we would need friction there , some sort of manual intervention. The best I could think of was something like bitwarden/keepassxc like cli where it requires a password and it would just straight up copy that token into github. If we are really talking / you have the source code and you want end to end security , you could theoeretically also compile git with the specific idea / implementation of whatever encrypted password manager you might use directly within the code of git / github but I know that can be an overkill for this problem.
      - benatkin 2y ago
        What I’d want is to be able to run top level commands differently from other commands and being able to have git and gh be a wrapper that injects the permissions. They could also filter the arguments and environment variables, which I know is hard to get right. Subshells and other programs would be able to run git and gh, but not with the permissions. I could even run git and gh in a container that has a volume to be able to access the directory. I think I have an idea of what this could look like and I might try and prototype it with fish and see what code parts it goes down to gauge how secure it’s likely to be.
        Imustaskforhelp 2y ago
        Well do let me know! sounds really interesting!
  - Kinrany 2y ago
    Okay, but _can_ you use it as your default shell? It's not much of a shell otherwise, is it?
agambrahma 2y ago
So basically nix-shell ?
- levlaz 2y ago
  There’s a tiny but of overlap in principle, but you can use dagger shell without having to learn a new language and the interface is much simpler. Give it a try and let me know which you prefer!
ledauphin 2y ago
I had missed the fact that Dagger is apparently trying to... replace? Docker for container definitions. That's a pretty big vision. I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling. At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...
- levlaz 2y ago
  What kind of tooling do you have in mind to replace? Also what would you prefer to see instead of bash compatible?
  - mikepurvis 2y ago
    I mean if we're going full pie in the sky, fully declarative containers exist today: https://github.com/nlewo/nix2container#nix2container https://github.com/nlewo/nix2container#nix2container Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.
    - shykes 2y ago
      Dagger is fully declarative. It's just built on a dynamic declarative API, instead of a static declarative DSL. So, if you took Nix, and replaced the static scheme-like DSL with a proper API, and then built SDKs in 5 languages for that API; and then built a bash-like shell also for easy scripting; then you would start to have something that approximates Dagger.
      - mikepurvis 2y ago
        I stand corrected! I shall investigate more closely with this additional context. Thanks.
elorm 2y ago
Did Dagger pivot their product ? I seem to recall the main selling point was an Independent pipeline-as-code service. Looks like they’re trying to rebuild Docker now?
- shykes 2y ago
  It's the same product. Dagger Shell is a new client for the same Dagger Engine and its API. It allows you to do more from the command-line, before you have to switch to code. But the SDKs and "pipeline as code" are still there, unchanged.
a_t48 2y ago
This is very cool, I often find myself composing together a mixture of Dockerfiles and shell scripts for various images (build X Dockerfile for Y arch, save it as a base, build Z Dockerfile on top of the last thing, with W args). And then having to run it different ways with different mounts, based on if you're on a developer machine/robot/CI, running as a service/adhoc/attaching. This _looks_ like it could solve some of that messiness. I like that it can just reference the output of a build without having to muck around with tags (which are system global and can collide if you're not careful).
- levlaz 2y ago
  Yeah for sure, this was specifically designed to solve the problems you describe!
- mikepurvis 2y ago
  I have a particular hate on for the sandwich pattern where you have a Dockerfile to a "build" environment that then gets entered, and a bunch of stuff done in there, and then the results ADDed into a different Dockerfile for the publishable asset. I know multi-stage builds sort of address this need, but a lot of people aren't using them or have found cases where it's desirable to have the "running" context for the container, for example if a bunch of stuff is being volumed-in (which is now possible at build time but is janky). Or there's the copy my ansible playbooks into the container and run them there as localhost approach, which can be mitigated by what Packer does with at least running the playbook over SSH. But then Packer has no concept of multi-stage or copying in assets from other containers. There's a lot of room to do better in this space.
  - a_t48 2y ago
    I don't like the sandwich either, it's too easy to accidentally miss a dependency. What I find myself doing more is building one Dockerfile as a base, and then doing `FROM $BASE` in the second to ease readability, caching, image pinning, etc.
    - mikepurvis 2y ago
wavemode 2y ago
Kinda off topic, but this page and many other pages on dagger.io seem to be loading very large GIFs that would be much better off in a compressed format (mp4 or webm).
- shykes 2y ago
  You're right, not sure what's going on there. Escalating internally.
jbverschoor 2y ago
Somewhat related, self promo Docker shell container - https://github.com/jrz/container-shell https://github.com/jrz/container-shell
- shykes 2y ago
  That looks very neat! Let me see if we can integrate it with Dagger :)
mrbluecoat 2y ago
Is the objective to get inside a container to do dev stuff? Reminds me of https://www.jetify.com/devbox https://www.jetify.com/devbox and https://flox.dev/ https://flox.dev/
hamandcheese 2y ago
My initial impression (looking at the examples, not actual usage) is that this feels like a weird half-way between Dockerfile and using actual code to define and compose software (i.e. like Nix). Not super appealing to me as a heavy Nix user.
shykes 2y ago
Someone already built a web UI for composing Dagger Shell scripts in a notebook format: http://docs.runme.dev/guide/dagger http://docs.runme.dev/guide/dagger It's really neat, I recommend checking it out.
- Imustaskforhelp 2y ago
  woah , that actually looks really nifty. I am imagining this with a simple cloudflare tunnel and self hosting gitlab and I am really really seeing an open source way that developers can REALLY scale. I mean docker is really great but dagger in notebook formats just seems really cool ngl.
  - sourishkrout 2y ago
    love the enthusiasm. cocreator of Runme here. the best part with the Dagger + Runme combo is that it runs entirely local. this isn't just a huge with for portability. it also cuts down development cycle times significantly.
    - oulipo 2y ago
      Nice! Would be lovely to have a few "workflow examples" and tutorials which really showcase the capabilities of this setup on your website, I'm not exactly sure I understand yet in what kind of cases it really shines
      - sourishkrout 2y ago
        Absolutely. It's in the works. I'll be sure to drop additional resources once they are available. Resources available so far: - https://docs.runme.dev/guide/dagger https://docs.runme.dev/guide/dagger - https://github.com/runmedev/vscode-runme/blob/58ea9a10c00df7f3f4e0ba15a06e3503a649eff8/dagger/notebook/shell.dag https://github.com/runmedev/vscode-runme/blob/58ea9a10c00df7...
tonymet 2y ago
I'd like to use this tool as a breakpoint into a container. e.g. dagger Dockerfile L4 and it pops open a shell at that point. Then I don't need new syntax.
- shepherdjerred 2y ago
  earthly has something like this. it's wonderful. https://docs.earthly.dev/docs/guides/debugging https://docs.earthly.dev/docs/guides/debugging there's also https://github.com/ktock/buildg https://github.com/ktock/buildg
  - tonymet 2y ago
    looks promising thanks!
- shykes 2y ago
  You can call 'dagger -i' to enable interactive debugging. If a step in your pipeline fails, you get an interactive terminal to inspect the current state.
randomtechguy 2y ago
I feel like it's getting harder to tell what dagger is _actually_ for these days. At first we'd hoped it could replace jenkins - it provided an alternative way to run and debug CI pipelines - right on your machine! You could write in golang and just import what you needed. The dev direction feels more scattered now, trying to replace docker, be a new shell(?), and weirdly trying to be some kind of langchain? Doing something different doesn't imply better. A new set of complicated CLI args is no better than what we started with (shell scripts, or jenkinsfiles to integrate docker builds). I'm a little bummed that the project has seemingly drifted (in my view) from the original mission.
- Imustaskforhelp 2y ago
  theoretically you could always use the previous version / iterate on them. I think this approach of see what sticks and them trying out a lot of different things can be nice but not in its current form. Although I have installed dagger to give it a try and I am just not sure how to make it work , the quickstart / hello world doesn't' work. No I don't want to make an AI , I just want to see what you really are. And so I do agree with your statement!
  - randomtechguy 2y ago
    Yup - Ever since the introduction of llm as a core API we're basically going to stop upgrading / using the project unless things start moving in a more sane direction.
    - levlaz 2y ago
      Why is that? It’s just an extra type that has no impact on any other types. It’s a tool in your toolbox that you can use when you need it and ignore otherwise.
      - randomtechguy 2y ago
        Asked the opposite way - why is this a core API type of a buildkit interface? Having API calls to various LLMs as core functionality of a build system is just straight up weird. Doesn't fit, confuses developers on my team when they try to read/contribute to our shared build libraries, worries me about the direction the project is going. Seems strange that especially with the push for modules this was integrated as a core type. It has nothing to do with buildkit or builds.
sleepybrett 2y ago
I do all my work in containers, i'm still left asking 'why?'
imiric 2y ago
> We still use the shell, but it's no longer the universal composition system it was meant to be. Huh? When did that change? Dagger seems interesting, but this opinion that the shell and standard Unix tools are archaic is a dubious starting principle. Every now and then a project with similar ideas comes along, and whether it rejects the notion of passing unstructured data between commands, addresses the footguns and frustrations of shell scripting, or is built with a more modern and "safe" programming language, it ultimately never seems to catch on as much as traditional Unix shells and tooling has. The reality is that these tools and the design choices made 50 years ago are timeless in nature. They're deliberately lacking in features and don't attach themselves to any specific tech du jour. It's this primitive nature and the "do one thing well" philosophy that makes them infinitely composable. The same pipelines Unix nerds were building 50 years ago to solve problems are still useful today, which is remarkable when you consider how quickly technology moves. Sure, new tools are invented all the time, and they might do things better than old ones. I use `eza` instead of `ls`; `fd` instead of `find` (mostly); `rg` instead of `grep` (mostly); `fzf` is a pretty essential addition to my workflow, and so on. But the underlying principles of these tools are still the same as the tools they're replacing. They're just slightly modernized and ergonomic versions of them. Whether or not we need a `container` command, `from alpine`, or an entire new shell for that, is a separate topic. It could be argued that this could be accomplished with a few functions or standalone commands. Even if we do need this new tooling, that's great, but don't tell me that it's meant to replace a proven set of tools and workflows[1]. When containers stop being popular, will we still need this? Also, "Daggerverse" and "modules"? Great, let's bring in the npm mentality to the shell, just what I needed. [1]: Ah, they don't, it's meant to serve as a complement. Alright, fair enough. I'm lowering my pitchfork.
- manno23 2y ago
  [dead]
remram 2y ago
In clear terms, what would I do with this? What activity does it help me with? What program do I replace with it? "The devops operating system" does what?
- shykes 2y ago
  The article gives a few examples of concrete things you can do: - Build, run, and publish containers - Containerize language-specific builds - Run integration tests in ephemeral environments, in a repeatable way - Run your tools in controlled sandboxes with only the files, secrets and network access they need It also mentions what you might replace with it: > When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
  - remram 2y ago
    I already build containers with Docker. I already have Makefiles that build containers and run commands in them. I already replace my complex shell scripts with Python scripts. How does Dagger help? I just can't get anything concrete from their site.
    - levlaz 2y ago
      If you’re working on your own you may not need dagger. If you’re working on a team often your shell scripts, python programs, and makes files are not always portable between local and CI, but even worse between your local machine and your colleagues local machine. This is where dagger shines because it lets you do all that stuff in a fully portable way.
      - touristtam 2y ago
        I need an example to illustrate that, because I had to read this thread to even understand what I was reading/looking at.
xedeon 2y ago
This reminds me of https://github.com/progrium/envy https://github.com/progrium/envy https://vimeo.com/131329120 https://vimeo.com/131329120
Kinrany 2y ago
How come the pipe is a method call? Does this really allow composing programs?
jatins 2y ago
Was curious what Dagger does and this is what the homepage says > The cross-platform composition engine > Build powerful software environments from modular components and simple functions. Perfect for complex builds and AI workflows. This is generic to point of being useless. Everything is a composition engine. Javascript is a composition engine. macOs is a composition engine.
- levlaz 2y ago
  I would lump dagger in with JS (programming language, ecosystem, runtime) and MacOS (operating system) before comparing it to any single tool. Answering "what does dagger do" can be tough, because there are very broad applications. Its always hard to describe general purpose platforms like these. Dagger is an open platform for building composable software. It has SDKs, sandboxed execution environment, observability, and now this shell interface.
manno23 2y ago
[dead]
cedws 2y ago
Looks weird, sorry. Bash pipes make intuitive sense, you’re taking the output of one process and pushing it into another. Here, the use of pipes makes little sense. Why is “from alpine” being passed into “with-exec”? What is moving between those processes?
- michaelmior 2y ago
  I see it as just the builder pattern that's pretty common in languages like Java. `container` is constructing the builder itself and eventually `terminal` (or other commands) use the state that has been incrementally built up in the builder to actually do something with the container.
- levlaz 2y ago
  Its the same idea. You are taking the instructions in with-exec and pushing it to the previous type. In this case, `from alpine` is a function attached to the container type that has many additional functions. You chain them together to do stuff. You can do it through code as if it was any other object, but this shell allows you to do things without code as well. Perhaps the example is too simple to feel useful, but being able to pipe primitives like files, directories, containers, secrets, and even any custom object makes it possible to rapidly experiment with and compose pipelines.
- myaccountonhn 2y ago
  You're correct. As other comments pointed out, this is an OOP builder pattern: `container().from('alpine').withExec(args: ["apk", "add"]).terminal()` If it was unix pipes, they'd be context independent tools being composed together. Dagger can be a useful but I feel a bit deceived, and it makes me worry about the rest of the quality of this product.
  - levlaz 2y ago
    It’s similar to a builder pattern but not just a builder pattern because depending on the type you can indefinitely pipe context independent types. So it’s a bit of a hybrid between bash and powershell.
throwdaggeraway 2y ago
Dagger is a frustrating time sink – it's interesting for what should be boring infrastructure. I've wasted too many hours debugging low-level details of buildkit and dealing with leaky kernel abstractions instead of building actual products. Nix, Bazel, and similar alternatives may have learning curves, but they provide clearer guardrails and more predictable outcomes without exposing so many implementation details. I deeply regret the time I spent working with Dagger.
- levlaz 2y ago
  Thanks for sharing your experience, I'd love to hear more details. Feel free to send me an email if you'd like lev@dagger.io, it would be great to learn from your experience so we can continue to improve the platform for everyone.
GONE_KLOUT 2y ago
It would be better to not confront potential users with a long text about unix history, nobody has time to read that. Instead just write a few bullet points about what it actually does. However, I am very happy somebody finally put the prompt on TOP of a terminal window, since years I am wondering why people like to bow their head to the bottom of the screen. I guess it is the expected gesture for IT devs and admins nowadays to follow and bow your head, so this is a great help for people to understand what is the right posture in life!
baalimago 2y ago
Right, so they've re-invented the java builder pattern, but in a shell. Why not just use a Java SDK? Or go SDK with options pattern?
- shykes 2y ago
  Dagger does also provide a Java and Go SDK. It's all the same API underneath, so you're free to use whichever client you prefer.
la64710 2y ago
When something breaks in dagger how do you troubleshoot it?
- shykes 2y ago
  You have a few options: - Dagger supports opentelemetry natively: it emits traces, metrics and logs that you can send to your favorite observability tool, or to our commercial product Dagger Cloud (which has a free tier) which lets you drill down into the trace. See https://docs.dagger.io/features/visualization https://docs.dagger.io/features/visualization - The Dagger CLI also renders the otel traces, and you can browse them as well. Press escape, then navigate with up, down, enter, escape. Same drill-down functionality but straight from the CLI - You can call dagger with the --interactive flag. If you do, in case of an error you will get an interactive shell to inspect the state at time of failure. - You can also call the `Container.terminal()` function from your code, to force an interactive terminal to appear for the user at that stage. Sort of like a "debug printf" but with an interactive terminal
Annatar 2y ago
[dead]
contingencies 2y ago
A: Let's do stuff with containers. B: What's a container? A: It lets you execute a sequence of commands in an isolated and controlled environment that can be readily rebuilt, cloned or disposed of. B: How do we do that? Using a standard procedural format such as a shell script or sequential command list? A: Far too simple. Let's pipe everything through a new and evolving middleware control stack that re-implements standard primitives like file creation but lets you introduce new bugs and edge cases in any language you choose to use, thereby ensuring the host environment also requires additional configuration! And keep that interface unstable! And definitely don't write man pages! B: TF? Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer

199 points by gk1 2y ago

I tried using nushell and had it as my default myshell for weeks but eventually it had too little familiarity and I settled on fish. So I’m probably not going to try this shell as my default shell. I think it’s going to be an uphill battle for them to get much adoption. Maybe the sandboxing will be nice and they can provide it in a way that isn’t tied to a particular shell. However sandboxing on a shell command level is too coarse grained for some things.

shykes 2y ago
Hi Ben! You should definitely not use Dagger Shell as your default shell. It's meant to complement it rather than replace it. From the post: > Dagger Shell isn’t meant to replace your system shell, but to complement it. When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
- benatkin 2y ago
  So part of the idea would be to keep my secrets out of my default shell? I’ve been looking for ways to do that but I would want specific commands to have access to them. For instance I would just want git and gh to have access to my GitHub credentials and for another program not to be able to spawn git or gh with these credentials. It would also be important not to be able to accidentally run one of these by copy and pasting something. It seems that it would need to at least be partly taken care of by my default shell for it to be usable though.
  - Imustaskforhelp 2y ago
    Hmm. Sounds as an interesting problem. But how can you really differentiate b/w a user opening git and some other program running git. I think we would need friction there , some sort of manual intervention. The best I could think of was something like bitwarden/keepassxc like cli where it requires a password and it would just straight up copy that token into github. If we are really talking / you have the source code and you want end to end security , you could theoeretically also compile git with the specific idea / implementation of whatever encrypted password manager you might use directly within the code of git / github but I know that can be an overkill for this problem.
    - benatkin 2y ago
      What I’d want is to be able to run top level commands differently from other commands and being able to have git and gh be a wrapper that injects the permissions. They could also filter the arguments and environment variables, which I know is hard to get right. Subshells and other programs would be able to run git and gh, but not with the permissions. I could even run git and gh in a container that has a volume to be able to access the directory. I think I have an idea of what this could look like and I might try and prototype it with fish and see what code parts it goes down to gauge how secure it’s likely to be.
      - Imustaskforhelp 2y ago
        Well do let me know! sounds really interesting!
- Kinrany 2y ago
  Okay, but _can_ you use it as your default shell? It's not much of a shell otherwise, is it?

agambrahma 2y ago

So basically nix-shell ?

levlaz 2y ago
There’s a tiny but of overlap in principle, but you can use dagger shell without having to learn a new language and the interface is much simpler. Give it a try and let me know which you prefer!

ledauphin 2y ago

I had missed the fact that Dagger is apparently trying to... replace? Docker for container definitions. That's a pretty big vision. I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling. At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...

levlaz 2y ago
What kind of tooling do you have in mind to replace? Also what would you prefer to see instead of bash compatible?
- mikepurvis 2y ago
  I mean if we're going full pie in the sky, fully declarative containers exist today: https://github.com/nlewo/nix2container#nix2container https://github.com/nlewo/nix2container#nix2container Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.
  - shykes 2y ago
    Dagger is fully declarative. It's just built on a dynamic declarative API, instead of a static declarative DSL. So, if you took Nix, and replaced the static scheme-like DSL with a proper API, and then built SDKs in 5 languages for that API; and then built a bash-like shell also for easy scripting; then you would start to have something that approximates Dagger.
    - mikepurvis 2y ago
      I stand corrected! I shall investigate more closely with this additional context. Thanks.

elorm 2y ago

Did Dagger pivot their product ? I seem to recall the main selling point was an Independent pipeline-as-code service. Looks like they’re trying to rebuild Docker now?

shykes 2y ago
It's the same product. Dagger Shell is a new client for the same Dagger Engine and its API. It allows you to do more from the command-line, before you have to switch to code. But the SDKs and "pipeline as code" are still there, unchanged.

a_t48 2y ago

This is very cool, I often find myself composing together a mixture of Dockerfiles and shell scripts for various images (build X Dockerfile for Y arch, save it as a base, build Z Dockerfile on top of the last thing, with W args). And then having to run it different ways with different mounts, based on if you're on a developer machine/robot/CI, running as a service/adhoc/attaching. This _looks_ like it could solve some of that messiness. I like that it can just reference the output of a build without having to muck around with tags (which are system global and can collide if you're not careful).

levlaz 2y ago
Yeah for sure, this was specifically designed to solve the problems you describe!
mikepurvis 2y ago
I have a particular hate on for the sandwich pattern where you have a Dockerfile to a "build" environment that then gets entered, and a bunch of stuff done in there, and then the results ADDed into a different Dockerfile for the publishable asset. I know multi-stage builds sort of address this need, but a lot of people aren't using them or have found cases where it's desirable to have the "running" context for the container, for example if a bunch of stuff is being volumed-in (which is now possible at build time but is janky). Or there's the copy my ansible playbooks into the container and run them there as localhost approach, which can be mitigated by what Packer does with at least running the playbook over SSH. But then Packer has no concept of multi-stage or copying in assets from other containers. There's a lot of room to do better in this space.
- a_t48 2y ago
  I don't like the sandwich either, it's too easy to accidentally miss a dependency. What I find myself doing more is building one Dockerfile as a base, and then doing `FROM $BASE` in the second to ease readability, caching, image pinning, etc.
  - mikepurvis 2y ago

wavemode 2y ago

Kinda off topic, but this page and many other pages on dagger.io seem to be loading very large GIFs that would be much better off in a compressed format (mp4 or webm).

shykes 2y ago
You're right, not sure what's going on there. Escalating internally.

jbverschoor 2y ago

Somewhat related, self promo Docker shell container - https://github.com/jrz/container-shell https://github.com/jrz/container-shell

shykes 2y ago
That looks very neat! Let me see if we can integrate it with Dagger :)

mrbluecoat 2y ago

Is the objective to get inside a container to do dev stuff? Reminds me of https://www.jetify.com/devbox https://www.jetify.com/devbox and https://flox.dev/ https://flox.dev/

hamandcheese 2y ago

My initial impression (looking at the examples, not actual usage) is that this feels like a weird half-way between Dockerfile and using actual code to define and compose software (i.e. like Nix). Not super appealing to me as a heavy Nix user.

shykes 2y ago

Someone already built a web UI for composing Dagger Shell scripts in a notebook format: http://docs.runme.dev/guide/dagger http://docs.runme.dev/guide/dagger It's really neat, I recommend checking it out.

Imustaskforhelp 2y ago
woah , that actually looks really nifty. I am imagining this with a simple cloudflare tunnel and self hosting gitlab and I am really really seeing an open source way that developers can REALLY scale. I mean docker is really great but dagger in notebook formats just seems really cool ngl.
- sourishkrout 2y ago
  love the enthusiasm. cocreator of Runme here. the best part with the Dagger + Runme combo is that it runs entirely local. this isn't just a huge with for portability. it also cuts down development cycle times significantly.
  - oulipo 2y ago
    Nice! Would be lovely to have a few "workflow examples" and tutorials which really showcase the capabilities of this setup on your website, I'm not exactly sure I understand yet in what kind of cases it really shines
    - sourishkrout 2y ago
      Absolutely. It's in the works. I'll be sure to drop additional resources once they are available. Resources available so far: - https://docs.runme.dev/guide/dagger https://docs.runme.dev/guide/dagger - https://github.com/runmedev/vscode-runme/blob/58ea9a10c00df7f3f4e0ba15a06e3503a649eff8/dagger/notebook/shell.dag https://github.com/runmedev/vscode-runme/blob/58ea9a10c00df7...

tonymet 2y ago

I'd like to use this tool as a breakpoint into a container. e.g. dagger Dockerfile L4 and it pops open a shell at that point. Then I don't need new syntax.

shepherdjerred 2y ago
earthly has something like this. it's wonderful. https://docs.earthly.dev/docs/guides/debugging https://docs.earthly.dev/docs/guides/debugging there's also https://github.com/ktock/buildg https://github.com/ktock/buildg
- tonymet 2y ago
  looks promising thanks!
shykes 2y ago
You can call 'dagger -i' to enable interactive debugging. If a step in your pipeline fails, you get an interactive terminal to inspect the current state.

randomtechguy 2y ago

I feel like it's getting harder to tell what dagger is _actually_ for these days. At first we'd hoped it could replace jenkins - it provided an alternative way to run and debug CI pipelines - right on your machine! You could write in golang and just import what you needed. The dev direction feels more scattered now, trying to replace docker, be a new shell(?), and weirdly trying to be some kind of langchain? Doing something different doesn't imply better. A new set of complicated CLI args is no better than what we started with (shell scripts, or jenkinsfiles to integrate docker builds). I'm a little bummed that the project has seemingly drifted (in my view) from the original mission.

Imustaskforhelp 2y ago
theoretically you could always use the previous version / iterate on them. I think this approach of see what sticks and them trying out a lot of different things can be nice but not in its current form. Although I have installed dagger to give it a try and I am just not sure how to make it work , the quickstart / hello world doesn't' work. No I don't want to make an AI , I just want to see what you really are. And so I do agree with your statement!
- randomtechguy 2y ago
  Yup - Ever since the introduction of llm as a core API we're basically going to stop upgrading / using the project unless things start moving in a more sane direction.
  - levlaz 2y ago
    Why is that? It’s just an extra type that has no impact on any other types. It’s a tool in your toolbox that you can use when you need it and ignore otherwise.
    - randomtechguy 2y ago
      Asked the opposite way - why is this a core API type of a buildkit interface? Having API calls to various LLMs as core functionality of a build system is just straight up weird. Doesn't fit, confuses developers on my team when they try to read/contribute to our shared build libraries, worries me about the direction the project is going. Seems strange that especially with the push for modules this was integrated as a core type. It has nothing to do with buildkit or builds.

sleepybrett 2y ago

I do all my work in containers, i'm still left asking 'why?'

imiric 2y ago

> We still use the shell, but it's no longer the universal composition system it was meant to be. Huh? When did that change? Dagger seems interesting, but this opinion that the shell and standard Unix tools are archaic is a dubious starting principle. Every now and then a project with similar ideas comes along, and whether it rejects the notion of passing unstructured data between commands, addresses the footguns and frustrations of shell scripting, or is built with a more modern and "safe" programming language, it ultimately never seems to catch on as much as traditional Unix shells and tooling has. The reality is that these tools and the design choices made 50 years ago are timeless in nature. They're deliberately lacking in features and don't attach themselves to any specific tech du jour. It's this primitive nature and the "do one thing well" philosophy that makes them infinitely composable. The same pipelines Unix nerds were building 50 years ago to solve problems are still useful today, which is remarkable when you consider how quickly technology moves. Sure, new tools are invented all the time, and they might do things better than old ones. I use `eza` instead of `ls`; `fd` instead of `find` (mostly); `rg` instead of `grep` (mostly); `fzf` is a pretty essential addition to my workflow, and so on. But the underlying principles of these tools are still the same as the tools they're replacing. They're just slightly modernized and ergonomic versions of them. Whether or not we need a `container` command, `from alpine`, or an entire new shell for that, is a separate topic. It could be argued that this could be accomplished with a few functions or standalone commands. Even if we do need this new tooling, that's great, but don't tell me that it's meant to replace a proven set of tools and workflows[1]. When containers stop being popular, will we still need this? Also, "Daggerverse" and "modules"? Great, let's bring in the npm mentality to the shell, just what I needed. [1]: Ah, they don't, it's meant to serve as a complement. Alright, fair enough. I'm lowering my pitchfork.

manno23 2y ago
[dead]

remram 2y ago

In clear terms, what would I do with this? What activity does it help me with? What program do I replace with it? "The devops operating system" does what?

shykes 2y ago
The article gives a few examples of concrete things you can do: - Build, run, and publish containers - Containerize language-specific builds - Run integration tests in ephemeral environments, in a repeatable way - Run your tools in controlled sandboxes with only the files, secrets and network access they need It also mentions what you might replace with it: > When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
- remram 2y ago
  I already build containers with Docker. I already have Makefiles that build containers and run commands in them. I already replace my complex shell scripts with Python scripts. How does Dagger help? I just can't get anything concrete from their site.
  - levlaz 2y ago
    If you’re working on your own you may not need dagger. If you’re working on a team often your shell scripts, python programs, and makes files are not always portable between local and CI, but even worse between your local machine and your colleagues local machine. This is where dagger shines because it lets you do all that stuff in a fully portable way.
    - touristtam 2y ago
      I need an example to illustrate that, because I had to read this thread to even understand what I was reading/looking at.

xedeon 2y ago

This reminds me of https://github.com/progrium/envy https://github.com/progrium/envy https://vimeo.com/131329120 https://vimeo.com/131329120

Kinrany 2y ago

How come the pipe is a method call? Does this really allow composing programs?

jatins 2y ago

Was curious what Dagger does and this is what the homepage says > The cross-platform composition engine > Build powerful software environments from modular components and simple functions. Perfect for complex builds and AI workflows. This is generic to point of being useless. Everything is a composition engine. Javascript is a composition engine. macOs is a composition engine.

levlaz 2y ago
I would lump dagger in with JS (programming language, ecosystem, runtime) and MacOS (operating system) before comparing it to any single tool. Answering "what does dagger do" can be tough, because there are very broad applications. Its always hard to describe general purpose platforms like these. Dagger is an open platform for building composable software. It has SDKs, sandboxed execution environment, observability, and now this shell interface.

manno23 2y ago

[dead]

cedws 2y ago

Looks weird, sorry. Bash pipes make intuitive sense, you’re taking the output of one process and pushing it into another. Here, the use of pipes makes little sense. Why is “from alpine” being passed into “with-exec”? What is moving between those processes?

michaelmior 2y ago
I see it as just the builder pattern that's pretty common in languages like Java. `container` is constructing the builder itself and eventually `terminal` (or other commands) use the state that has been incrementally built up in the builder to actually do something with the container.
levlaz 2y ago
Its the same idea. You are taking the instructions in with-exec and pushing it to the previous type. In this case, `from alpine` is a function attached to the container type that has many additional functions. You chain them together to do stuff. You can do it through code as if it was any other object, but this shell allows you to do things without code as well. Perhaps the example is too simple to feel useful, but being able to pipe primitives like files, directories, containers, secrets, and even any custom object makes it possible to rapidly experiment with and compose pipelines.
myaccountonhn 2y ago
You're correct. As other comments pointed out, this is an OOP builder pattern: `container().from('alpine').withExec(args: ["apk", "add"]).terminal()` If it was unix pipes, they'd be context independent tools being composed together. Dagger can be a useful but I feel a bit deceived, and it makes me worry about the rest of the quality of this product.
- levlaz 2y ago
  It’s similar to a builder pattern but not just a builder pattern because depending on the type you can indefinitely pipe context independent types. So it’s a bit of a hybrid between bash and powershell.

throwdaggeraway 2y ago

Dagger is a frustrating time sink – it's interesting for what should be boring infrastructure. I've wasted too many hours debugging low-level details of buildkit and dealing with leaky kernel abstractions instead of building actual products. Nix, Bazel, and similar alternatives may have learning curves, but they provide clearer guardrails and more predictable outcomes without exposing so many implementation details. I deeply regret the time I spent working with Dagger.

levlaz 2y ago
Thanks for sharing your experience, I'd love to hear more details. Feel free to send me an email if you'd like lev@dagger.io, it would be great to learn from your experience so we can continue to improve the platform for everyone.

GONE_KLOUT 2y ago

It would be better to not confront potential users with a long text about unix history, nobody has time to read that. Instead just write a few bullet points about what it actually does. However, I am very happy somebody finally put the prompt on TOP of a terminal window, since years I am wondering why people like to bow their head to the bottom of the screen. I guess it is the expected gesture for IT devs and admins nowadays to follow and bow your head, so this is a great help for people to understand what is the right posture in life!

baalimago 2y ago

Right, so they've re-invented the java builder pattern, but in a shell. Why not just use a Java SDK? Or go SDK with options pattern?

shykes 2y ago
Dagger does also provide a Java and Go SDK. It's all the same API underneath, so you're free to use whichever client you prefer.

la64710 2y ago

When something breaks in dagger how do you troubleshoot it?

shykes 2y ago
You have a few options: - Dagger supports opentelemetry natively: it emits traces, metrics and logs that you can send to your favorite observability tool, or to our commercial product Dagger Cloud (which has a free tier) which lets you drill down into the trace. See https://docs.dagger.io/features/visualization https://docs.dagger.io/features/visualization - The Dagger CLI also renders the otel traces, and you can browse them as well. Press escape, then navigate with up, down, enter, escape. Same drill-down functionality but straight from the CLI - You can call dagger with the --interactive flag. If you do, in case of an error you will get an interactive shell to inspect the state at time of failure. - You can also call the `Container.terminal()` function from your code, to force an interactive terminal to appear for the user at that stage. Sort of like a "debug printf" but with an interactive terminal

Annatar 2y ago

[dead]

contingencies 2y ago

A: Let's do stuff with containers. B: What's a container? A: It lets you execute a sequence of commands in an isolated and controlled environment that can be readily rebuilt, cloned or disposed of. B: How do we do that? Using a standard procedural format such as a shell script or sequential command list? A: Far too simple. Let's pipe everything through a new and evolving middleware control stack that re-implements standard primitives like file creation but lets you introduce new bugs and edge cases in any language you choose to use, thereby ensuring the host environment also requires additional configuration! And keep that interface unstable! And definitely don't write man pages! B: TF? Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer