3 ms·
One thing that radicalized me was building an agent that tested network connectivity for our fleet. Early on, in like 2021, I deployed a little mini-fleet of of
by tptacek 1y ago
One thing that radicalized me was building an agent that tested network connectivity for our fleet. Early on, in like 2021, I deployed a little mini-fleet of off-network DNS probes on, like, Vultr to check on our DNS routing, and actually devising metrics for them and making the data that stuff generated legible/operationalizable was annoying and error prone. But you can give basic Unix network tools --- ping, dig, traceroute --- to an agent and ask it for a clean, usable signal, and they'll do a reasonable job! They know all the flags and are generally better at interpreting tool output than I am.
I'm not saying that the agent would do a better job than a good "hardcoded" human telemetry system, and we don't use agents for this stuff right now. But I do know that getting an agent across the 90% threshold of utility for a problem like this is much, much easier than building the good telemetry system is.
- foobarian 1y agoHonestly the top AI use case for me right now is personal throwaway dev tools. Where I used to write shell oneliners with dozen pipes including greps and seds and jq and other stuff, now I get an AI to write me a node script and throw in a nice Web UI to boot. Edit: reflecting on what the lesson is here, in either case I suppose we're avoiding the pain of dealing with Unix CLI tools :-D
- jacquesm 1y agoInteresting. You have to wonder if all the tools that is based on would have been written in the first place if that kind of thing had been possible all along. Who needs 'grep' when you can write a prompt?
- tptacek 1y agoMy long running joke is that the actual good `jq` is just the LLM interface that generates `jq` queries; 'simonw actually went and built that.
- dannyobrien 1y agohttps://github.com/simonw/llm-jq https://github.com/simonw/llm-jq for those following along at home https://github.com/simonw/llm-cmd https://github.com/simonw/llm-cmd is what i use as the "actually good ffmpeg etc front end" and just to toot my own horn, I hand Simon's `llm` command lone tool access to its own todo list and read/write access to the cwd with my own tools, https://github.com/dannyob/llm-tools-todo https://github.com/dannyob/llm-tools-todo and https://github.com/dannyob/llm-tools-patch https://github.com/dannyob/llm-tools-patch Even with just these and no shell access it can get a lot done, because these tools encode the fundamental tricks of Claude Code ( I have `llmw` aliased to `llm --tool Patch --tool Todo --cl 0` so it will have access to these tools and can act in a loop, as Simon defines an agent. )
- a-french-anon 1y agoTried gron (https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron) a bit? If you know your UNIX, I think it can replace jq in a lot of cases. And when it can't, well, you can reach for Python, I guess.
- agumonkey 1y agoIt's highly plausible that all we assumed was good design / engineering will disappear if LLMs/Agents can produce more without having the be modular. (sadly)
- jacquesm 1y agoThere is some kind of parallel behind 'AI' and 'Fuzzy Logic'. Fuzzy logic to me always appeared like a large number of patches to get enough coverage for a system to work even if you didn't understand it. AI just increases the number of patches to billions.
- agumonkey 1y agotrue, there's often a point where your system becomes a blurry miracle
- andai 1y agoCould you give some examples? I'm having the AI write the shell scripts, wondering if I'm missing out on some comfy UIs...
- foobarian 1y agoI was debugging a service that was spitting out a particular log line. I gave Copilot an example line, told it to write a script that tails the log line and serves a UI via port 8080 with a table of those log lines parsed and printed nicely. Then I iterated by adding filter buttons, aggregation stats, simple things like that. I asked it to add a "clear" button to reset the UI. I probably would not even have done this without an AI because the CLI equivalent would be parsing out and aggregating via some form of uniq -c | sort -n with a bunch of other tuning and it would be too much trouble.
- sumedh 1y agoIt can be anything. It depends on what you want to do with the output. You can have a simple dashboard site which collects the data from our shell scripts and shows your a summary or red/green signals so that you can focus on things which are interested in.
- zahlman 1y ago> They know all the flags and are generally better at interpreting tool output than I am. In the toy example, you explicitly restrict the agent to supply just a `host`, and hard-code the rest of the command. Is the idea that you'd instead give a `description` something like "invoke the UNIX `ping` command", and a parameter described as constituting all the arguments to `ping`?
- tptacek 1y agoHonestly, I didn't think very hard about how to make `ping` do something interesting here, and in serious code I'd give it all the `ping` options (and also run it in a Fly Machine or Sprite where I don't have to bother checking to make sure none of those options gives code exec). It's possible the post would have been better had I done that; it might have come up with an even better test. I was telling a friend online that they should bang out an agent today, and the example I gave her was `ps`; like, I think if you gave a local agent every `ps` flag, it could tell you super interesting things about usage on your machine pretty quickly.
- chickensong 1y agoI hadn't given much thought to building agents, but the article and this comment are inspiring, thx. It's interesting to consider agents as a new kind of interface/function/broker within a system.
- 0xbadcafebee 1y ago> I'm not saying that the agent would do a better job than a good "hardcoded" human telemetry system, and we don't use agents for this stuff right now. And that's why I won't touch 'em. All the agents will be abandoned when people realize their inherent flaws (security, reliability, truthfulness, etc) are not worth the constant low-grade uncertainty. In a way it fits our times. Our leaders don't find truth to be a very useful notion. So we build systems that hallucinate and act unpredictably, and then invest all our money and infrastructure in them. Humans are weird.
- simonw 1y agoSome of us have been happily using agentic coding tools (Claude Code etc) since February and we're still not abandoning them for their inherent flaws.
- crystal_revenge 1y agoThe problem with statements like these is that I work with people who make the same claims, but are slowly building useless, buggy monstrosities that for various reasons nobody can/will call out. Obviously I’m reasonably willing to believe that you are an exception. However every person I’ve interacted with who makes this same claim has presented me with a dumpster fire and expected me to marvel at it.
- simonw 1y agoI'm not going to dispute your own experience with people who aren't using this stuff effectively, but the great thing about the internet is that you can use it to track the people who are making the very best use of any piece of technology.
- crystal_revenge 1y agoThis line of reasoning is smelling pretty "no true Scotsman" to me. I'm sure there were amazing ColdFusion devs, but that hardly justifies the use of the technology. Likewise "This tool works great on the condition that you need to hire a Simon Willison level dev" is almost a fault. I'm pretty confident you could squeeze some juice out of a Markov Chain (ignoring, of course, that decoder-only LLMs are basically fancy MCs). In a weird way it sort of reminds me of Common Lisp. When I was younger I thought it was the most beautiful language and a shame that it wasn't more widely adopted. After a few decades in the field I've realized it's probably for the best since the average dev would only use it to create elaborate foot guns.