7 ms·

There was surprisingly little information on how they actually do this, but we run our business with a large number of, what we call "AI employees" in addition

by mchusma 14d ago

There was surprisingly little information on how they actually do this, but we run our business with a large number of, what we call "AI employees" in addition to regular employees, and they act in interesting ways. We've been building out orchestration tools to handle this, and we'll probably do a write-up or blog post on it soon. May even open source some of it. The preview is that the problem with most agents (and this includes frameworks like Grokbot and Openclaw and Hermes) is that for many of them, they're black boxes. They say they learn or improve, but it's a black box in what they do. Getting agents to reliably do things is hard, and getting agents to build out software tools to help themselves improve and do better over time is also hard. Our approach at a high level is pretty simple: every single AI employee is a standalone GitHub repo that shares some characteristics, but we direct them to build as much software as possible to make their goal as easy and reliable to manage as possible. Then we have a shared communication layer for bots across the company to interact with humans and AI. We have decided to organize these like departments similar to the way you might hire out humans. I'm not 100% sure if that's the best approach, but I will say it's been easier for people to understand because they're more mentally easily able to traverse the bot org chart if it somewhat reflects a traditional business org chart. Each of these AI employees has specific sets of goals and KPIs, instructions that they manage the business with manager bots. We have layers of management, which we actually have found helpful. We also run different bots with different models and harnesses, and some using different models and harnesses to check the work before anything can get done, along with lots and lots of testing. Every single time, actions have a massive amount of tests based off of previous failures to prevent failures in the future. Sorry for rambling. I do think this is a very interesting space. I didn't see anything interesting in Pion that was public on this website, but I do anticipate that more companies will be "AI and software first," as in the substrate of the company is basically a software application powered by autonomous agents, with humans as a fallback.

orourke 14d ago
This is fascinating. What is the split between human and AI labor? What kinds of tasks are the bots doing? How much autonomy do they have to make decisions (i.e. spending money, issuing refunds, touch cloud infra, etc)? I'd love to learn more about how you do this.
- mchusma 14d ago
  I hope to post something within a few weeks. But our philosophy is generally if it can be done deterministically with software (eg run payroll on autopilot via an api to gusto) then do that. If it can be done by an ai agent, do it with that but add as much software as possible to make it reliable at that thing. And the ai agents are all tuned to escalate to humans as needed. Then there is also just things that are completely human. A typical example of something that is AI vs human is the AI most commonly operates like “managers”. For example, reviewing transcripts of every demo call, compiling results, figuring out insights, learnings that need to update our company docs, feedback to humans (who run the demos). We are big fans of having AI agents “own” koi’s because now anytime we say “we really should be doing this” we try to set it up on the spot. The “downside” here is that I do occasionally get busy, and if I’m the only one who can approve or unstick one of these bots, it just keeps harassing me until it gets done. This is a sign generally that I need to hire someone to own a set of bots.
  - gong_hits 14d ago
    [dead]
  - SoftTalker 14d ago
    Another downside is that if the agent goes rogue on the payroll you'll have a lot of angry humans and possibly major legal problems also.
    - fragmede 14d ago
      Embezzlement is much, much older than LLMs.
      - jdiff 13d ago
        You can convict a human. What do you do when your LLM blows all your money on paperclips?
ianberdin 14d ago
What worries me most are questions like this: these autonomous AI employees or automations - it doesn't change the essence - consume a lot of LLM tokens. Please tell me, how do you pay for this? Do you use, for example, the Anthropic or OpenAI API directly, or do you connect, for example, a Codex subscription?
- lnenad 14d ago
  If you use a bajillion tokens your economical approach is to self host.
  - noir_lord 14d ago
    There is a point where those two lines do cross but with them effectively subsidising token cost by burning debt, it's further away than it will be at some point. That said I use local only models purely because I don't want to use remote models, never having to think about token costs is worth it and no one is training anything on my data either.
- mchusma 14d ago
  Our control panel is on a server but individual agents actually are run on anyone’s machine. This allows us to use the native harnesses including subscriptions. Yes it uses a lot more tokens, but i have have Claude $200, OpenAI $200, and SuperGrok Heavy $300 (or whatever it is called) that includes Cursor Ultra. My machine runs most of them, but some other team members have agents running on their machines using 1-2 $200/mo subs. I would say this setup probably costs us about $1,000/month total. (I’m excluding traditional engineering use of LLMs from this number. This is the cost of all the “AI employees”. One reason we did it this way was to use subs.
  - grvdrm 14d ago
    Do I understand correctly that by using your machine and Claude Code or something like it you are avoiding API pricing?
    - 14d ago
jgilias 14d ago
At $DAYJOB we do something very similar shape-wise. I wonder - it sounds like you have a dedicated agent comms plane? In our case we found that the easiest and most straightforward was to just use our default company chat app directly for this. Because most of the context that the agent workers need to do work is there, but also, it’s just much easier for teams to conceptualise an agent colleague if it just hangs out in their channels. What do you do here, and how’s it going?
- mchusma 14d ago
  We do have a control panel, but it’s in effect a server that has things in a database. All chats, tasks, assignments are all there. This felt easier to debug and manage versus putting it all in slack, although we considered it. I don’t think anything we are doing is particularly “fancy”, but basically one agent sends a message to another one. It saves the message in the db, then adds the message to that other bot same as any other user chat. We have an internal website where anyone in the company can see the bits they are authorized to see and can see all chats. These AI workers are single threaded, but we see that as more of a feature than a bug (minimize complexity). They all work their way through a shared task list, which is just another table in our remote server sqllite.
  - jgilias 14d ago
    What harnesses do you use? Ours is basically Claude in a box. There’s some complexity because of that, but the advantage is that it’s very flexible and people who have a bunch of Claude-shaped skills can just basically give those to an agent. I’m thinking to take a deeper look at Pi. I’m really liking that project.
  - messh 14d ago
    how do you decide to start a new agent, and when to kill? also... do you use some works-tealing style task board, or otherwise how would the agents get new tasks.
idiotsecant 14d ago
The first ASI will use a vast army of middle managers as it's neurons. We won't be fighting terminators, we'll be submitting TPS reports to skynet.
- dofm 14d ago
  It can't be bargained with. It can't be reasoned with. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop... ever, until you go ahead and come in on Saturday.
classified 14d ago
> we direct them to build as much software as possible That's not a cure-all. Sometimes not writing software is the better choice.
- tomtomtom777 14d ago
  This is exactly the problem we're ignoring. In the end, most software is a necessary evil. It solves a problem that shouldn't be there. In the end it's no different than healthcare or prisons. We don't need as much as possible. We need as little as possible. This automation isn't actually helping us.
- avereveard 14d ago
  A tool aggregation layer made of code is a good way to save tokens Composition is an issue only as long as one keep demanding tool calls in json. If tools are goal predicates in prolog, it's easier.
- stillpointlab 14d ago
  That may or may not be true, but the calculus has changed with agents. A process that was better manual for a human org may not be better for an agent org. I think what is interesting here is that the industry is in this experimentation flux. Some people will choose to automate and write the software, others will not. And aggregate over the industry and over time we will learn. So just saying "sometimes it works and sometimes it doesn't" isn't really adding value, compared to the people actually experimenting and sharing the results.
StilesCrisis 14d ago
Is this actually cost-effective versus hiring a few humans? Seems like a huge amount of tokens in use.
- sevenzero 14d ago
  The whole premise of AI is to replace the human in the loop. People who go out of their way to set up this kind of stuff rather than just hiring a person dont even think about just hiring a person.
  - fragmede 14d ago
    Why would you assume that?
- resonious 14d ago
  I think it's more of an experiment at this point. We don't know whether it's viable or not.
  - pcthrowaway 13d ago
    Did you respond on your alt account or are you answering for the top-level poster for some other reason?
maxnevermind 14d ago
What type of roles "AI employees" play, can it be any position in your company or you limit it to something specific? What is your goal, are you trying to find out if fully autonomous bots are more efficiently help to deliver projects than when people drive them or is it something else?
theplumber 14d ago
Don’t you end up with useless slop? That was my experience with these kinds of systems. In theory they should work but in practice unless hand holded they just produce slop and waste.
alexhans 14d ago
> We have decided to organize these like departments similar to the way you might hire out humans. I'm not 100% sure if that's the best approach, but I will say it's been easier for people to understand because they're more mentally easily able to traverse the bot org chart if it somewhat reflects a traditional business org chart I think this way of thinking is going to be very important to drive adoption of AI systems because the human analogies benefit from the pre-existing domain knowledge and expectations of people. I like using the exam analogy for evals as a qualifier for work for your "AI hires" so you can trust them to work on a specific domain. I'd be quite curious to see what your approach to evals/testing/tracing and agent/system mutation is.
root_axis 13d ago
> We have layers of management, which we actually have found helpful Interesting discovery.
edg5000 13d ago
Can you mention some specific tasks/KPIs that you've found bots can do a large amount of work autonomously on? By large I mean something that would take Claude Code or Codex at least a few hours.

by mchusma 14d ago

orourke 14d ago

This is fascinating. What is the split between human and AI labor? What kinds of tasks are the bots doing? How much autonomy do they have to make decisions (i.e. spending money, issuing refunds, touch cloud infra, etc)? I'd love to learn more about how you do this.

mchusma 14d ago
I hope to post something within a few weeks. But our philosophy is generally if it can be done deterministically with software (eg run payroll on autopilot via an api to gusto) then do that. If it can be done by an ai agent, do it with that but add as much software as possible to make it reliable at that thing. And the ai agents are all tuned to escalate to humans as needed. Then there is also just things that are completely human. A typical example of something that is AI vs human is the AI most commonly operates like “managers”. For example, reviewing transcripts of every demo call, compiling results, figuring out insights, learnings that need to update our company docs, feedback to humans (who run the demos). We are big fans of having AI agents “own” koi’s because now anytime we say “we really should be doing this” we try to set it up on the spot. The “downside” here is that I do occasionally get busy, and if I’m the only one who can approve or unstick one of these bots, it just keeps harassing me until it gets done. This is a sign generally that I need to hire someone to own a set of bots.
- gong_hits 14d ago
  [dead]
- SoftTalker 14d ago
  Another downside is that if the agent goes rogue on the payroll you'll have a lot of angry humans and possibly major legal problems also.
  - fragmede 14d ago
    Embezzlement is much, much older than LLMs.
    - jdiff 13d ago
      You can convict a human. What do you do when your LLM blows all your money on paperclips?

ianberdin 14d ago

What worries me most are questions like this: these autonomous AI employees or automations - it doesn't change the essence - consume a lot of LLM tokens. Please tell me, how do you pay for this? Do you use, for example, the Anthropic or OpenAI API directly, or do you connect, for example, a Codex subscription?

lnenad 14d ago
If you use a bajillion tokens your economical approach is to self host.
- noir_lord 14d ago
  There is a point where those two lines do cross but with them effectively subsidising token cost by burning debt, it's further away than it will be at some point. That said I use local only models purely because I don't want to use remote models, never having to think about token costs is worth it and no one is training anything on my data either.
mchusma 14d ago
Our control panel is on a server but individual agents actually are run on anyone’s machine. This allows us to use the native harnesses including subscriptions. Yes it uses a lot more tokens, but i have have Claude $200, OpenAI $200, and SuperGrok Heavy $300 (or whatever it is called) that includes Cursor Ultra. My machine runs most of them, but some other team members have agents running on their machines using 1-2 $200/mo subs. I would say this setup probably costs us about $1,000/month total. (I’m excluding traditional engineering use of LLMs from this number. This is the cost of all the “AI employees”. One reason we did it this way was to use subs.
- grvdrm 14d ago
  Do I understand correctly that by using your machine and Claude Code or something like it you are avoiding API pricing?
  - 14d ago

jgilias 14d ago

At $DAYJOB we do something very similar shape-wise. I wonder - it sounds like you have a dedicated agent comms plane? In our case we found that the easiest and most straightforward was to just use our default company chat app directly for this. Because most of the context that the agent workers need to do work is there, but also, it’s just much easier for teams to conceptualise an agent colleague if it just hangs out in their channels. What do you do here, and how’s it going?

mchusma 14d ago
We do have a control panel, but it’s in effect a server that has things in a database. All chats, tasks, assignments are all there. This felt easier to debug and manage versus putting it all in slack, although we considered it. I don’t think anything we are doing is particularly “fancy”, but basically one agent sends a message to another one. It saves the message in the db, then adds the message to that other bot same as any other user chat. We have an internal website where anyone in the company can see the bits they are authorized to see and can see all chats. These AI workers are single threaded, but we see that as more of a feature than a bug (minimize complexity). They all work their way through a shared task list, which is just another table in our remote server sqllite.
- jgilias 14d ago
  What harnesses do you use? Ours is basically Claude in a box. There’s some complexity because of that, but the advantage is that it’s very flexible and people who have a bunch of Claude-shaped skills can just basically give those to an agent. I’m thinking to take a deeper look at Pi. I’m really liking that project.
- messh 14d ago
  how do you decide to start a new agent, and when to kill? also... do you use some works-tealing style task board, or otherwise how would the agents get new tasks.

idiotsecant 14d ago

The first ASI will use a vast army of middle managers as it's neurons. We won't be fighting terminators, we'll be submitting TPS reports to skynet.

dofm 14d ago
It can't be bargained with. It can't be reasoned with. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop... ever, until you go ahead and come in on Saturday.

classified 14d ago

> we direct them to build as much software as possible That's not a cure-all. Sometimes not writing software is the better choice.

tomtomtom777 14d ago
This is exactly the problem we're ignoring. In the end, most software is a necessary evil. It solves a problem that shouldn't be there. In the end it's no different than healthcare or prisons. We don't need as much as possible. We need as little as possible. This automation isn't actually helping us.
avereveard 14d ago
A tool aggregation layer made of code is a good way to save tokens Composition is an issue only as long as one keep demanding tool calls in json. If tools are goal predicates in prolog, it's easier.
stillpointlab 14d ago
That may or may not be true, but the calculus has changed with agents. A process that was better manual for a human org may not be better for an agent org. I think what is interesting here is that the industry is in this experimentation flux. Some people will choose to automate and write the software, others will not. And aggregate over the industry and over time we will learn. So just saying "sometimes it works and sometimes it doesn't" isn't really adding value, compared to the people actually experimenting and sharing the results.

StilesCrisis 14d ago

Is this actually cost-effective versus hiring a few humans? Seems like a huge amount of tokens in use.

sevenzero 14d ago
The whole premise of AI is to replace the human in the loop. People who go out of their way to set up this kind of stuff rather than just hiring a person dont even think about just hiring a person.
- fragmede 14d ago
  Why would you assume that?
resonious 14d ago
I think it's more of an experiment at this point. We don't know whether it's viable or not.
- pcthrowaway 13d ago
  Did you respond on your alt account or are you answering for the top-level poster for some other reason?

maxnevermind 14d ago

What type of roles "AI employees" play, can it be any position in your company or you limit it to something specific? What is your goal, are you trying to find out if fully autonomous bots are more efficiently help to deliver projects than when people drive them or is it something else?

theplumber 14d ago

Don’t you end up with useless slop? That was my experience with these kinds of systems. In theory they should work but in practice unless hand holded they just produce slop and waste.

alexhans 14d ago

> We have decided to organize these like departments similar to the way you might hire out humans. I'm not 100% sure if that's the best approach, but I will say it's been easier for people to understand because they're more mentally easily able to traverse the bot org chart if it somewhat reflects a traditional business org chart I think this way of thinking is going to be very important to drive adoption of AI systems because the human analogies benefit from the pre-existing domain knowledge and expectations of people. I like using the exam analogy for evals as a qualifier for work for your "AI hires" so you can trust them to work on a specific domain. I'd be quite curious to see what your approach to evals/testing/tracing and agent/system mutation is.

root_axis 13d ago

> We have layers of management, which we actually have found helpful Interesting discovery.

edg5000 13d ago

Can you mention some specific tasks/KPIs that you've found bots can do a large amount of work autonomously on? By large I mean something that would take Claude Code or Codex at least a few hours.