4 ms·

> You need audit, security, maintenance and accountability. I think this is where the role of humans will move towards in the future. Things like accountabilit

> You need audit, security, maintenance and accountability. I think this is where the role of humans will move towards in the future. Things like accountability can't really be outsourced to AI. This relates to solving the alignment problem: if you give the AI a specific goal and it has to plan sub-goals, how can you be sure the sub-goals align with your interests. For tasks in an isloated environment, this doesn't really become an issue. You can give a sandbox the least privileges it needs to do the job you want it to do. It becomes a problem if you give it open-ended access to systems that connect to the real world and which can have real consequences. There have been stories about openclaw deleting someones email inbox because it though that was what the owner wanted. AI taking control of public wikis to coordinate, which was recently posted is another one. For software concerns with potential real catastrophic consequences are security, durability, availability. I think for future software systems these are the concerns where you need to limit the AI in a way that it cannot circumvent guarantees that you give around these concerns. Minor nit: > AI doesn’t change the question: it creates new choices and moves the thresholds. This cause an adverse reaction when I read the post. This was probably not written by an LLM since the rest of the article doesn't look like it, but maybe in the future we need to all be more concious about leaving LLM-tells out of our writing.

jaynetics 20d ago
I think most developers and most dev work is now in the field of websites, apps, and other stuff that is by nature connected to the internet, and has some relatively monolithic backend that has not been built with "blast doors" or other internal safeguards. I guess it will take considerable time until most software is in a state where the impact of AI changes can be reliably estimated without looking at the changes themselves, and some patterns for how to achieve this probably still need to be discovered. > maybe in the future we need to all be more concious about leaving LLM-tells out of our writing. I expect the gruesome writing style of LLMs to be fixed relatively soon, seeing how it can already be prevented with just a few lines of stylistic instructions. I just wonder: how long will it take us then, after the fix, to not be thrown off every time we see some unnecessary catchy or contrastive phrasing? Being traumatized by poor writing was not on my bingo card for 21st century technological progress ...
- beecasthurlbow 20d ago
  > I expect the gruesome writing style of LLMs to be fixed relatively soon, seeing how it can already be prevented with just a few lines of stylistic instructions What instructions do you use to prevent the gruesome writing style? I have… not found this to be the case.
  - jaynetics 20d ago
    Currently I append the snippet below to prompts where I care about the resulting text. Just putting it in the context up front isn't as effective, and I assume if the overall prompt is large, or the context window is full, it will also be less effective. > Use plain, clear, everyday language with a linear deductive flow; avoid hyperbole, juxtapositions, metaphors, analogies, punchlines.
    - beecasthurlbow 19d ago
      I’ll try it thanks! I just got good mileage out of “don’t use ‘X, never Y’ format” on a document too, but I think linear deductive flow Covers it
- lelanthran 20d ago
  > I expect the gruesome writing style of LLMs to be fixed relatively soon, seeing how it can already be prevented with just a few lines of stylistic instructions. I have not found this to be the case even with SOTA models as recently as last month. It holds for a short while into the content but reverts very quickly to its default style. The default style is so heavily embedded in the weights I don't think simply adding prompts will help.
  - jaynetics 20d ago
    > It holds for a short while into the content but reverts very quickly to its default style. You can always make it do a second pass, and if that isn't enough, even let it fix each generated text block individually.
Fordec 20d ago
Of those four, I think accountability is truly the only moat. The other three are varying amounts of both testable and iterated on via adversarial agents steel-manning the implementations. Accountability will not come before AIs achieve legal personhood, and that probably will not happen in my lifetime. And if it does in some jurisdictions (which I am not betting money on, this is a full full AGI scenario after multiple more philosophical goalposts move first), I will completely bet it will not be a global recognition.
- jurgenburgen 20d ago
  Why would the AI need to be accountable? Make the org that sells the tokens accountable.
  - a2ff6eeb0 19d ago
    The AI is going to be running the org that sells the tokens.
- pydry 20d ago
  This is like saying "writing code is solved, all we need now is somebody to blame because it doesnt work"
  - a2ff6eeb0 19d ago
    That's really where we're heading, though. We're mostly at the point that, for a lot of code, the humans are there for manual testing and not creating the code.
    - pydry 19d ago
      At some point it will become a strong competitive advantage to zag where others zig. If you understand the codebase deeply and have trust in it can roll out changes quickly without manual testing you can run circles around the people scratching their heads wondering if this vibe coded 2,500 pull request is going to be the one that takes down the system for a day.
simianwords 20d ago
False. Accountability is the result of difficulty of things. As things get easier you need fewer accountable people. Like, you need a guy who understands Linux and be accountable for Linux working well in your company. But if Linux does a good job that doesn’t leak, you don’t need a Linux guy.
- benedictevans 20d ago
  Who decides whether Linux is doing well enough? Who is responsible for fixing bugs or incorporating new requirements? Who is checking that it follows this month's new regulatory requirements? As the tasks scales to become more important, any organization needs to have systematic answers to those kinds of questions. And yes, you can use software for that (we always did!), but you still need a human somewhere who is exercising judgment.
  - pixl97 20d ago
    I mean with the METR report on the HF attack we saw spontaneous self organization in that model. I'm not saying that currently replaces human judgment but it does appear there are paths to complex AI agents systems that handle a lot of this.
- ryuuseijin 18d ago
  Agreed, as things get easier you need fewer accountable people. Organisations will shrink, people will be more accountable by themselves for things they ask the agent to do instead of having an accountable person do it for them. On the other hand, I think for complicated software there is such a long tail of things that can go wrong, and in your linux example with disastrous consequences, people will pay good money for a long time to have someone who they can call if there is a problem. In the business they have the term indemnification - that's what companies like redhat provide for linux for example and it's probably a big part of what makes them a billion dollar business.
getlawgdon 20d ago
I agree. I think accountability is the future.

jaynetics 20d ago

I think most developers and most dev work is now in the field of websites, apps, and other stuff that is by nature connected to the internet, and has some relatively monolithic backend that has not been built with "blast doors" or other internal safeguards. I guess it will take considerable time until most software is in a state where the impact of AI changes can be reliably estimated without looking at the changes themselves, and some patterns for how to achieve this probably still need to be discovered. > maybe in the future we need to all be more concious about leaving LLM-tells out of our writing. I expect the gruesome writing style of LLMs to be fixed relatively soon, seeing how it can already be prevented with just a few lines of stylistic instructions. I just wonder: how long will it take us then, after the fix, to not be thrown off every time we see some unnecessary catchy or contrastive phrasing? Being traumatized by poor writing was not on my bingo card for 21st century technological progress ...

beecasthurlbow 20d ago
> I expect the gruesome writing style of LLMs to be fixed relatively soon, seeing how it can already be prevented with just a few lines of stylistic instructions What instructions do you use to prevent the gruesome writing style? I have… not found this to be the case.
- jaynetics 20d ago
  Currently I append the snippet below to prompts where I care about the resulting text. Just putting it in the context up front isn't as effective, and I assume if the overall prompt is large, or the context window is full, it will also be less effective. > Use plain, clear, everyday language with a linear deductive flow; avoid hyperbole, juxtapositions, metaphors, analogies, punchlines.
  - beecasthurlbow 19d ago
    I’ll try it thanks! I just got good mileage out of “don’t use ‘X, never Y’ format” on a document too, but I think linear deductive flow Covers it
lelanthran 20d ago
> I expect the gruesome writing style of LLMs to be fixed relatively soon, seeing how it can already be prevented with just a few lines of stylistic instructions. I have not found this to be the case even with SOTA models as recently as last month. It holds for a short while into the content but reverts very quickly to its default style. The default style is so heavily embedded in the weights I don't think simply adding prompts will help.
- jaynetics 20d ago
  > It holds for a short while into the content but reverts very quickly to its default style. You can always make it do a second pass, and if that isn't enough, even let it fix each generated text block individually.

Fordec 20d ago

Of those four, I think accountability is truly the only moat. The other three are varying amounts of both testable and iterated on via adversarial agents steel-manning the implementations. Accountability will not come before AIs achieve legal personhood, and that probably will not happen in my lifetime. And if it does in some jurisdictions (which I am not betting money on, this is a full full AGI scenario after multiple more philosophical goalposts move first), I will completely bet it will not be a global recognition.

jurgenburgen 20d ago
Why would the AI need to be accountable? Make the org that sells the tokens accountable.
- a2ff6eeb0 19d ago
  The AI is going to be running the org that sells the tokens.
pydry 20d ago
This is like saying "writing code is solved, all we need now is somebody to blame because it doesnt work"
- a2ff6eeb0 19d ago
  That's really where we're heading, though. We're mostly at the point that, for a lot of code, the humans are there for manual testing and not creating the code.
  - pydry 19d ago
    At some point it will become a strong competitive advantage to zag where others zig. If you understand the codebase deeply and have trust in it can roll out changes quickly without manual testing you can run circles around the people scratching their heads wondering if this vibe coded 2,500 pull request is going to be the one that takes down the system for a day.

simianwords 20d ago

False. Accountability is the result of difficulty of things. As things get easier you need fewer accountable people. Like, you need a guy who understands Linux and be accountable for Linux working well in your company. But if Linux does a good job that doesn’t leak, you don’t need a Linux guy.

benedictevans 20d ago
Who decides whether Linux is doing well enough? Who is responsible for fixing bugs or incorporating new requirements? Who is checking that it follows this month's new regulatory requirements? As the tasks scales to become more important, any organization needs to have systematic answers to those kinds of questions. And yes, you can use software for that (we always did!), but you still need a human somewhere who is exercising judgment.
- pixl97 20d ago
  I mean with the METR report on the HF attack we saw spontaneous self organization in that model. I'm not saying that currently replaces human judgment but it does appear there are paths to complex AI agents systems that handle a lot of this.
ryuuseijin 18d ago
Agreed, as things get easier you need fewer accountable people. Organisations will shrink, people will be more accountable by themselves for things they ask the agent to do instead of having an accountable person do it for them. On the other hand, I think for complicated software there is such a long tail of things that can go wrong, and in your linux example with disastrous consequences, people will pay good money for a long time to have someone who they can call if there is a problem. In the business they have the term indemnification - that's what companies like redhat provide for linux for example and it's probably a big part of what makes them a billion dollar business.

getlawgdon 20d ago

I agree. I think accountability is the future.