3 ms·

Seriously LLMs are remarkable tools, but they are horribly unreliable. What tasks could such autonomous agent do (beyond what a chat bot, perhaps extended with

Seriously LLMs are remarkable tools, but they are horribly unreliable. What tasks could such autonomous agent do (beyond what a chat bot, perhaps extended with web access, already does)? I mean which task is so complex one can't just automate it with simple scripting and non critical if it goes wrong to the point of letting an AI LLM do it? BTW, running those models is rather expensive so also the task has to be quite expensive now, perhaps completed by a human.

ryanSrich 3y ago
Imagine a task that can automatically create and post ads on LinkedIn, Adsense, etc. Imagine you can also provide analytics as feedback, thus making each ad more analytics optimized. These are the types of mundane activities I dream of offloading to GPTs until the entire concept of online ads become irrelevant.
kolinko 3y ago
Example task that you cannot automate with a basic script - for each e-mail you receive, convert it into json of a given format for a given e-mail type (e.g. for delivery notifications, newsletters, meeting requests and so on). Also, tag it if it relates to a specific area of interest. Another case that a friend of mine built - summarise a stream of news from Twitter/Telegram so that no news are lost, but duplicates are removed and bad writing is reformatted.
TeMPOraL 3y ago
> I mean which task is so complex one can't just automate it with simple scripting and non critical if it goes wrong to the point of letting an AI LLM do it? Many of the things that get to deal with unstructured human inputs. Translation, summarization, data extraction, etc. The choice isn't between "reliable code" and "unreliable LLM", but rather "unreliable LLM" vs. nothing at all. But I think more important aspect is that LLMs are getting quite good at coding - meaning they can be used to automate all the tasks that you could automate with scripting, but it wouldn't be simple, or you don't know the relevant tooling well enough, or you just can't be arsed to do it yourself. For example, I've recently been using GPT-4 to automate little things in my Emacs. GPT-4 actually sucks at writing Emacs Lisp, but even having to correct it myself is much less effort than writing those bits from scratch, as the LLM saves me the step of figuring out how to do it. This is enough to make a difference between automating and not automating those little things.
- mmiliauskas 3y ago
  First we had outsourcing to Indians, now we have ChatGPT. There is almost a rule of thumb, the less you pay, the bigger pile of shit you get. At least with ChatGPT you can vet it first, but with the market being flush with 1-2 year experience devs, globally vetting will be shit too. I honestly wonder, what will start happening to all these LLMs when the training set will get over-represented with cheap, fast, crappy code written by LLMs themselves. I bet "content inbreeding" will become the topic in the future.
  - TeMPOraL 3y ago
    It might be that the moat for large LLM providers will be their ability to pay good developers to write good code solely for the purpose of feeding the training corpus of the Coding-LLM-as-a-Service.
  - valval 3y ago
    At a quick glance the sentiment that the market being flush with newer developers would somehow automatically lead to a dip in technical advancements in the near future seems completely made up. I'd believe it to be the opposite, frankly. I'd see a dip in the near future if a large portion of the more senior engineers died all of a sudden and took their knowledge with them. That's not going to happen.
    - mmiliauskas 3y ago
      Have you seen the JavaScript world?
      - TeMPOraL 3y ago
        Or Python world for that matter? It all seems to be correlated with growth - as in, the fastest-growing areas of the industry, ones that are hot and considered a career gateway, attracting the largest amount of outsiders - are the biggest circuses on wheels.
  - DebtDeflation 3y ago
ilaksh 3y ago
It depends on the LLM, but GPT-4 at least is very good at translating natural language into API calls or structured data and vice versa.
visarga 3y ago
LLMs can be used to propose candidates for chemical, medical, coding, and basically any domain where you got enough data to create a model but it is exponentially hard to search for solutions. They can be the core module in an agent that learns by RL or evolutionary methods - so the quality only depends on how much data the agent can generate. We know that when an agent can do proper exploration like AlphaGo Zero it can reach superhuman levels, it builds its own specialised dataset as it runs. For example: Evolution through Large Models - https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896
- throwuwu 3y ago
  Well this is now top of my reading list, thanks!
kohlerm 3y ago
I agree, The whole LLM agent stuff is totally overhyped. LLMs cannot really plan and all the workarounds people are currently using are not very reliable. Nice for demos, but unusable for a "product".
xtracto 3y ago
I've successfully used ChatGPT to proof-edit/fix 250+ scanned and OCRd recipes from my grandma's collection. We wanted to digitize them, but after using google Text ocr service, we couldn't get enough quality. Using GPT I was able to fix typos and output everything as Markdown in a structured way. With that, we created a nice Google doc . Now I'm using DallE to generate suitable pics for each recipe to add to the book. Its actually coming up quite good IMHO. We hope to give it as a present to my grandma on her 100 year birthday in a couple of weeks.

ryanSrich 3y ago

Imagine a task that can automatically create and post ads on LinkedIn, Adsense, etc. Imagine you can also provide analytics as feedback, thus making each ad more analytics optimized. These are the types of mundane activities I dream of offloading to GPTs until the entire concept of online ads become irrelevant.

kolinko 3y ago

Example task that you cannot automate with a basic script - for each e-mail you receive, convert it into json of a given format for a given e-mail type (e.g. for delivery notifications, newsletters, meeting requests and so on). Also, tag it if it relates to a specific area of interest. Another case that a friend of mine built - summarise a stream of news from Twitter/Telegram so that no news are lost, but duplicates are removed and bad writing is reformatted.

TeMPOraL 3y ago

> I mean which task is so complex one can't just automate it with simple scripting and non critical if it goes wrong to the point of letting an AI LLM do it? Many of the things that get to deal with unstructured human inputs. Translation, summarization, data extraction, etc. The choice isn't between "reliable code" and "unreliable LLM", but rather "unreliable LLM" vs. nothing at all. But I think more important aspect is that LLMs are getting quite good at coding - meaning they can be used to automate all the tasks that you could automate with scripting, but it wouldn't be simple, or you don't know the relevant tooling well enough, or you just can't be arsed to do it yourself. For example, I've recently been using GPT-4 to automate little things in my Emacs. GPT-4 actually sucks at writing Emacs Lisp, but even having to correct it myself is much less effort than writing those bits from scratch, as the LLM saves me the step of figuring out how to do it. This is enough to make a difference between automating and not automating those little things.

mmiliauskas 3y ago
First we had outsourcing to Indians, now we have ChatGPT. There is almost a rule of thumb, the less you pay, the bigger pile of shit you get. At least with ChatGPT you can vet it first, but with the market being flush with 1-2 year experience devs, globally vetting will be shit too. I honestly wonder, what will start happening to all these LLMs when the training set will get over-represented with cheap, fast, crappy code written by LLMs themselves. I bet "content inbreeding" will become the topic in the future.
- TeMPOraL 3y ago
  It might be that the moat for large LLM providers will be their ability to pay good developers to write good code solely for the purpose of feeding the training corpus of the Coding-LLM-as-a-Service.
- valval 3y ago
  At a quick glance the sentiment that the market being flush with newer developers would somehow automatically lead to a dip in technical advancements in the near future seems completely made up. I'd believe it to be the opposite, frankly. I'd see a dip in the near future if a large portion of the more senior engineers died all of a sudden and took their knowledge with them. That's not going to happen.
  - mmiliauskas 3y ago
    Have you seen the JavaScript world?
    - TeMPOraL 3y ago
      Or Python world for that matter? It all seems to be correlated with growth - as in, the fastest-growing areas of the industry, ones that are hot and considered a career gateway, attracting the largest amount of outsiders - are the biggest circuses on wheels.
- DebtDeflation 3y ago

ilaksh 3y ago

It depends on the LLM, but GPT-4 at least is very good at translating natural language into API calls or structured data and vice versa.

visarga 3y ago

LLMs can be used to propose candidates for chemical, medical, coding, and basically any domain where you got enough data to create a model but it is exponentially hard to search for solutions. They can be the core module in an agent that learns by RL or evolutionary methods - so the quality only depends on how much data the agent can generate. We know that when an agent can do proper exploration like AlphaGo Zero it can reach superhuman levels, it builds its own specialised dataset as it runs. For example: Evolution through Large Models - https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896

throwuwu 3y ago
Well this is now top of my reading list, thanks!

kohlerm 3y ago

I agree, The whole LLM agent stuff is totally overhyped. LLMs cannot really plan and all the workarounds people are currently using are not very reliable. Nice for demos, but unusable for a "product".

xtracto 3y ago

I've successfully used ChatGPT to proof-edit/fix 250+ scanned and OCRd recipes from my grandma's collection. We wanted to digitize them, but after using google Text ocr service, we couldn't get enough quality. Using GPT I was able to fix typos and output everything as Markdown in a structured way. With that, we created a nice Google doc . Now I'm using DallE to generate suitable pics for each recipe to add to the book. Its actually coming up quite good IMHO. We hope to give it as a present to my grandma on her 100 year birthday in a couple of weeks.