4 ms·

One area that's getting close to hitting the mainstream is AI agents that control web browsers or desktop apps. Basically, you tell your AI what you want to do,

One area that's getting close to hitting the mainstream is AI agents that control web browsers or desktop apps. Basically, you tell your AI what you want to do, and it does the computer work for you, showing the results the way you like. You won't have to dig through tons of useless apps anymore. It'll search, sort, and create feeds for you. This is what scares advertisers the most - when we start saying, "Just talk to my AI." God knows I'd love an AI that can fill tax forms for me. Operating on top of UI it is compatible with all devices, kind of like an android robot would fit well with all human spaces and tools. There is plenty of screen capture with narration to learn from. You can generate more training data by essentially giving free reign over a VM to an AI agent. Ask it to do office work. We can also collect this kind of data very easily.

falcor84 3y ago
I recall imagining something s quite like this as a kid when I first encountered the term "User Agent". I'd be all in on this if I knew I had full control and explainability of its decisions, but am afraid that reality will be a lot more depressing than what I imagine.
crooked-v 3y ago
For a very immediate example, I can already see a clear path to somebody coming up with an LLM service that will automatically filter out political spam texts and emails. (Maybe it already exists and I just haven't seen it yet.)
- kaibee 3y ago
  I am endlessly frustrated every day that this is not a thing yet. I also don't want to send all of my emails to OpenAI though.
  - samus 3y ago
    I recently did a Kaggle experiment and finetuned a LLM to classify SMS as SPAM or HAM. It was not so difficult; following the text classification example in the Hugginface transformers docs was enough. Without trying hard at all, accuracy was above 90%. Impressive even though even the messages marked as HAM seemed to be quite trash. The model itself was small - few hundred million parameters only. Models of this size run well on CPUs and an even smaller might have workd too. And non-neural classifiers might also do fine. You could do something similar and unleash it on your mailbox. The tricky part are integrations. At work, I don't even bother since interacting with the email system there is way more tricky than with GMail.
  - jart 3y ago
    You can configure Postfix to filter content based on a shell script. https://www.postfix.org/FILTER_README.html https://www.postfix.org/FILTER_README.html You can turn the email into plain text from your shell script using links and then ask Mistral if it's spam or political: https://justine.lol/oneliners/#url https://justine.lol/oneliners/#url You can use the grammar flag to force Mistral to only provide a YES or NO answer, which is easy to check from your script using an IF statement. Then boom, no more spam or politics.
- chasd00 3y ago
  seems like that would be very straightforward to the point that i could probably do it but maybe not in the most elegant way. I would be surprised if there wasn't a model on hugging face trained to identify the political leaning of some text. I bet it would be small enough to run locally, then it's a matter of creating a plugin/module or something that you add to your email client that runs on every new mail. Based on the result, move the email to your political spam folder.
- samus 3y ago
  LLM is just quite compute-intensive. You could send emails to the OpenAI API to classify them as SPAM or HAM, but you'd probably cry at the bill. That should explain why it doesn't exist in free-tier accounts. Worry not, even in the event you belong to the GPU-poor, a beefy CPU should allow you to run a small LLM on your local machine. And if you find a good dataset for it, you can train the model in a Google Collab.
- galdosdi 3y ago
  On a related note, in 2002, Paul Graham arguably invented (IIRC) the spam filtering that is now commonly in use. While email spam still exists today, it was way, way, way, way worse before Gmail introduced bayesian spam filtering around that time. Today it's an occasional amusement. In those last days before spam filtering got a lot better it was reaching the point of seriously making it difficult to use email because the majority of your inbox was spam. The crazy part is it's really just this simple bayesian approach that solved like 95% of the problem. Simple math that already existed. Stuff that can be explained to a high schooler. No new crazy ML/AI, none of that. Just someone stepping back and pointing out "hey, I think we can use this tool here...." There was a lot more lower hanging fruit in those days. https://paulgraham.com/spam.html https://paulgraham.com/spam.html
  - visarga 3y ago
    I remember "A plan for spam", I implemented the algo myself, and it's my first contact with ML.
    - galdosdi 3y ago
      You did? At Google? Cool! :) And thanks! As you can guess I still very pleasantly remember when the feature hit and the spam issue got so much better. I always wonder which pieces of my code from my career I'm proud of are still running somewhere. Unfortunately, it's mostly been proprietary stuff so there's really no telling.
pavel_lishin 3y ago
> Basically, you tell your AI what you want to do, and it does the computer work for you, showing the results the way you like. You won't have to dig through tons of useless apps anymore. It'll search, sort, and create feeds for you. This is what scares advertisers the most Not just advertisers. There isn't much point in creating any sort of fun content on the web if it never actually gets seen by human eyes, but instead is just regurgitated into a slurry.
- visarga 3y ago
  Funny, I like to play a LLM game - copy paste a whole conversation thread into a LLM and ask "Write a 500 word article based on this material". The result is well grounded enjoyable "regurgitated slurry". It's a match made in heaven - people's opinions debating stuff and LLMs ability to phrase that like professional written articles. Grounding and style. I'm afraid in the future not even our social network posts will be directly read by other people. Everything will pass through an AI extractor-reformatter. As an experiment, copy paste a few pages of comments into a LLM and see the quality of its outputs. I often grok a new angle or make a new connection after reading the AI text. Here's a sample, what it understood from our thread: "The discussion touches on concerns around AI interfering with traditional content creation, advertising models, and how we discover and consume information online as AI language understanding and generation capabilities advance." Pretty spot on framing.
  - 12_throw_away 3y ago
    > Here's a sample, what it understood from our thread: "The discussion touches on concerns around AI interfering with traditional content creation, advertising models, and how we discover and consume information online as AI language understanding and generation capabilities advance." Pretty spot on framing. Could you clarify what value, exactly, you're finding in that series of words?
  - pavel_lishin 3y ago
    Uh-huh. And what would you have understood from that summary?
galdosdi 3y ago
> God knows I'd love an AI that can fill tax forms for me. Why? A good accountant can do it for an insignificant[1] annual fee, and if he screws up, you know where to find him. Or if you're poor enough to really need to save a hundred bucks once a year, you probably qualify for freefile, probably have a really simple tax situation (1 or more W2s, end of story), and can file really quickly. Have you used an e-file program lately? It, much like a human accountant, asks a bunch of questions. Not sure what an AI could do differently, except fuck up and ask the wrong questions confidently once in a while. The US has some screwed up tax filing practices but throwing more magic at them would only make it worse. If AI does my taxes, who do I get mad at when it files it wrong and the IRS comes after me? Why are we even talking about AI to apply to a problem that really isn't particularly messy and is already easily solved by good data import and by a nest of ifs and elses maintained once a year by a team of programmers who can read tax laws? Like, we could make tax filing unnecessary tomorrow for most people with no AI, just by passing a law that allows the IRS to use the imported W2 data they already receive to compute and file default tax returns for most people who have simple tax situations.

falcor84 3y ago

I recall imagining something s quite like this as a kid when I first encountered the term "User Agent". I'd be all in on this if I knew I had full control and explainability of its decisions, but am afraid that reality will be a lot more depressing than what I imagine.

crooked-v 3y ago

For a very immediate example, I can already see a clear path to somebody coming up with an LLM service that will automatically filter out political spam texts and emails. (Maybe it already exists and I just haven't seen it yet.)

kaibee 3y ago
I am endlessly frustrated every day that this is not a thing yet. I also don't want to send all of my emails to OpenAI though.
- samus 3y ago
  I recently did a Kaggle experiment and finetuned a LLM to classify SMS as SPAM or HAM. It was not so difficult; following the text classification example in the Hugginface transformers docs was enough. Without trying hard at all, accuracy was above 90%. Impressive even though even the messages marked as HAM seemed to be quite trash. The model itself was small - few hundred million parameters only. Models of this size run well on CPUs and an even smaller might have workd too. And non-neural classifiers might also do fine. You could do something similar and unleash it on your mailbox. The tricky part are integrations. At work, I don't even bother since interacting with the email system there is way more tricky than with GMail.
- jart 3y ago
  You can configure Postfix to filter content based on a shell script. https://www.postfix.org/FILTER_README.html https://www.postfix.org/FILTER_README.html You can turn the email into plain text from your shell script using links and then ask Mistral if it's spam or political: https://justine.lol/oneliners/#url https://justine.lol/oneliners/#url You can use the grammar flag to force Mistral to only provide a YES or NO answer, which is easy to check from your script using an IF statement. Then boom, no more spam or politics.
chasd00 3y ago
seems like that would be very straightforward to the point that i could probably do it but maybe not in the most elegant way. I would be surprised if there wasn't a model on hugging face trained to identify the political leaning of some text. I bet it would be small enough to run locally, then it's a matter of creating a plugin/module or something that you add to your email client that runs on every new mail. Based on the result, move the email to your political spam folder.
samus 3y ago
LLM is just quite compute-intensive. You could send emails to the OpenAI API to classify them as SPAM or HAM, but you'd probably cry at the bill. That should explain why it doesn't exist in free-tier accounts. Worry not, even in the event you belong to the GPU-poor, a beefy CPU should allow you to run a small LLM on your local machine. And if you find a good dataset for it, you can train the model in a Google Collab.
galdosdi 3y ago
On a related note, in 2002, Paul Graham arguably invented (IIRC) the spam filtering that is now commonly in use. While email spam still exists today, it was way, way, way, way worse before Gmail introduced bayesian spam filtering around that time. Today it's an occasional amusement. In those last days before spam filtering got a lot better it was reaching the point of seriously making it difficult to use email because the majority of your inbox was spam. The crazy part is it's really just this simple bayesian approach that solved like 95% of the problem. Simple math that already existed. Stuff that can be explained to a high schooler. No new crazy ML/AI, none of that. Just someone stepping back and pointing out "hey, I think we can use this tool here...." There was a lot more lower hanging fruit in those days. https://paulgraham.com/spam.html https://paulgraham.com/spam.html
- visarga 3y ago
  I remember "A plan for spam", I implemented the algo myself, and it's my first contact with ML.
  - galdosdi 3y ago
    You did? At Google? Cool! :) And thanks! As you can guess I still very pleasantly remember when the feature hit and the spam issue got so much better. I always wonder which pieces of my code from my career I'm proud of are still running somewhere. Unfortunately, it's mostly been proprietary stuff so there's really no telling.

pavel_lishin 3y ago

> Basically, you tell your AI what you want to do, and it does the computer work for you, showing the results the way you like. You won't have to dig through tons of useless apps anymore. It'll search, sort, and create feeds for you. This is what scares advertisers the most Not just advertisers. There isn't much point in creating any sort of fun content on the web if it never actually gets seen by human eyes, but instead is just regurgitated into a slurry.

visarga 3y ago
Funny, I like to play a LLM game - copy paste a whole conversation thread into a LLM and ask "Write a 500 word article based on this material". The result is well grounded enjoyable "regurgitated slurry". It's a match made in heaven - people's opinions debating stuff and LLMs ability to phrase that like professional written articles. Grounding and style. I'm afraid in the future not even our social network posts will be directly read by other people. Everything will pass through an AI extractor-reformatter. As an experiment, copy paste a few pages of comments into a LLM and see the quality of its outputs. I often grok a new angle or make a new connection after reading the AI text. Here's a sample, what it understood from our thread: "The discussion touches on concerns around AI interfering with traditional content creation, advertising models, and how we discover and consume information online as AI language understanding and generation capabilities advance." Pretty spot on framing.
- 12_throw_away 3y ago
  > Here's a sample, what it understood from our thread: "The discussion touches on concerns around AI interfering with traditional content creation, advertising models, and how we discover and consume information online as AI language understanding and generation capabilities advance." Pretty spot on framing. Could you clarify what value, exactly, you're finding in that series of words?
- pavel_lishin 3y ago
  Uh-huh. And what would you have understood from that summary?

galdosdi 3y ago

> God knows I'd love an AI that can fill tax forms for me. Why? A good accountant can do it for an insignificant[1] annual fee, and if he screws up, you know where to find him. Or if you're poor enough to really need to save a hundred bucks once a year, you probably qualify for freefile, probably have a really simple tax situation (1 or more W2s, end of story), and can file really quickly. Have you used an e-file program lately? It, much like a human accountant, asks a bunch of questions. Not sure what an AI could do differently, except fuck up and ask the wrong questions confidently once in a while. The US has some screwed up tax filing practices but throwing more magic at them would only make it worse. If AI does my taxes, who do I get mad at when it files it wrong and the IRS comes after me? Why are we even talking about AI to apply to a problem that really isn't particularly messy and is already easily solved by good data import and by a nest of ifs and elses maintained once a year by a team of programmers who can read tax laws? Like, we could make tax filing unnecessary tomorrow for most people with no AI, just by passing a law that allows the IRS to use the imported W2 data they already receive to compute and file default tax returns for most people who have simple tax situations.