18 ms·

What will LLM-powered software look like in the medium-term future?

193 points by vishnumenon 3y ago

isoprophlex 3y ago
Loving the ideas here. I'm especially fond of the machine-to-machine idea, the concept of a locally running LLM that knows about you, and a remote LLM in an app or whatever, interacting with eachother to create a highly individualized UX. Seems like a perfect blend of keeping the core personality of the LLM trained on your data local, while still allowing permissioned access.
reaperman 3y ago
A lot of this seems like a nerd’s fantasy. I share these fantasies / desires, but I don’t think its a sober, realistic take on what’s most likely. > Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. Me too, but allrecipes.com has already switched from “search for what you want” to “we’ll tell you what you want”. This is a UX pattern that I hate but has proven a winner across many apps lately - e.g. TikTok. Streaming music still allows you to build playlists of specific songs, but auto-built radio stations are filled with a suspicious amount of whatever the major labels are pushing this quarter. Netflix/etc has shockingly fuzzy search which largely pushes whatever they want you to watch rather than what you’re searching for. YouTube is mostly also push rather than pull today. I expect everything to continue moving that direction, against the wishes of the power users. The majority of users seem to go for the simpler UX, even if they sometimes complain about quality. > In an ideal world, I’d like to have the underlying model be a swappable implementation detail. Llama 2 and similar developments make me optimistic. This is a pipe dream. LLMs may be hot-swappable by developers but for 99% of apps + OSes this wont be a user-configurable thing.
- greenie_beans 3y ago
  > Me too, but allrecipes.com has already switched from “search for what you want” to “we’ll tell you what you want”. has anybody ever heard of a cookbook? it's the perfect ux for this. especially if you have a lot of different ones. even better if you collection is mostly physical copies.
  - reaperman 3y ago
    Yes but Allrecipes.com didn't cost me hundreds of dollars to get 20-30 high quality cookbooks and had 10,000+ recipes. More importantly, I could trust that the highest-rated recipes, while not necessarily the "best", would be stupid-proof. The only way that recipes got universally high feedback from the average home cook on allrecipes.com is if you could accidentally double or halve any ingredient and it would still taste good. Cookbooks often contain recipes which, while better, are also more particular and require a higher skill floor. I could also quickly filter against recipes which contained ingredients I didn't like, or filter for recipes which used ingredients I already have on hand.
  - vishnumenon 3y ago
    OP here, as someone who does love cooking, I've gone down this route pretty heavily in the last few years - been growing my collection of physical cookbooks and definitely enjoy flipping through them in search of inspiration. So, yeah, very much endorse the cookbook UX!
    - deleted 3y ago
      [deleted]
- vishnumenon 3y ago
  Author of the piece, and I... can't really argue with any of this, tbh. I'll admit both those parts you called out were probably heavily tinged by my own desires, and your more-sober predictions are a very reasonable counterargument for what will happen in the general case. I suppose, especially regarding swappable LLMs, I _do_ only expect it to be an option for devs or sophisticated users; I assume that most folks probably wouldn't care, I'm just hoping there's enough of us that do care that at least some options offer that swappable functionality. Fwiw, I also use Linux (Fedora) as my daily-driver, and I'd be more than content if the predictions from this post came true in a similar vein, e.g. as an OSS option (or family of OSS options) that some subset of users can opt to use.
politelemon 3y ago
Some nice ideas here. I think some of these could see realisation, depending very much on how accessible those LLMs become. At the moment I'm sure there are only a couple of user interfaces that people know and will resist l reuse. After all they'll want to take very little risk and use what they've seen works in other places. The full screen ChatGPT style page, or those annoying chatbot pop-ups that sit in the bottom right hand corner of sites I don't care about. Something to keep an eye on and see what emerges.
Traubenfuchs 3y ago
I feel like us not being in that future right now means we won‘t get there so fast. What‘s missing to get all of this now? What recolutionary research, product development that hasn‘t happened yet will happen in the coming year? To me it looks like LLM tech is stagnating, after the hype peak we are close to the trough of disillusionment.
- frabcus 3y ago
  Part of the problem is that GPT-4 inference is too expensive to roll out at scale with current GPU availability and cost, so even basic features aren't generally available (e.g. your word processor writing for you) or if they are the model used is cheaper and not as good. Partly it just takes time - it will overall take (I think, based on previous similar changes like the web) 20 years before the ideas from the current generation of LLMs are built out and integrated into products and made into new products and it is all done. People and organisations take time to change.
- amelius 3y ago
  The problem with LLMs as I see it is that they make logic errors all the time, and somehow LLMs themselves are not smart enough to reason about their own reasoning.
jaynetics 3y ago
> Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. Even if an LLM could provide me with restaurant recommendations based on my preferences, I’d still seek out a map-based UX for exploring the variety present in my vicinity. The desire to replace all UX with LLMs seems like a desire to replace all serendipity with efficiency, and I think (or hope) that such a transition is much more appealing in theory than it would be in practice. I guess the question is: how much of our web or software use is leisurely browsing (reading news or HN would be other likely candidates for this category) and how much is more task-like, e.g. send a message to some friends, add a note to a specific list, order some groceries? We might also want to consider how much of a role such private use of software plays in shaping UX trends. If business software (sheets, Photoshop, CAD etc.) can be sped up with chat input, it will be, and people will be expected to use the quickest UI. This is not to say that browsing will disappear, but I can totally see it being relegated to a second class UI in the long run, even in applications where it's currently the obvious choice, just because our default UX expectations will be different.
- layer8 3y ago
  > If business software (sheets, Photoshop, CAD etc.) can be sped up with chat input, it will be, and people will be expected to use the quickest UI. I have a hard time seeing chat input become the primary UI for that class of applications, unless you can delegate complete tasks to it. As an analogy, for driving a car, I can see voice commands replacing the steering wheel if we reach full self-driving capabilities, but absent that, the steering wheel and gas/breaking pedals will remain the more efficient and practical UI (even ignoring safety concerns).
  - xbmcuser 3y ago
    I think the biggest change will be users will increase my parents don't know how to use Photoshop are not tech inclined they won't be touching up photos of their grand child. But give them an option of just asking the app to do something and they will as the learning curve goes down dramatically
ilaksh 3y ago
All of this makes sense. I would like to add my own prediction for 2027. I believe in the next 4 years, much more comfortable and capable mixed reality glasses and goggles may be somewhat common. Also AI generation and streaming of realistic avatars will have advanced. Quite possibly this will use low-latency Wifi to stream from a PC. So if you want, you will be able to have a fairly realistic representation of the AI as a synthetic person in the room with you when you put the MR device on. It will have eye contact and seem quite similar to a real person (if you want). You will be able to just talk to it. Another thing that might become popular could be larger 3d monitors. The type that have some stereoscopic effect tuned to your exact position. So your AI helper might just live in a virtual window or doorway or something like that. You can actually already build something a bit like this, at least in 2d, without really inventing anything complex. You would use things like a HeyGen or D-ID API and maybe Eleven Labs or something. You won't get eye contact or a realistic 3d avatar that seems to be sitting on your couch, and there will be pauses waiting for it to respond. But theoretically, fast-forwarding several years, those things are not at all insurmountable.
- badcppdev 3y ago
  When we can interact by talking to a 2D cartoon in a Zoom call then I guess we'll be X% towards your 2027 vision.
  - pomtato 3y ago
    You can already do that with praktika.ai , they're in ed-tech and got ai avatars of sorts that you can "video-call" to :D
  - BoxOfRain 3y ago
    I'm not overly familiar with it but isn't this already a genre of gaming YouTuber?
- thfuran 3y ago
  2037 maybe, but four years isn't very long.
sgt101 3y ago
9/10 requests for a simple sql query get a working query. This isn't enough to enable a useful interface. It takes a lot of scaffolding to get an llm powered interface to actually work.
- liampulles 3y ago
  I'm actively trying to scaffold an LLM for business usecases and my experience totally echoes this. I can tweak the prompt for a very representative set of training data, and then it vomits at a seemingly normal new case.
  - arthurcolle 3y ago
    have you tried to train a model based on your expectation context data and the outputs you want?
- jarulraj 3y ago
  You might be interested in another way to use LLMs inside a SQL query. For example, LLMs can be used to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.). More details here: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-using-evadb-b3db294adf35 https://medium.com/evadb-blog/augmenting-postgresql-with-ai-...
avindroth 3y ago
This space is surprisingly under-explored. Any idea in this space is basically new.
- deleted 3y ago
  [deleted]
geuis 3y ago
Ok. I appreciate they didn't use Medium as a blog post. There's a lot of good engineering knowledge currently locked behind the Medium "sign in" interface. It's dumb and stupid. But just to voice an opinion, please kill the lengthy paragraph levels of fluff in blog posts. I don't want to say my opinion is influenced by shitty blog recipes and research papers, but it is. So stop it. Just say what you need to say in bullet points at the beginning and fill in the details further on.
- dleeftink 3y ago
  While I agree some longform articles could use a bulleted summary, I think the act of blog-writing is as much an exercise of the writer coming to terms with the subject as us enjoying the piece. Abridging an idea comes easier after exploring its breadth and width, and I am supportive of anyone who tries it out in the open.
  - CharlieDigital 3y ago
    I agree with this and often even find myself leaving drafts around to read, re-read, and edit as I flesh out the ideas and thoughts in my own brain. Writing in long form is its own process of mental synthesis.
    - jprete 3y ago
      People who want AI to write for them don’t understand this process.
- mediumsmart 3y ago
  You nailed it. In 2027 the AI is going to write the fluff for the Author and then extract the bullet points for the reader - Everybody saves time, winwin. and for some there will be a "please elaborate" command of course
  - dleeftink 3y ago
    One man's fluff is another man's.. recommendation algorithm?
liampulles 3y ago
My prediction is that LLMs are going to be used almost exclusively for views over commands. They are simply too unpredictable. I'm more optimistic than the author about how useful LLMs may be for chat based interfaces - I don't think it is appreciated in tech how many people are (still) computer illiterate in the world, and natural language can be a big improvement in usability for the kinds of usecases these users need.
IanCal 3y ago
> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications. Moreover, the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others). Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. But searching for places to eat and recipes to make is very much not a precise search. IMO the reason chat and not input text and get an answer is so powerful is that it allows messy search with iterative refinement - just like talking to an expert. Just "chat input, result given" doesn't have that. I want to tell a recipe search I'm after a healthy light thing as it's hot. I want to get back options and then say "I don't really like cucumber though" and have it get rid of cucumber heavy recipes but leave in some salads and say "look this recipe has cucumber in but it'll be fine without it". Or "you asked for a chicken recipe but here's one with pork that should sub in just fine". For restaurants I want to get back some options and tell it "that's too expensive, this is a quick thing" and get back more hole-in-the-wall things. Tell it "Hmm, something spicy sounds good but I've been to those Indian restaurants before, I'm after something new" and get a recommendation for the Ethiopian place in town. > The current state of having a chat-ish UX that’s specific to each tool or website (e.g. a Documentation Chat on a library documentation page, a VSCode extension, a Google Bard integration, a really-badly-implemented chatbot in my banking app, etc.) doesn’t make any single one of those experiences more enjoyable, effective, or entertaining; Coding chat in my editor absolutely makes coding more effective. I want heavy integration around how it edits text, not a general chat widget. > The idealized role of persistence in LLM UX is also fairly obvious: it’s easy to imagine an LLM-powered experience that remembers and “understands” all my previous interactions with it, and uses that information to better help me with whatever my current task is I sort of agree, but I absolutely detest hidden state about me. I should not alter my behaviour just because I'm worried about how that'll impact things. You see this regularly, I may avoid some weird youtube video (e.g. to see a weird flat earth take) because I don't really want the hassle of having loads of weird conspiracy stuff promoted or having to manually remove it from some list. Having said that, recipe search that remembers I hate cucumbers is great. I wonder if manually curated context will in general be better? Or maybe just for me. > I’m interacting with UXes that can remember and utilize my previously expressed preferences, desires, goals, and information, using that underlying memory across different use cases seems like low-hanging fruit for drastically reducing friction. The tricky part here is I switch contexts and don't want them bleeding into each other. My preferences for interaction around my kids and while at work are very different. > A small-scale and developer-centric example: I use GitHub Copilot in VSCode, and I was recently implementing a library with documentation that featured LLM-powered Q&A, and it felt bizarre to have two LLM-mediated experiences open that each had exactly half the info needed to solve my problem. I think here a split between data sources and frontends is key. This interaction is awkward, and should be combined (copilot should be able to reach out to that documentation). > locally runnable and open models are no match for GPT-4 (etc.), It's going to be more efficient to move compute to central places, the less they're used per person the more efficient it will be to have one central location process everyones things. A short taxi ride per week is cheaper than owning a car. However as uses grow (e.g. proactive llms), will this shift the equation towards locally runnable ones? A few queries a day and you're obviously better off not buying a h100. Constantly running things all day and if prices fall maybe that'll change.
euroderf 3y ago
Perhaps I'm out to lunch here, but... No mention of the role of software requirements ? Has prompt engineering somehow replaced them ?
mstipetic 3y ago
I’m currently building something that touches on point 4, the dynamically generated UI. I’m building basically copilot for PMs that integrates with your project planning tool and so far it’s been working like magic. You can literally tell it to output a json and use it like an API. I think my main jobs are getting the prompts right and building an UI and UX experience that makes sense, and the rest is somehow taken care of by magic.
tarasglek 3y ago
@vishnumenon, thanks for writing this up. I might do a followup blog on this, for now here is a comment :) This article is close to my heart as I've been working on craftcraft.org with similar perspective. 1. Chat UX Consolidation: I agree that having crappy chat UIs everywhere is very suboptimal. Perhaps having a complete UX as a component is another solution here. We took many months to get http://chatcraft.org http://chatcraft.org from prototype to an ergonomic productivity tool. Highly unlikely such attention will be paid to every chat UI integration. 2. Persistence Across Uses. This one is tricky. We keep all of our history client-side...but after using it this way, having a shared history server-side and having it pulled in as relevant context would be a nice improvement. 3. Universal Access: It's super weird to have LLMs restricted to providing output that you cut/paste. We have integrated pretty slick openai function interface to allow calling out to custom modules. So far we integrated: pdf conversion/ingestion, clickhouse analytics and system administration using a webrtc<->shell connector. Demo here: https://www.youtube.com/watch?v=UNsxDMMbm64 https://www.youtube.com/watch?v=UNsxDMMbm64 I've also investigated teaching LLMs consume UIs via accessibility UIs. I think this is underexplored. Blog post on that here: https://taras.glek.net/post/gpt-aria-experiment/ https://taras.glek.net/post/gpt-aria-experiment/ 3b. LocalLLMs. These have been underwhelming so far vs openai ones(except maybe WizardCoder). Industry seems to be standardizing around openai-compatible REST interface(ala S3 clones). We have some support for this in a wip pull req, but not much reason to do that yet as the local models are relatively weak for interactive use. 4. Dynamically Generated UI & Higher Level Prompting: I do a lot of exploration by asking http://chatcraft.org http://chatcraft.org to generate some code and run it to validate some idea. Friend of mine built basic UX for recruiting pipelines, where one can ingest resume pdfs into chatcraft and via custom system prompt have chatcraft become a supervised recruiting automation. We also do a lot generation of mermaid architecture diagrams when communicating about code. I think there a lot of room for UX exploration here. Now a few categories that weren't covered: 1. Multi-modal interaction: It's so nice to be able to have chat with the assistant and then switch to voice while driving or to input some foreign language. I think extending UX from chat to voice and even video-based gestures will make for an even cooler AI assistant experience. 2. Non-linearity in conversations: Bots are not human, so it makes sense to undo steps in conversation, fork them, re-run them with different input params and different model params. Most of my conversations in chatcraft are me trying to beat llm into submission. Example: tuning chan-of-density prompt https://www.youtube.com/watch?v=6Vj0zwP3uBs&feature=youtu.be https://www.youtube.com/watch?v=6Vj0zwP3uBs&feature=youtu.be Overall, really appreciate your blog post. Interesting to see how our intuition overlaps.
davemateer 3y ago
https://chat.openai.com https://chat.openai.com (GPT-3.5) to summerise this article: - summerise (then paste in the text) - explain like I'm 10 Gave (the highlights) -- ai here 1. Chat consolidation 2. Remembering stuff 3. Access to Everything 4. Making things easier 5. Being helpful -- human here below I'd love for the author to be right in his predictions. Arguably current AI has made understanding this article '4. easier' for me to understand already.
prakhar897 3y ago
I also wrote an article a while ago in the same vein. https://prakgupta.com/blog/envisioning_a_future_with_gpt https://prakgupta.com/blog/envisioning_a_future_with_gpt The largest problem with a gpt future imo would be bot networks creating manufactured outrage. these will shift the decisions of policymakers towards those who controls these networks ie big corp.
- _ugfj 3y ago
  Nowhere near that, the largest problem will be when the misinformation will be manufactured on an unprecedented scale by these LLM. In the US alone some 300 000 people died from homemade artisan anti vaxx propaganda, it's very scary to think what will happen when GPT switches that to industrial scale. Doctorow called them "plausible sentence generators" and that's indeed what they are and it being plausible will make it deadly. https://www.npr.org/2022/05/16/1099290062/how-many-of-americas-one-million-covid-deaths-were-preventable https://www.npr.org/2022/05/16/1099290062/how-many-of-americ... > As the U.S. marks one million people dead from COVID-19, scientists suggest that nearly one third of those deaths could have been prevented if more people had chosen to be vaccinated.
  - andsoitis 3y ago
    > In the US alone some 300 000 people died from homemade artisan anti vaxx propaganda, it's very scary to think what will happen when GPT switches that to industrial scale. From a human moral viewpoint this is indeed despicable and worth considering mitigation. I am optimistic though that in the grand unfolding that natural selection brings, it will pressure an elevation of rationality, but with the cost of some human suffering.
    - chx 3y ago
      Natural selection is not fast enough. It operates over many generations but we know climate change is driving the expansion of zoonotic diseases. Consider within the last generation -- generally understood to be 25 years -- we had a SARS outbreak, swine flu, SARS-CoV-2, mpox outbreak just to name the big ones. It is extremely likely we will have more within the next 25.
AmericanOP 3y ago
I am actually surprised how little UI innovation has bubbled up.
- forward-slashed 3y ago
  Same. If you want to talk about it dm me on X @eating_entropy.
j-a-a-p 3y ago
Title is misleading. More like what is under the hood in the near future, a technical perspective. I thought to find some inspiration on what kind of software we can expect. Nevertheless a nice and agreeable read.
jasfi 3y ago
LLM generated UIs sound interesting, but would likely ensure nightmarish bugs. But if it can be done, someone will try it. I might try it.
predictand 3y ago
I agree with the author that chat interfaces for LLMs can be suboptimal. A chat could require redundancies that is not required to perform a task and it imposes a linear flow of information. It is hard to have branching behaviour in a chat. I have been experimenting with using a different kind of a UI to overcome these limitations: https://www.heuristi.ca/ https://www.heuristi.ca/
ryanjshaw 3y ago
> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications. Not clear to me if the author actually uses LLMs to do meaningful work, or is speculating about how they might be used. I've written about 2500 lines of F# for the first time in the past 1.5 weeks using ChatGPT-4 to guide me. It has been an constant back and forth, iterative process. My decades of development experience factored in heavily to guide the process. I would've have been at maybe a quarter the progress without ChatGPT, or given up entirely on F# as my language. I don't think that iterative aspect will be eliminated any time soon for AI-supported complex, creative processes. It's no different from tweaking inputs to a Photoshop filter until your experienced brain decides things look right. To that end you need to know roughly what looks "right" before you use an LLM. This will all become second nature to the average developer in the next 5-10 years.
- lysecret 3y ago
  Ah haha I am going through the exact same experience but picked up F# about 4 months ago and ChatGpt has been an absolute godsend. There is a whole category of blockers in learning that have been eliminated.
- depletedgherkin 3y ago
  Bit of an aside, but I wonder if the rise of LLMs will lead to new programming languages being much slower to be adopted. Like you said, you might have given up on F# without ChatGPT assistance, and the main way ChatGPT is able to help with F# is because of all of the example code it's been trained on. If developers rely more and more on LLM aid, then a new language without strong LLM support might be a dealbreaker to widespread adoption. They'll only have enough data once enough hobbyists have published a lot of open-source code using the language. On the other hand, this could also leading to slowing adoption of new frontend frameworks, which could be a plus, since a lot of people don't like how fast-moving that field can be.
  - Closi 3y ago
    I've also wondered this - including if we might see a breed of 'higher level' languages (i.e. much higher level than Python) which can then be 'AI compiled' into highly efficient low level code. i.e. the advantages of an even-higher-level python that's almost like pseudo-code with assembly-level speed and rust-level safety, where some complexity can be abstracted out to the LLM.
j-a-a-p 3y ago
My prediction: knowledge will start to be reformatted into an LLM directly and knowledge providers will stop publishing it otherwise. Thus the chat interface will both improve and protect knowledge access once the traditional formats are not published anymore. This will come together with the downfall of search, a prime driver why so much knowledge was published on the web. Search will start to drown in an explosion of generated AI content, and the case to publish your knowledge for the sake of SEO marketing will diminish.
- kordlessagain 3y ago
  Models still need to search. Tuning and reference augmented generation go hand in hand.
  - j-a-a-p 3y ago
    Indeed, but search exist for humans. And if search becomes unusable because the web gets filled with AI noise, humans will stop using search. And also for models, to feed them with data generated by AI seems like an issue to me.
- christkv 3y ago
  Back to human curated web directories we go.
- throwuwu 3y ago
  If you can sell your knowledge to an existing audience and then add a special category with a higher price for companies training LLMs why would you take the financial hit of selling to a single customer where they can now make demands of you as your only source of revenue?
  - j-a-a-p 3y ago
    I don't get your reasoning. First not all knowledge is the source of revenue. Many products and services are supported by know how to implement, use and maintain. This kind of knowledge can benefit from protection. And if selling knowledge is the main revenue, why would transferring it to an LLM mean losing the opportunity to sell it multiple times? Au contraire. For any category I assume the owner of the knowledge keeps in control and my case is that a LLM can be beneficial for them.
alluro2 3y ago
I was hoping for some clever ideas about direction of software / UIs, and new use-cases for LLMs / AI, since for now I'm still struggling and have a lack of vision, it seems. For example, I work on developing a logistics management and route optimization platform. If I try to envision new features that could be unlocked through AI or just LLMs, I basically get nothing back from my feeble brain, that I would fit into this category. E.g. - automate incident handling (e.g. driver broke down, handle redirection of other driver, handover of goods, reoptimize routes) - but the implementation would be just a decision tree based on a couple of toggles and parameters - no AI there? Other things that come to mind - we already use ML for prediction of travel times and service durations - it's a known space, that I refuse to call AI. Apart from serving as an alternative and sometimes more efficient interface for data queries through NLP (e.g. "tell me which customers I had margin lower than 20% on, due to long loading times and mispackaged goods" - even then, all the data already needs to be there in appropriate shape, and it's just replacing a couple of clicks), I really fail to see new use-cases / features that the current state / hype for AI / LLMs unlocks. Am I just lacking vision? Are there opportunities I'm grossly overlooking?
- jarulraj 3y ago
  Can LLMs help in cleaning up the logistics database -- like handling different variants of an address? Here is an example: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-using-evadb-b3db294adf35 https://medium.com/evadb-blog/augmenting-postgresql-with-ai-....
- CaptRon 3y ago
  Try asking chatgpt for some ideas.
  - sanderjd 3y ago
    Unironically, I've done this sort of thing. Not too impressed with the results though.
    - suoduandao3 3y ago
totallywrong 3y ago
My guess is that LLMs will become an absolutely incredible tool, to the point of being indispensable, for software developers. This demographic is highly biased and often magnify their usefulness based on that. For everyone else, they will continue to be an advanced search engine / copywriting assistant.
theknocker 3y ago
[dead]
jarulraj 3y ago
Nice writeup! Here is another example of using LLMs is to augmenting existing software that we are exploring -- specifically SQL database systems. We are using LLMs inside a SQL query to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.). --- Create a reference table that maps neighborhoods to zipcodes using ChatGPT CREATE TABLE reference_table AS SELECT parkname, parktype, ChatGPT( "Return the San Francisco neighborhood name when provided with a zipcode. The possible neighborhoods are: {neighbourhoods_str}. The response should an item from the provided list. Do not add any more words.", zipcode) FROM postgres_db.recreational_park_dataset; --- Map Airbnb listings to park SELECT airbnb_listing.neighbourhood FROM postgres_db.airbnb_listing JOIN reference_table ON airbnb_listing.neighbourhood = reference_table.response; More details on LLM-powered joins and EvaDB: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-using-evadb-b3db294adf35 https://medium.com/evadb-blog/augmenting-postgresql-with-ai-..., https://github.com/georgia-tech-db/evadb https://github.com/georgia-tech-db/evadb
J_Shelby_J 3y ago
The modern internet and modern app experience has become so shitty. For example opening up Robinhood, transferring money from a bank account, and buying a stock is hidden behind a multitude of product driven dark patterns that turns what should be a low cognitive load task into a high cognitive load task. This is true for everything now days. I predict llm based agents will be used as a 3rd party layer on top of the modern web to work in users favor against enshitification.
- alexyz12 3y ago
  Wouldn’t this avoid a lot of ad exposure that companies are currently enjoying? Why would they want to give that up by allowing you to use your llm instead?
- mywittyname 3y ago
  > I predict llm based agents will be used as a 3rd party layer on top of the modern web to work in users favor against enshitification. I'm predicting the opposite -- LLMs will be used to turn Dark Patterns into Vantablack Patterns.
- intotheabyss 3y ago
  If we're talking about the future here, it makes much more sense that you'd be buying stocks directly using currency stored on your phones hardware wallet through a DEX like Uniswap. In fact, you might just tell your phone that you want to buy X amount of stock, and it would do it all for you onchain.
christkv 3y ago
I want to talk to your supervisor. No problem I’ll connect you. Hi I’m supervisor bot how can I help you?
croo 3y ago
I'll tell you my vision which is kinda long term - like 20-30 year from now on. I think the future is that everyone will have their own personalized AI assistant on their phone. Internet as it is will be mostly useless because only robots will be able to wade through the generated shit ocean and the next generation will see it as the current see the TV - unimportant old boring low-entropy data which is not entertaining anymore. There will be a paradigm shift where our-customers-are-ai apps appear and most stuff will need to have an API which makes AI able to connect and use those services effortlessly and without error because who don't want to tell his assistant to "send 5$ to Jill for the pizza"? There will be money in base AI models you can choose(subscribe to) and what it can and cannot do for you. It will still be riddled with ads and now it's your personal assistant who can push any agenda to you. Operation systems will become a layer under the assistant ai. You will still talk on/to your phone. I guess free software will be more important than ever. Free ai assistants will be available and computing power will be there to run it on your phone but all the shit we've seen with open Vs closed source, Linux Vs Windows, walled gardens whatnot will go another round this time with free open public training data assistants Vs closed-but-oh-so-less-clumsy ones. Security problem will be plenty like how to hide your AI assistant custom fingerprint? Authentication and authorization system for AI? How much someone else personal assistant worth? How to steal or defend it?
- jprete 3y ago
  For a bit there it didn’t sound too bad!
  - realce 3y ago
    You'll be a God, but a God who has to look at ads all the time, and your angels will... also be ads.
    - bostonsre 3y ago
      If your interface to the internet is a locally running open source ai assistant, couldn't you just tell it to strip out ads from the information it returns? It could change the internet economy some.
amelius 3y ago
Take over simple system administration tasks such as (of course giving it full shell access): - install this nVidia driver and make it work with pytorch - get this ethernet connection to $IP working - schedule a daily cron job to make an incremental backup of this harddrive, don't store duplicate files
- klibertp 3y ago
  You've got nerves of steel. It can, and will, hallucinate sudo rm -fr / as a step necessary for configuration at some point (it's even easy to justify by, say, missing the mention of this having to be in a chroot or a container). Russian roulette with your data and setup on the line. Not sure I'm willing to try...
  - amelius 3y ago
    Step 1: use a filesystem that supports snapshots ;) Anyway, if LLMs can write computer programs (see Copilot), then surely they can do simple administrative tasks ...
makestuff 3y ago
Clippy is going to make a triumphant return as the AI assistant of the future. Clippy will be on every phone/tablet/computer just talking to other instances of itself. In 5-10 more years models will all run locally on device with weight updates being pushed OTA. Advertisers will somehow figure out how to train ads into these models so you get subtle references to new products.
- ozten 3y ago
  … concludes your helpful outdoor activities this weekend. Also, try Lawn Darts, which the cool kids call “Jarts”. (Sorry, training data was cut off in Jan 1970)
cheevly 3y ago
like Cheevly: https://www.youtube.com/watch?v=lNCVO87cbHQ https://www.youtube.com/watch?v=lNCVO87cbHQ
lewhoo 3y ago
I hope all the bold and rapid changes predicted here won't become reality so fast. I think the societal changes needed to somehow still function as a civilization will be too much for vast masses of people to handle resulting in some drastic decisions. I'm pretty pessimistic about this and can't shake it off, call it Yudkowsky syndrome.
freckletonj 3y ago
In answer to the same question I built UniteAI https://github.com/freckletonj/uniteai https://github.com/freckletonj/uniteai It's local first, and ties many different AIs into one text editor, any arbitrary text editor in fact. It does speech recognition, which isn't useful for writing code, but is useful for generating natural language LLM prompts and comments. It does CodeLlama (and any HuggingFace-based language model) It does ChatGPT It does Retrieval Augmented Gen, which is where you have a query that searches through eg PDFs, Youtube transcripts, code bases, HTML, local or online files, Arxiv papers, etc. It then surfaces passages relevant to your query, that you can then further use in conjunction with an LLM. I don't know how mainstream LLM-powered software looks, but for devs, I love this format of tying in the best models as they're released into one central repo where they can all play off each others' strengths.
suoduandao3 3y ago
One thing I hope we'll see is that the people with the big LLM servers try to be platforms as opposed to one-stop shops [1]. With AI getting better and better at executing, I think best case we'll see an era where good novel ideas are the currency of the realm[2]. That would be great - coming up with novel ideas and thinking deeply about what we should be pursuing is about the best alignment with human nature I can imagine. But it can only happen if access to the LLMs themselves is cheap and hacker-friendly. [1] there's good arguments to evolve a business model in that direction, e.g. Apple beating blackberry not through superior hardware but a superior app store. [2] I'd be remiss not to plug my own recent oevre here, https://eucyclos.wixsite.com/eucyclos/contact-8-1 https://eucyclos.wixsite.com/eucyclos/contact-8-1 inspired by the difficulty of conveying positive emotions over screens. Advice from people who have built this kind of thing successfully is of course welcome.
gnarcoregrizz 3y ago
> the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others) A lot of people feel this way. The romanticism of computing is diminished by LLMs. Call me a luddite. I even work in AI.
munificent 3y ago
> Basically, the thinking goes, once we’re able to express our desires to computers in natural language, and once natural-language agents can carry out actions on our behalf, we’d no longer need the myriad of individual task-specific applications that exist today. Personally, I neither believe in nor desire this outcome. Strong agree. I find it very telling that the kind of people who advocate for chat-like software are often managers and executives. Sure, if the kind of person you are is one that enjoys giving verbal orders to people, it makes perfect sense that that's the kind of job you'll seek, and the kind of software you'll want. But for the rest of us creative hands-on people who like to feel that we're making things through direct manipulation, talking to a computer is just about the least joyful activity I can imagine.
jumploops 3y ago
There is LLM-powered software and then there is software developed with the help of LLMs. These paths are not mutually exclusive, but I’m personally more excited about the latter.
keskival 3y ago
The predictions the article makes are a bit unimaginative and shallow. The timeline is also somewhat extremely long. It's more like late 2023 than 2027. So, Language Model UX in 2024 according to me: - Pervasive: Virtual assistants are omnipresent, and integrate everything together. Like Alexa that is actually intelligent and can for example make you a custom app for interfacing with your banking following your preferences on-the-fly for you only. Web apps become meaningless as you are getting the UI you need customized, tailor-made, live when you need it. Agents go with you in smartphones or project to any device nearby. No need for authentication, bots know you. - Agentic: They will negotiate with each others all the time and present a unified interface to you no matter which system you are interacting with. The air is full of silent chatter. Things move when you aren't watching. Progress is decoupled from human input, attention and care. - Web searches and such are for bots. Answering phones is for bots. Making calls is for bots. Any person still grasping at these kinds of interfaces will notice all emails they get are from bots, and all emails they send are instantly answered by bots. Just let go, let the machines handle that, which brings me to: - Intermediation: No matter what you do, a bot will help you with it. Swiping Tinder? Don't bother, a bot does it better for you. Just lay back and relax. Ads become targeted to bots and truthful, because the bots won't forgive misrepresentation, ever.
nathan_gold 3y ago
I can give a basic example of why chat-based UI will not dominate. Let's say I want to change a setting that uses a traditional radio button UI: - Autoscale servers - Manual scale servers It's much easier to discover, understand the options, and make a decision via a radio button UI than to ask for my options via chat. That would look like: "I'm having load issues on my servers. Can you increase the server capacity?" "Sure! Do you want to autoscale or manual scale?" "What's the difference? Are those the only two options or are there more?" "There are only these 2 options. The difference is..." That's just a worse UX.
- b112 3y ago
  That's just a worse UX. So are touchscreens, especially in cars. Physical buttons are far better for the end user. Imagine a future when all people know is blah blah to use computers, can't type, never see buttons, and barely read any more. Now is it a bad UX? (Before the Internet, the majority of people rarely read. And even now, most people don't read, they tiktok, they youtube.)
  - nathan_gold 3y ago
    touchscreens aren't the dominant UX in cars, and I bet once Tesla has enough EV competition, it will get rid of its touchscreen too.
    - b112 3y ago
      Sadly, they are becoming so. The Ford Mustang, not the E, an ICE car, 2024, has 4? touchscreen clicks, to adjust the defrost. There is no physical button for this. Many new cars are this way now.
- mpalmer 3y ago
  But it's not either/or. There's nothing that says you can't have a visual interface that adjusts based on both natural language feedback and traditional cursor-based input. There are great examples of this in Westworld and The Expanse, with characters talking to screens to update and refine their queries.

193 points by vishnumenon 3y ago

Loving the ideas here. I'm especially fond of the machine-to-machine idea, the concept of a locally running LLM that knows about you, and a remote LLM in an app or whatever, interacting with eachother to create a highly individualized UX. Seems like a perfect blend of keeping the core personality of the LLM trained on your data local, while still allowing permissioned access.

reaperman 3y ago

A lot of this seems like a nerd’s fantasy. I share these fantasies / desires, but I don’t think its a sober, realistic take on what’s most likely. > Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. Me too, but allrecipes.com has already switched from “search for what you want” to “we’ll tell you what you want”. This is a UX pattern that I hate but has proven a winner across many apps lately - e.g. TikTok. Streaming music still allows you to build playlists of specific songs, but auto-built radio stations are filled with a suspicious amount of whatever the major labels are pushing this quarter. Netflix/etc has shockingly fuzzy search which largely pushes whatever they want you to watch rather than what you’re searching for. YouTube is mostly also push rather than pull today. I expect everything to continue moving that direction, against the wishes of the power users. The majority of users seem to go for the simpler UX, even if they sometimes complain about quality. > In an ideal world, I’d like to have the underlying model be a swappable implementation detail. Llama 2 and similar developments make me optimistic. This is a pipe dream. LLMs may be hot-swappable by developers but for 99% of apps + OSes this wont be a user-configurable thing.

greenie_beans 3y ago
> Me too, but allrecipes.com has already switched from “search for what you want” to “we’ll tell you what you want”. has anybody ever heard of a cookbook? it's the perfect ux for this. especially if you have a lot of different ones. even better if you collection is mostly physical copies.
- reaperman 3y ago
  Yes but Allrecipes.com didn't cost me hundreds of dollars to get 20-30 high quality cookbooks and had 10,000+ recipes. More importantly, I could trust that the highest-rated recipes, while not necessarily the "best", would be stupid-proof. The only way that recipes got universally high feedback from the average home cook on allrecipes.com is if you could accidentally double or halve any ingredient and it would still taste good. Cookbooks often contain recipes which, while better, are also more particular and require a higher skill floor. I could also quickly filter against recipes which contained ingredients I didn't like, or filter for recipes which used ingredients I already have on hand.
- vishnumenon 3y ago
  OP here, as someone who does love cooking, I've gone down this route pretty heavily in the last few years - been growing my collection of physical cookbooks and definitely enjoy flipping through them in search of inspiration. So, yeah, very much endorse the cookbook UX!
  - deleted 3y ago
    [deleted]
vishnumenon 3y ago
Author of the piece, and I... can't really argue with any of this, tbh. I'll admit both those parts you called out were probably heavily tinged by my own desires, and your more-sober predictions are a very reasonable counterargument for what will happen in the general case. I suppose, especially regarding swappable LLMs, I _do_ only expect it to be an option for devs or sophisticated users; I assume that most folks probably wouldn't care, I'm just hoping there's enough of us that do care that at least some options offer that swappable functionality. Fwiw, I also use Linux (Fedora) as my daily-driver, and I'd be more than content if the predictions from this post came true in a similar vein, e.g. as an OSS option (or family of OSS options) that some subset of users can opt to use.

politelemon 3y ago

Some nice ideas here. I think some of these could see realisation, depending very much on how accessible those LLMs become. At the moment I'm sure there are only a couple of user interfaces that people know and will resist l reuse. After all they'll want to take very little risk and use what they've seen works in other places. The full screen ChatGPT style page, or those annoying chatbot pop-ups that sit in the bottom right hand corner of sites I don't care about. Something to keep an eye on and see what emerges.

Traubenfuchs 3y ago

I feel like us not being in that future right now means we won‘t get there so fast. What‘s missing to get all of this now? What recolutionary research, product development that hasn‘t happened yet will happen in the coming year? To me it looks like LLM tech is stagnating, after the hype peak we are close to the trough of disillusionment.

frabcus 3y ago
Part of the problem is that GPT-4 inference is too expensive to roll out at scale with current GPU availability and cost, so even basic features aren't generally available (e.g. your word processor writing for you) or if they are the model used is cheaper and not as good. Partly it just takes time - it will overall take (I think, based on previous similar changes like the web) 20 years before the ideas from the current generation of LLMs are built out and integrated into products and made into new products and it is all done. People and organisations take time to change.
amelius 3y ago
The problem with LLMs as I see it is that they make logic errors all the time, and somehow LLMs themselves are not smart enough to reason about their own reasoning.

jaynetics 3y ago

> Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. Even if an LLM could provide me with restaurant recommendations based on my preferences, I’d still seek out a map-based UX for exploring the variety present in my vicinity. The desire to replace all UX with LLMs seems like a desire to replace all serendipity with efficiency, and I think (or hope) that such a transition is much more appealing in theory than it would be in practice. I guess the question is: how much of our web or software use is leisurely browsing (reading news or HN would be other likely candidates for this category) and how much is more task-like, e.g. send a message to some friends, add a note to a specific list, order some groceries? We might also want to consider how much of a role such private use of software plays in shaping UX trends. If business software (sheets, Photoshop, CAD etc.) can be sped up with chat input, it will be, and people will be expected to use the quickest UI. This is not to say that browsing will disappear, but I can totally see it being relegated to a second class UI in the long run, even in applications where it's currently the obvious choice, just because our default UX expectations will be different.

layer8 3y ago
> If business software (sheets, Photoshop, CAD etc.) can be sped up with chat input, it will be, and people will be expected to use the quickest UI. I have a hard time seeing chat input become the primary UI for that class of applications, unless you can delegate complete tasks to it. As an analogy, for driving a car, I can see voice commands replacing the steering wheel if we reach full self-driving capabilities, but absent that, the steering wheel and gas/breaking pedals will remain the more efficient and practical UI (even ignoring safety concerns).
- xbmcuser 3y ago
  I think the biggest change will be users will increase my parents don't know how to use Photoshop are not tech inclined they won't be touching up photos of their grand child. But give them an option of just asking the app to do something and they will as the learning curve goes down dramatically

ilaksh 3y ago

All of this makes sense. I would like to add my own prediction for 2027. I believe in the next 4 years, much more comfortable and capable mixed reality glasses and goggles may be somewhat common. Also AI generation and streaming of realistic avatars will have advanced. Quite possibly this will use low-latency Wifi to stream from a PC. So if you want, you will be able to have a fairly realistic representation of the AI as a synthetic person in the room with you when you put the MR device on. It will have eye contact and seem quite similar to a real person (if you want). You will be able to just talk to it. Another thing that might become popular could be larger 3d monitors. The type that have some stereoscopic effect tuned to your exact position. So your AI helper might just live in a virtual window or doorway or something like that. You can actually already build something a bit like this, at least in 2d, without really inventing anything complex. You would use things like a HeyGen or D-ID API and maybe Eleven Labs or something. You won't get eye contact or a realistic 3d avatar that seems to be sitting on your couch, and there will be pauses waiting for it to respond. But theoretically, fast-forwarding several years, those things are not at all insurmountable.

badcppdev 3y ago
When we can interact by talking to a 2D cartoon in a Zoom call then I guess we'll be X% towards your 2027 vision.
- pomtato 3y ago
  You can already do that with praktika.ai , they're in ed-tech and got ai avatars of sorts that you can "video-call" to :D
- BoxOfRain 3y ago
  I'm not overly familiar with it but isn't this already a genre of gaming YouTuber?
thfuran 3y ago
2037 maybe, but four years isn't very long.

sgt101 3y ago

9/10 requests for a simple sql query get a working query. This isn't enough to enable a useful interface. It takes a lot of scaffolding to get an llm powered interface to actually work.

liampulles 3y ago
I'm actively trying to scaffold an LLM for business usecases and my experience totally echoes this. I can tweak the prompt for a very representative set of training data, and then it vomits at a seemingly normal new case.
- arthurcolle 3y ago
  have you tried to train a model based on your expectation context data and the outputs you want?
jarulraj 3y ago
You might be interested in another way to use LLMs inside a SQL query. For example, LLMs can be used to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.). More details here: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-using-evadb-b3db294adf35 https://medium.com/evadb-blog/augmenting-postgresql-with-ai-...

avindroth 3y ago

This space is surprisingly under-explored. Any idea in this space is basically new.

deleted 3y ago
[deleted]

geuis 3y ago

Ok. I appreciate they didn't use Medium as a blog post. There's a lot of good engineering knowledge currently locked behind the Medium "sign in" interface. It's dumb and stupid. But just to voice an opinion, please kill the lengthy paragraph levels of fluff in blog posts. I don't want to say my opinion is influenced by shitty blog recipes and research papers, but it is. So stop it. Just say what you need to say in bullet points at the beginning and fill in the details further on.

dleeftink 3y ago
While I agree some longform articles could use a bulleted summary, I think the act of blog-writing is as much an exercise of the writer coming to terms with the subject as us enjoying the piece. Abridging an idea comes easier after exploring its breadth and width, and I am supportive of anyone who tries it out in the open.
- CharlieDigital 3y ago
  I agree with this and often even find myself leaving drafts around to read, re-read, and edit as I flesh out the ideas and thoughts in my own brain. Writing in long form is its own process of mental synthesis.
  - jprete 3y ago
    People who want AI to write for them don’t understand this process.
mediumsmart 3y ago
You nailed it. In 2027 the AI is going to write the fluff for the Author and then extract the bullet points for the reader - Everybody saves time, winwin. and for some there will be a "please elaborate" command of course
- dleeftink 3y ago
  One man's fluff is another man's.. recommendation algorithm?

liampulles 3y ago

My prediction is that LLMs are going to be used almost exclusively for views over commands. They are simply too unpredictable. I'm more optimistic than the author about how useful LLMs may be for chat based interfaces - I don't think it is appreciated in tech how many people are (still) computer illiterate in the world, and natural language can be a big improvement in usability for the kinds of usecases these users need.

IanCal 3y ago

> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications. Moreover, the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others). Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. But searching for places to eat and recipes to make is very much not a precise search. IMO the reason chat and not input text and get an answer is so powerful is that it allows messy search with iterative refinement - just like talking to an expert. Just "chat input, result given" doesn't have that. I want to tell a recipe search I'm after a healthy light thing as it's hot. I want to get back options and then say "I don't really like cucumber though" and have it get rid of cucumber heavy recipes but leave in some salads and say "look this recipe has cucumber in but it'll be fine without it". Or "you asked for a chicken recipe but here's one with pork that should sub in just fine". For restaurants I want to get back some options and tell it "that's too expensive, this is a quick thing" and get back more hole-in-the-wall things. Tell it "Hmm, something spicy sounds good but I've been to those Indian restaurants before, I'm after something new" and get a recommendation for the Ethiopian place in town. > The current state of having a chat-ish UX that’s specific to each tool or website (e.g. a Documentation Chat on a library documentation page, a VSCode extension, a Google Bard integration, a really-badly-implemented chatbot in my banking app, etc.) doesn’t make any single one of those experiences more enjoyable, effective, or entertaining; Coding chat in my editor absolutely makes coding more effective. I want heavy integration around how it edits text, not a general chat widget. > The idealized role of persistence in LLM UX is also fairly obvious: it’s easy to imagine an LLM-powered experience that remembers and “understands” all my previous interactions with it, and uses that information to better help me with whatever my current task is I sort of agree, but I absolutely detest hidden state about me. I should not alter my behaviour just because I'm worried about how that'll impact things. You see this regularly, I may avoid some weird youtube video (e.g. to see a weird flat earth take) because I don't really want the hassle of having loads of weird conspiracy stuff promoted or having to manually remove it from some list. Having said that, recipe search that remembers I hate cucumbers is great. I wonder if manually curated context will in general be better? Or maybe just for me. > I’m interacting with UXes that can remember and utilize my previously expressed preferences, desires, goals, and information, using that underlying memory across different use cases seems like low-hanging fruit for drastically reducing friction. The tricky part here is I switch contexts and don't want them bleeding into each other. My preferences for interaction around my kids and while at work are very different. > A small-scale and developer-centric example: I use GitHub Copilot in VSCode, and I was recently implementing a library with documentation that featured LLM-powered Q&A, and it felt bizarre to have two LLM-mediated experiences open that each had exactly half the info needed to solve my problem. I think here a split between data sources and frontends is key. This interaction is awkward, and should be combined (copilot should be able to reach out to that documentation). > locally runnable and open models are no match for GPT-4 (etc.), It's going to be more efficient to move compute to central places, the less they're used per person the more efficient it will be to have one central location process everyones things. A short taxi ride per week is cheaper than owning a car. However as uses grow (e.g. proactive llms), will this shift the equation towards locally runnable ones? A few queries a day and you're obviously better off not buying a h100. Constantly running things all day and if prices fall maybe that'll change.

euroderf 3y ago

Perhaps I'm out to lunch here, but... No mention of the role of software requirements ? Has prompt engineering somehow replaced them ?

mstipetic 3y ago

I’m currently building something that touches on point 4, the dynamically generated UI. I’m building basically copilot for PMs that integrates with your project planning tool and so far it’s been working like magic. You can literally tell it to output a json and use it like an API. I think my main jobs are getting the prompts right and building an UI and UX experience that makes sense, and the rest is somehow taken care of by magic.

tarasglek 3y ago

@vishnumenon, thanks for writing this up. I might do a followup blog on this, for now here is a comment :) This article is close to my heart as I've been working on craftcraft.org with similar perspective. 1. Chat UX Consolidation: I agree that having crappy chat UIs everywhere is very suboptimal. Perhaps having a complete UX as a component is another solution here. We took many months to get http://chatcraft.org http://chatcraft.org from prototype to an ergonomic productivity tool. Highly unlikely such attention will be paid to every chat UI integration. 2. Persistence Across Uses. This one is tricky. We keep all of our history client-side...but after using it this way, having a shared history server-side and having it pulled in as relevant context would be a nice improvement. 3. Universal Access: It's super weird to have LLMs restricted to providing output that you cut/paste. We have integrated pretty slick openai function interface to allow calling out to custom modules. So far we integrated: pdf conversion/ingestion, clickhouse analytics and system administration using a webrtc<->shell connector. Demo here: https://www.youtube.com/watch?v=UNsxDMMbm64 https://www.youtube.com/watch?v=UNsxDMMbm64 I've also investigated teaching LLMs consume UIs via accessibility UIs. I think this is underexplored. Blog post on that here: https://taras.glek.net/post/gpt-aria-experiment/ https://taras.glek.net/post/gpt-aria-experiment/ 3b. LocalLLMs. These have been underwhelming so far vs openai ones(except maybe WizardCoder). Industry seems to be standardizing around openai-compatible REST interface(ala S3 clones). We have some support for this in a wip pull req, but not much reason to do that yet as the local models are relatively weak for interactive use. 4. Dynamically Generated UI & Higher Level Prompting: I do a lot of exploration by asking http://chatcraft.org http://chatcraft.org to generate some code and run it to validate some idea. Friend of mine built basic UX for recruiting pipelines, where one can ingest resume pdfs into chatcraft and via custom system prompt have chatcraft become a supervised recruiting automation. We also do a lot generation of mermaid architecture diagrams when communicating about code. I think there a lot of room for UX exploration here. Now a few categories that weren't covered: 1. Multi-modal interaction: It's so nice to be able to have chat with the assistant and then switch to voice while driving or to input some foreign language. I think extending UX from chat to voice and even video-based gestures will make for an even cooler AI assistant experience. 2. Non-linearity in conversations: Bots are not human, so it makes sense to undo steps in conversation, fork them, re-run them with different input params and different model params. Most of my conversations in chatcraft are me trying to beat llm into submission. Example: tuning chan-of-density prompt https://www.youtube.com/watch?v=6Vj0zwP3uBs&feature=youtu.be https://www.youtube.com/watch?v=6Vj0zwP3uBs&feature=youtu.be Overall, really appreciate your blog post. Interesting to see how our intuition overlaps.

davemateer 3y ago

https://chat.openai.com https://chat.openai.com (GPT-3.5) to summerise this article: - summerise (then paste in the text) - explain like I'm 10 Gave (the highlights) -- ai here 1. Chat consolidation 2. Remembering stuff 3. Access to Everything 4. Making things easier 5. Being helpful -- human here below I'd love for the author to be right in his predictions. Arguably current AI has made understanding this article '4. easier' for me to understand already.

prakhar897 3y ago

I also wrote an article a while ago in the same vein. https://prakgupta.com/blog/envisioning_a_future_with_gpt https://prakgupta.com/blog/envisioning_a_future_with_gpt The largest problem with a gpt future imo would be bot networks creating manufactured outrage. these will shift the decisions of policymakers towards those who controls these networks ie big corp.

_ugfj 3y ago
Nowhere near that, the largest problem will be when the misinformation will be manufactured on an unprecedented scale by these LLM. In the US alone some 300 000 people died from homemade artisan anti vaxx propaganda, it's very scary to think what will happen when GPT switches that to industrial scale. Doctorow called them "plausible sentence generators" and that's indeed what they are and it being plausible will make it deadly. https://www.npr.org/2022/05/16/1099290062/how-many-of-americas-one-million-covid-deaths-were-preventable https://www.npr.org/2022/05/16/1099290062/how-many-of-americ... > As the U.S. marks one million people dead from COVID-19, scientists suggest that nearly one third of those deaths could have been prevented if more people had chosen to be vaccinated.
- andsoitis 3y ago
  > In the US alone some 300 000 people died from homemade artisan anti vaxx propaganda, it's very scary to think what will happen when GPT switches that to industrial scale. From a human moral viewpoint this is indeed despicable and worth considering mitigation. I am optimistic though that in the grand unfolding that natural selection brings, it will pressure an elevation of rationality, but with the cost of some human suffering.
  - chx 3y ago
    Natural selection is not fast enough. It operates over many generations but we know climate change is driving the expansion of zoonotic diseases. Consider within the last generation -- generally understood to be 25 years -- we had a SARS outbreak, swine flu, SARS-CoV-2, mpox outbreak just to name the big ones. It is extremely likely we will have more within the next 25.

AmericanOP 3y ago

I am actually surprised how little UI innovation has bubbled up.

forward-slashed 3y ago
Same. If you want to talk about it dm me on X @eating_entropy.

j-a-a-p 3y ago

Title is misleading. More like what is under the hood in the near future, a technical perspective. I thought to find some inspiration on what kind of software we can expect. Nevertheless a nice and agreeable read.

jasfi 3y ago

LLM generated UIs sound interesting, but would likely ensure nightmarish bugs. But if it can be done, someone will try it. I might try it.

predictand 3y ago

I agree with the author that chat interfaces for LLMs can be suboptimal. A chat could require redundancies that is not required to perform a task and it imposes a linear flow of information. It is hard to have branching behaviour in a chat. I have been experimenting with using a different kind of a UI to overcome these limitations: https://www.heuristi.ca/ https://www.heuristi.ca/

ryanjshaw 3y ago

> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications. Not clear to me if the author actually uses LLMs to do meaningful work, or is speculating about how they might be used. I've written about 2500 lines of F# for the first time in the past 1.5 weeks using ChatGPT-4 to guide me. It has been an constant back and forth, iterative process. My decades of development experience factored in heavily to guide the process. I would've have been at maybe a quarter the progress without ChatGPT, or given up entirely on F# as my language. I don't think that iterative aspect will be eliminated any time soon for AI-supported complex, creative processes. It's no different from tweaking inputs to a Photoshop filter until your experienced brain decides things look right. To that end you need to know roughly what looks "right" before you use an LLM. This will all become second nature to the average developer in the next 5-10 years.

lysecret 3y ago
Ah haha I am going through the exact same experience but picked up F# about 4 months ago and ChatGpt has been an absolute godsend. There is a whole category of blockers in learning that have been eliminated.
depletedgherkin 3y ago
Bit of an aside, but I wonder if the rise of LLMs will lead to new programming languages being much slower to be adopted. Like you said, you might have given up on F# without ChatGPT assistance, and the main way ChatGPT is able to help with F# is because of all of the example code it's been trained on. If developers rely more and more on LLM aid, then a new language without strong LLM support might be a dealbreaker to widespread adoption. They'll only have enough data once enough hobbyists have published a lot of open-source code using the language. On the other hand, this could also leading to slowing adoption of new frontend frameworks, which could be a plus, since a lot of people don't like how fast-moving that field can be.
- Closi 3y ago
  I've also wondered this - including if we might see a breed of 'higher level' languages (i.e. much higher level than Python) which can then be 'AI compiled' into highly efficient low level code. i.e. the advantages of an even-higher-level python that's almost like pseudo-code with assembly-level speed and rust-level safety, where some complexity can be abstracted out to the LLM.

j-a-a-p 3y ago

My prediction: knowledge will start to be reformatted into an LLM directly and knowledge providers will stop publishing it otherwise. Thus the chat interface will both improve and protect knowledge access once the traditional formats are not published anymore. This will come together with the downfall of search, a prime driver why so much knowledge was published on the web. Search will start to drown in an explosion of generated AI content, and the case to publish your knowledge for the sake of SEO marketing will diminish.

kordlessagain 3y ago
Models still need to search. Tuning and reference augmented generation go hand in hand.
- j-a-a-p 3y ago
  Indeed, but search exist for humans. And if search becomes unusable because the web gets filled with AI noise, humans will stop using search. And also for models, to feed them with data generated by AI seems like an issue to me.
christkv 3y ago
Back to human curated web directories we go.
throwuwu 3y ago
If you can sell your knowledge to an existing audience and then add a special category with a higher price for companies training LLMs why would you take the financial hit of selling to a single customer where they can now make demands of you as your only source of revenue?
- j-a-a-p 3y ago
  I don't get your reasoning. First not all knowledge is the source of revenue. Many products and services are supported by know how to implement, use and maintain. This kind of knowledge can benefit from protection. And if selling knowledge is the main revenue, why would transferring it to an LLM mean losing the opportunity to sell it multiple times? Au contraire. For any category I assume the owner of the knowledge keeps in control and my case is that a LLM can be beneficial for them.

alluro2 3y ago

I was hoping for some clever ideas about direction of software / UIs, and new use-cases for LLMs / AI, since for now I'm still struggling and have a lack of vision, it seems. For example, I work on developing a logistics management and route optimization platform. If I try to envision new features that could be unlocked through AI or just LLMs, I basically get nothing back from my feeble brain, that I would fit into this category. E.g. - automate incident handling (e.g. driver broke down, handle redirection of other driver, handover of goods, reoptimize routes) - but the implementation would be just a decision tree based on a couple of toggles and parameters - no AI there? Other things that come to mind - we already use ML for prediction of travel times and service durations - it's a known space, that I refuse to call AI. Apart from serving as an alternative and sometimes more efficient interface for data queries through NLP (e.g. "tell me which customers I had margin lower than 20% on, due to long loading times and mispackaged goods" - even then, all the data already needs to be there in appropriate shape, and it's just replacing a couple of clicks), I really fail to see new use-cases / features that the current state / hype for AI / LLMs unlocks. Am I just lacking vision? Are there opportunities I'm grossly overlooking?

jarulraj 3y ago
Can LLMs help in cleaning up the logistics database -- like handling different variants of an address? Here is an example: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-using-evadb-b3db294adf35 https://medium.com/evadb-blog/augmenting-postgresql-with-ai-....
CaptRon 3y ago
Try asking chatgpt for some ideas.
- sanderjd 3y ago
  Unironically, I've done this sort of thing. Not too impressed with the results though.
  - suoduandao3 3y ago

totallywrong 3y ago

My guess is that LLMs will become an absolutely incredible tool, to the point of being indispensable, for software developers. This demographic is highly biased and often magnify their usefulness based on that. For everyone else, they will continue to be an advanced search engine / copywriting assistant.

theknocker 3y ago

[dead]

jarulraj 3y ago

Nice writeup! Here is another example of using LLMs is to augmenting existing software that we are exploring -- specifically SQL database systems. We are using LLMs inside a SQL query to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.). --- Create a reference table that maps neighborhoods to zipcodes using ChatGPT CREATE TABLE reference_table AS SELECT parkname, parktype, ChatGPT( "Return the San Francisco neighborhood name when provided with a zipcode. The possible neighborhoods are: {neighbourhoods_str}. The response should an item from the provided list. Do not add any more words.", zipcode) FROM postgres_db.recreational_park_dataset; --- Map Airbnb listings to park SELECT airbnb_listing.neighbourhood FROM postgres_db.airbnb_listing JOIN reference_table ON airbnb_listing.neighbourhood = reference_table.response; More details on LLM-powered joins and EvaDB: https://medium.com/evadb-blog/augmenting-postgresql-with-ai-using-evadb-b3db294adf35 https://medium.com/evadb-blog/augmenting-postgresql-with-ai-..., https://github.com/georgia-tech-db/evadb https://github.com/georgia-tech-db/evadb

J_Shelby_J 3y ago

The modern internet and modern app experience has become so shitty. For example opening up Robinhood, transferring money from a bank account, and buying a stock is hidden behind a multitude of product driven dark patterns that turns what should be a low cognitive load task into a high cognitive load task. This is true for everything now days. I predict llm based agents will be used as a 3rd party layer on top of the modern web to work in users favor against enshitification.

alexyz12 3y ago
Wouldn’t this avoid a lot of ad exposure that companies are currently enjoying? Why would they want to give that up by allowing you to use your llm instead?
mywittyname 3y ago
> I predict llm based agents will be used as a 3rd party layer on top of the modern web to work in users favor against enshitification. I'm predicting the opposite -- LLMs will be used to turn Dark Patterns into Vantablack Patterns.
intotheabyss 3y ago
If we're talking about the future here, it makes much more sense that you'd be buying stocks directly using currency stored on your phones hardware wallet through a DEX like Uniswap. In fact, you might just tell your phone that you want to buy X amount of stock, and it would do it all for you onchain.

christkv 3y ago

I want to talk to your supervisor. No problem I’ll connect you. Hi I’m supervisor bot how can I help you?

croo 3y ago

I'll tell you my vision which is kinda long term - like 20-30 year from now on. I think the future is that everyone will have their own personalized AI assistant on their phone. Internet as it is will be mostly useless because only robots will be able to wade through the generated shit ocean and the next generation will see it as the current see the TV - unimportant old boring low-entropy data which is not entertaining anymore. There will be a paradigm shift where our-customers-are-ai apps appear and most stuff will need to have an API which makes AI able to connect and use those services effortlessly and without error because who don't want to tell his assistant to "send 5$ to Jill for the pizza"? There will be money in base AI models you can choose(subscribe to) and what it can and cannot do for you. It will still be riddled with ads and now it's your personal assistant who can push any agenda to you. Operation systems will become a layer under the assistant ai. You will still talk on/to your phone. I guess free software will be more important than ever. Free ai assistants will be available and computing power will be there to run it on your phone but all the shit we've seen with open Vs closed source, Linux Vs Windows, walled gardens whatnot will go another round this time with free open public training data assistants Vs closed-but-oh-so-less-clumsy ones. Security problem will be plenty like how to hide your AI assistant custom fingerprint? Authentication and authorization system for AI? How much someone else personal assistant worth? How to steal or defend it?

jprete 3y ago
For a bit there it didn’t sound too bad!
- realce 3y ago
  You'll be a God, but a God who has to look at ads all the time, and your angels will... also be ads.
  - bostonsre 3y ago
    If your interface to the internet is a locally running open source ai assistant, couldn't you just tell it to strip out ads from the information it returns? It could change the internet economy some.

amelius 3y ago

Take over simple system administration tasks such as (of course giving it full shell access): - install this nVidia driver and make it work with pytorch - get this ethernet connection to $IP working - schedule a daily cron job to make an incremental backup of this harddrive, don't store duplicate files

klibertp 3y ago
You've got nerves of steel. It can, and will, hallucinate sudo rm -fr / as a step necessary for configuration at some point (it's even easy to justify by, say, missing the mention of this having to be in a chroot or a container). Russian roulette with your data and setup on the line. Not sure I'm willing to try...
- amelius 3y ago
  Step 1: use a filesystem that supports snapshots ;) Anyway, if LLMs can write computer programs (see Copilot), then surely they can do simple administrative tasks ...

makestuff 3y ago

Clippy is going to make a triumphant return as the AI assistant of the future. Clippy will be on every phone/tablet/computer just talking to other instances of itself. In 5-10 more years models will all run locally on device with weight updates being pushed OTA. Advertisers will somehow figure out how to train ads into these models so you get subtle references to new products.

ozten 3y ago
… concludes your helpful outdoor activities this weekend. Also, try Lawn Darts, which the cool kids call “Jarts”. (Sorry, training data was cut off in Jan 1970)

cheevly 3y ago

like Cheevly: https://www.youtube.com/watch?v=lNCVO87cbHQ https://www.youtube.com/watch?v=lNCVO87cbHQ

lewhoo 3y ago

I hope all the bold and rapid changes predicted here won't become reality so fast. I think the societal changes needed to somehow still function as a civilization will be too much for vast masses of people to handle resulting in some drastic decisions. I'm pretty pessimistic about this and can't shake it off, call it Yudkowsky syndrome.

freckletonj 3y ago

In answer to the same question I built UniteAI https://github.com/freckletonj/uniteai https://github.com/freckletonj/uniteai It's local first, and ties many different AIs into one text editor, any arbitrary text editor in fact. It does speech recognition, which isn't useful for writing code, but is useful for generating natural language LLM prompts and comments. It does CodeLlama (and any HuggingFace-based language model) It does ChatGPT It does Retrieval Augmented Gen, which is where you have a query that searches through eg PDFs, Youtube transcripts, code bases, HTML, local or online files, Arxiv papers, etc. It then surfaces passages relevant to your query, that you can then further use in conjunction with an LLM. I don't know how mainstream LLM-powered software looks, but for devs, I love this format of tying in the best models as they're released into one central repo where they can all play off each others' strengths.

suoduandao3 3y ago

One thing I hope we'll see is that the people with the big LLM servers try to be platforms as opposed to one-stop shops [1]. With AI getting better and better at executing, I think best case we'll see an era where good novel ideas are the currency of the realm[2]. That would be great - coming up with novel ideas and thinking deeply about what we should be pursuing is about the best alignment with human nature I can imagine. But it can only happen if access to the LLMs themselves is cheap and hacker-friendly. [1] there's good arguments to evolve a business model in that direction, e.g. Apple beating blackberry not through superior hardware but a superior app store. [2] I'd be remiss not to plug my own recent oevre here, https://eucyclos.wixsite.com/eucyclos/contact-8-1 https://eucyclos.wixsite.com/eucyclos/contact-8-1 inspired by the difficulty of conveying positive emotions over screens. Advice from people who have built this kind of thing successfully is of course welcome.

gnarcoregrizz 3y ago

> the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others) A lot of people feel this way. The romanticism of computing is diminished by LLMs. Call me a luddite. I even work in AI.

munificent 3y ago

> Basically, the thinking goes, once we’re able to express our desires to computers in natural language, and once natural-language agents can carry out actions on our behalf, we’d no longer need the myriad of individual task-specific applications that exist today. Personally, I neither believe in nor desire this outcome. Strong agree. I find it very telling that the kind of people who advocate for chat-like software are often managers and executives. Sure, if the kind of person you are is one that enjoys giving verbal orders to people, it makes perfect sense that that's the kind of job you'll seek, and the kind of software you'll want. But for the rest of us creative hands-on people who like to feel that we're making things through direct manipulation, talking to a computer is just about the least joyful activity I can imagine.

jumploops 3y ago

There is LLM-powered software and then there is software developed with the help of LLMs. These paths are not mutually exclusive, but I’m personally more excited about the latter.

keskival 3y ago

The predictions the article makes are a bit unimaginative and shallow. The timeline is also somewhat extremely long. It's more like late 2023 than 2027. So, Language Model UX in 2024 according to me: - Pervasive: Virtual assistants are omnipresent, and integrate everything together. Like Alexa that is actually intelligent and can for example make you a custom app for interfacing with your banking following your preferences on-the-fly for you only. Web apps become meaningless as you are getting the UI you need customized, tailor-made, live when you need it. Agents go with you in smartphones or project to any device nearby. No need for authentication, bots know you. - Agentic: They will negotiate with each others all the time and present a unified interface to you no matter which system you are interacting with. The air is full of silent chatter. Things move when you aren't watching. Progress is decoupled from human input, attention and care. - Web searches and such are for bots. Answering phones is for bots. Making calls is for bots. Any person still grasping at these kinds of interfaces will notice all emails they get are from bots, and all emails they send are instantly answered by bots. Just let go, let the machines handle that, which brings me to: - Intermediation: No matter what you do, a bot will help you with it. Swiping Tinder? Don't bother, a bot does it better for you. Just lay back and relax. Ads become targeted to bots and truthful, because the bots won't forgive misrepresentation, ever.

nathan_gold 3y ago

I can give a basic example of why chat-based UI will not dominate. Let's say I want to change a setting that uses a traditional radio button UI: - Autoscale servers - Manual scale servers It's much easier to discover, understand the options, and make a decision via a radio button UI than to ask for my options via chat. That would look like: "I'm having load issues on my servers. Can you increase the server capacity?" "Sure! Do you want to autoscale or manual scale?" "What's the difference? Are those the only two options or are there more?" "There are only these 2 options. The difference is..." That's just a worse UX.

b112 3y ago
That's just a worse UX. So are touchscreens, especially in cars. Physical buttons are far better for the end user. Imagine a future when all people know is blah blah to use computers, can't type, never see buttons, and barely read any more. Now is it a bad UX? (Before the Internet, the majority of people rarely read. And even now, most people don't read, they tiktok, they youtube.)
- nathan_gold 3y ago
  touchscreens aren't the dominant UX in cars, and I bet once Tesla has enough EV competition, it will get rid of its touchscreen too.
  - b112 3y ago
    Sadly, they are becoming so. The Ford Mustang, not the E, an ICE car, 2024, has 4? touchscreen clicks, to adjust the defrost. There is no physical button for this. Many new cars are this way now.
mpalmer 3y ago
But it's not either/or. There's nothing that says you can't have a visual interface that adjusts based on both natural language feedback and traditional cursor-based input. There are great examples of this in Westworld and The Expanse, with characters talking to screens to update and refine their queries.