3 ms·

Using structured outputs for generative ui is such a cool idea does anyone know some cool web demos related to this ?

jiggawatts 2y ago
I just had an evil thought: once AIs are fast enough, it would be possible to create a “dynamic” user interface on the fly using an AI. Instead of Java or C# code running in an event loop processing mouse clicks, in principle we could have a chat bot generate the UI elements in a script like WPF or plain HTML and process user mouse and keyboard input events! If you squint at it, this is what chat bots do now, except with a “terminal” style text UI instead of a GUI or true Web UI. The first incremental step had already been taken: pretty-printing of maths and code. Interactive components are a logical next step. It would be a mere afternoon of work to write a web server where the dozens of “controllers” is replaced with a single call to an LLM API that simply sends the previous page HTML and the request HTML with headers and all. “Based on the previous HTML above and the HTTP request below, output the response HTML.” Just sprinkle on some function calling and a database schema, and the site is done!
- ghthor 2y ago
  That actually sounds pretty entertaining. Especially if there is dynamic user input, like text box input
  - jiggawatts 2y ago
    Other than being borderline impossible to secure, it “should just work” once the AIs get smart enough. Fine-tuning the model based on example pages and responses might be all that’s required for a sufficient level of consistency. An immediate use-case might be prototyping in-place. If you have an existing site, you can capture the request-response pairs and train the AI on it, annotated with the spec docs. Then tell it to implement some new functionality and it should be able to. Just route a subset of the site to the AI instead of the normal controllers. One could “design” new components and functionality in English and try it instantly with no compilation or deployment steps!
- porridgeraisin 2y ago
  Karpathy talks about this in https://karpathy.medium.com/software-2-0-a64152b37c35 https://karpathy.medium.com/software-2-0-a64152b37c35

I just had an evil thought: once AIs are fast enough, it would be possible to create a “dynamic” user interface on the fly using an AI. Instead of Java or C# code running in an event loop processing mouse clicks, in principle we could have a chat bot generate the UI elements in a script like WPF or plain HTML and process user mouse and keyboard input events! If you squint at it, this is what chat bots do now, except with a “terminal” style text UI instead of a GUI or true Web UI. The first incremental step had already been taken: pretty-printing of maths and code. Interactive components are a logical next step. It would be a mere afternoon of work to write a web server where the dozens of “controllers” is replaced with a single call to an LLM API that simply sends the previous page HTML and the request HTML with headers and all. “Based on the previous HTML above and the HTTP request below, output the response HTML.” Just sprinkle on some function calling and a database schema, and the site is done!

ghthor 2y ago
That actually sounds pretty entertaining. Especially if there is dynamic user input, like text box input
- jiggawatts 2y ago
  Other than being borderline impossible to secure, it “should just work” once the AIs get smart enough. Fine-tuning the model based on example pages and responses might be all that’s required for a sufficient level of consistency. An immediate use-case might be prototyping in-place. If you have an existing site, you can capture the request-response pairs and train the AI on it, annotated with the spec docs. Then tell it to implement some new functionality and it should be able to. Just route a subset of the site to the AI instead of the normal controllers. One could “design” new components and functionality in English and try it instantly with no compilation or deployment steps!
porridgeraisin 2y ago
Karpathy talks about this in https://karpathy.medium.com/software-2-0-a64152b37c35 https://karpathy.medium.com/software-2-0-a64152b37c35