9 ms·

I recorded a screen capture of a task. Gemini generated code to replicate it

112 points by Michelangelo11 3y ago

freediver 3y ago
Is the code actually working?
juniperror 3y ago
Web repeating actually feels like a good fit for these current gen multimodal agents. Unsure if the code was provided (logged out I only see the one post) so unable to speak to the code or how it runs, but judging by the poster’s profile I think I can trust they know when code runs correctly. Though it does appear they’re currently working at this specific agent’s parent company. Honestly, this impresses me more in the work the selenium team has been doing than the llm that used their api.
- internetter 3y ago
  They said it worked with modifications (the code got them 90% of the way)
eden_hazard 3y ago
I feel like most engineers on hackernews/reddit have been underestimating AI tech. People always say it sucks and it gets starts hallucinating nonexistent libraries and what not but we are still in the early phase of this technology. And that how it could never replace them as AI will make current engineers more efficient. I've been scared of AI since seeing chatgbt a couple of years ago. I feel like it's only a matter of time until a dev can feed an AI machine it's entire code base and business requirements. And then a separate AI could carry out the manual/integration testing tasks. AI could potentially cut down the number of devs required to maintain a webapp or ios app after it's built. I feel triggered by this post especially because I've made it a career writing automation code haha.
- wolverine876 3y ago
  This stage is called denial. There's also anger, bargaining, depression, and finally, in the end, acceptence. I'm saying in a funny way, but I suspect it's true. Who can say how AI will develop, but beware of happily-ever-after stories. It could be a nightmare for all of civilization, for just engineers, or just not develop much futher.
  - jhrmnn 3y ago
    The problem with this is that it could be a universal argument against any skepticism. So unless skeptics are always wrong, it doesn‘t really work
    - deleted 3y ago
      [deleted]
    - pfannkuchen 3y ago
      Absolutely! The "stages of grief" are just the stages of approaching a problem. Grief is just special because it is an unsolvable problem and so you go through all of the stages. Entering one or more stages of this process does not mean that you are going to go through all of them. In most non-grief cases it stops earlier.
    - wolverine876 3y ago
      That's a very interesting approach! Ideally, skepticism has none of that: Denial is arguably not part of skepticism - it lacks skepticism of yourself (i.e., denial includes certainty). Skeptics has nothing to do with anger, or bargaining (you can't negotiate truth); and depression and emotional acceptance also are out of place. Skepticism is unemotional in that its definition doesn't reach into emotion. But skeptics - all humans, as far as we know - are emotional creatures, and beyond a doubt those emotions play a role in driving much skepticism. Which returns me to my point: Many responses to AI are driven more by those emotions than by skepicism.
  - hooverd 3y ago
    Nah, it will bifurcate the industry between people who have a mental model of how things work and people who only know how to press the code dispenser button. Offshore teams will be able to sling 10x times the slop 10x times faster.
urbandw311er 3y ago
I am really sorry to whinge but is there any way at all to avoid posting links to tweets? Visiting twitter is such a horrible soul destroying experience. I can’t work out how to full screen the video if it’s even possible. The resolution on the video is too poor for me to be able to read the code. There are ads and troll comments everywhere. It seems to want me to create an account. It’s just horrific. Is there really no better way to showcase some cool observation or story about Gemini doing something useful?
- deleted 3y ago
  [deleted]
- lowlop 3y ago
  Just press the X in the top right corner
- MetalGuru 3y ago
  People post on twitter. It's a thing. Asking to avoid posting links to twitter is ridiculous
  - replwoacause 3y ago
    Maybe but i agree wholeheartedly about how terrible twitter is. Absolute trash heap of an app.
  - urbandw311er 3y ago
    Point taken, but I'm not sure that it is so ridiculous. In some cases (obviously not this case) there might be a better source for the article. Or even changing the URL to nitter or similar would help! Keeping the discussions about this sort of thing active and spreading awareness is a valid way of trying to promote long-term quality content on HN rather than preventing it from turning into a cesspool.
- a1o 3y ago
  I agree, I am really happy with blog posts, articles and pdfs but everytime there is a Twitter or YouTube post I just die inside
WinnieRallycar 3y ago
Has anybody tried feeding a LLM a bunch of business rules and having it generate a program that meets them?
- TrueGeek 3y ago
  I saw one of the Microsoft product owners give a talk on Power Apps at Atlanta Developer's Conference: https://www.microsoft.com/en-us/power-platform/products/power-apps https://www.microsoft.com/en-us/power-platform/products/powe... You can write just a few sentences, or even upload a drawing of what you want your app to look like, and it'll create it for you. The catch is it's only for corporate apps. These can't be publicly facing. Everyone needs to sign in with M365 and have an individual license.
Buttons840 3y ago
If an AI can do this, an AI can also watch my screen and blank out all ads and edit out the sound. The controversy and panic that will happen when such an AI is eventually revealed will be interesting.
- AtlasBarfed 3y ago
  Buy that is the REAL issue. AI is a slave in service to corporations, and those corps have a culture of three decades of depraved sociopathy, with comical levels of contempt for customers, people, humanity, and civilization. AI is not your friend. Whatever service it provides is a Trojan horse or, at a minimum, surveillance.
- pavel_lishin 3y ago
  I read something about this on Garbageday today: > Case in point: the Arc Browser. > For years, The Browser Company has been promising to save the internet. Its Arc Browser is a smart refresh of what a modern gateway to the web should look and feel like and it generated a lot of goodwill with early users. And then, earlier this month, they released their AI-powered search app, which “browses the internet for you.” > The Browser Company’s new app lets you ask semantic questions to a chatbot, which then summarizes live internet results in a simulation of a conversation. Which is great, in theory, as long as you don’t have any concerns about whether what it’s saying is accurate, don’t care where that information is coming from or who wrote it, and don’t think through the long-term feasibility of a product like this even a little bit. > But the base logic of something like Arc’s AI search doesn’t even really make sense. As Engadget recently asked in their excellent teardown of Arc’s AI search pivot, “Who makes money when AI reads the internet for us?” But let’s take a step even further here. Why even bother making new websites if no one’s going to see them?
- g-b-r 3y ago
  I mean, adblockers have existed for a while...
- CatWChainsaw 3y ago
  AI can also watch your screen on Big Tech's behalf and market to you based on your porn watching habits. Or immediately alert interested parties of your torrent setup. Or allow a hacker access to your tax returns when they inevitably jailbreak said AI and it spews out the extremely sensitive personal information of billions of unaware people.
babyshake 3y ago
One thing I am now wondering is why bother with the Selenium script at all? Why not have the AI model describe the same things it would do in a Selenium script in detailed natural language, you could always store that in a DB or file with more efficiency than storing the video, and you could just feed the natural language description to a model for automation? And the major benefit is it is much easier for humans to review and modify if needed.
- from-nibly 3y ago
  Selenium scripts are already so flaky. Why would you want to add more ambiguity to them?
- MetalGuru 3y ago
  Eventually; openai, adept, etc. are working on these types of agents. But currently, name a model that can replace selenium (ie. engage with the browser)
coconutpalms 3y ago
Raise your hand if you know an actual position at an actual company that was made redundant by ChatGPT, DaVinci, Stable Diffusion or related tech
- brutal_chaos_ 3y ago
  https://in.indeed.com/m/jobs?q=wayfair&l=&sc=0kf%3Afcckey%28364a8702098d2cd6%29occ%28EHPW9%29%3B https://in.indeed.com/m/jobs?q=wayfair&l=&sc=0kf%3Afcckey%28... wayfair laid off a bunch of US IT staff and are trying to build an AI in india to replace workers (I think, based on events and new job opennings). edit:typos adt'l: tbf, AI hasn't actually replaced jobs there yet (afaik)
  - from-nibly 3y ago
    Trying doesn't count.
- inlined 3y ago
  https://arstechnica.com/google/2024/01/google-lays-off-hundreds-more-as-ad-division-switches-to-ai-powered-sales/ https://arstechnica.com/google/2024/01/google-lays-off-hundr...
- suddenclarity 3y ago
  More and more board games are released with AI generated art. It's still controversial so larger companies are hesitant to join due to the risk of flack on social media but it's definitely a real thing replacing jobs at the moment. Fryx Games that made Terraforming Mars is probably one of the more well-known examples. Art in general seems to be a high risk occupation because the stakes are low. It's a similar case for stock photography. In the gaming world, Embracer recently fired a bunch of artists with the intent of having fewer people be more productive with the use of AI generated content.
karmasimida 3y ago
This is exciting yet a little scary TBH
GaryNumanVevo 3y ago
AI is going to bifurcate the job market. Short of AGI, the current code gen tools seem to be plateauing at a junior SWE level. Any simple tasks that are done by American IT, and couldn't be easily off-shored to India will be AI tasks. Why even bother hiring junior engineers when those tasks will be covered?

Is the code actually working?

Web repeating actually feels like a good fit for these current gen multimodal agents. Unsure if the code was provided (logged out I only see the one post) so unable to speak to the code or how it runs, but judging by the poster’s profile I think I can trust they know when code runs correctly. Though it does appear they’re currently working at this specific agent’s parent company. Honestly, this impresses me more in the work the selenium team has been doing than the llm that used their api.

internetter 3y ago
They said it worked with modifications (the code got them 90% of the way)

eden_hazard 3y ago

I feel like most engineers on hackernews/reddit have been underestimating AI tech. People always say it sucks and it gets starts hallucinating nonexistent libraries and what not but we are still in the early phase of this technology. And that how it could never replace them as AI will make current engineers more efficient. I've been scared of AI since seeing chatgbt a couple of years ago. I feel like it's only a matter of time until a dev can feed an AI machine it's entire code base and business requirements. And then a separate AI could carry out the manual/integration testing tasks. AI could potentially cut down the number of devs required to maintain a webapp or ios app after it's built. I feel triggered by this post especially because I've made it a career writing automation code haha.

wolverine876 3y ago
This stage is called denial. There's also anger, bargaining, depression, and finally, in the end, acceptence. I'm saying in a funny way, but I suspect it's true. Who can say how AI will develop, but beware of happily-ever-after stories. It could be a nightmare for all of civilization, for just engineers, or just not develop much futher.
- jhrmnn 3y ago
  The problem with this is that it could be a universal argument against any skepticism. So unless skeptics are always wrong, it doesn‘t really work
  - deleted 3y ago
    [deleted]
  - pfannkuchen 3y ago
    Absolutely! The "stages of grief" are just the stages of approaching a problem. Grief is just special because it is an unsolvable problem and so you go through all of the stages. Entering one or more stages of this process does not mean that you are going to go through all of them. In most non-grief cases it stops earlier.
  - wolverine876 3y ago
    That's a very interesting approach! Ideally, skepticism has none of that: Denial is arguably not part of skepticism - it lacks skepticism of yourself (i.e., denial includes certainty). Skeptics has nothing to do with anger, or bargaining (you can't negotiate truth); and depression and emotional acceptance also are out of place. Skepticism is unemotional in that its definition doesn't reach into emotion. But skeptics - all humans, as far as we know - are emotional creatures, and beyond a doubt those emotions play a role in driving much skepticism. Which returns me to my point: Many responses to AI are driven more by those emotions than by skepicism.
- hooverd 3y ago
  Nah, it will bifurcate the industry between people who have a mental model of how things work and people who only know how to press the code dispenser button. Offshore teams will be able to sling 10x times the slop 10x times faster.

urbandw311er 3y ago

I am really sorry to whinge but is there any way at all to avoid posting links to tweets? Visiting twitter is such a horrible soul destroying experience. I can’t work out how to full screen the video if it’s even possible. The resolution on the video is too poor for me to be able to read the code. There are ads and troll comments everywhere. It seems to want me to create an account. It’s just horrific. Is there really no better way to showcase some cool observation or story about Gemini doing something useful?

deleted 3y ago
[deleted]
lowlop 3y ago
Just press the X in the top right corner
MetalGuru 3y ago
People post on twitter. It's a thing. Asking to avoid posting links to twitter is ridiculous
- replwoacause 3y ago
  Maybe but i agree wholeheartedly about how terrible twitter is. Absolute trash heap of an app.
- urbandw311er 3y ago
  Point taken, but I'm not sure that it is so ridiculous. In some cases (obviously not this case) there might be a better source for the article. Or even changing the URL to nitter or similar would help! Keeping the discussions about this sort of thing active and spreading awareness is a valid way of trying to promote long-term quality content on HN rather than preventing it from turning into a cesspool.
a1o 3y ago
I agree, I am really happy with blog posts, articles and pdfs but everytime there is a Twitter or YouTube post I just die inside

WinnieRallycar 3y ago

Has anybody tried feeding a LLM a bunch of business rules and having it generate a program that meets them?

TrueGeek 3y ago
I saw one of the Microsoft product owners give a talk on Power Apps at Atlanta Developer's Conference: https://www.microsoft.com/en-us/power-platform/products/power-apps https://www.microsoft.com/en-us/power-platform/products/powe... You can write just a few sentences, or even upload a drawing of what you want your app to look like, and it'll create it for you. The catch is it's only for corporate apps. These can't be publicly facing. Everyone needs to sign in with M365 and have an individual license.

Buttons840 3y ago

If an AI can do this, an AI can also watch my screen and blank out all ads and edit out the sound. The controversy and panic that will happen when such an AI is eventually revealed will be interesting.

AtlasBarfed 3y ago
Buy that is the REAL issue. AI is a slave in service to corporations, and those corps have a culture of three decades of depraved sociopathy, with comical levels of contempt for customers, people, humanity, and civilization. AI is not your friend. Whatever service it provides is a Trojan horse or, at a minimum, surveillance.
pavel_lishin 3y ago
I read something about this on Garbageday today: > Case in point: the Arc Browser. > For years, The Browser Company has been promising to save the internet. Its Arc Browser is a smart refresh of what a modern gateway to the web should look and feel like and it generated a lot of goodwill with early users. And then, earlier this month, they released their AI-powered search app, which “browses the internet for you.” > The Browser Company’s new app lets you ask semantic questions to a chatbot, which then summarizes live internet results in a simulation of a conversation. Which is great, in theory, as long as you don’t have any concerns about whether what it’s saying is accurate, don’t care where that information is coming from or who wrote it, and don’t think through the long-term feasibility of a product like this even a little bit. > But the base logic of something like Arc’s AI search doesn’t even really make sense. As Engadget recently asked in their excellent teardown of Arc’s AI search pivot, “Who makes money when AI reads the internet for us?” But let’s take a step even further here. Why even bother making new websites if no one’s going to see them?
g-b-r 3y ago
I mean, adblockers have existed for a while...
CatWChainsaw 3y ago
AI can also watch your screen on Big Tech's behalf and market to you based on your porn watching habits. Or immediately alert interested parties of your torrent setup. Or allow a hacker access to your tax returns when they inevitably jailbreak said AI and it spews out the extremely sensitive personal information of billions of unaware people.

babyshake 3y ago

One thing I am now wondering is why bother with the Selenium script at all? Why not have the AI model describe the same things it would do in a Selenium script in detailed natural language, you could always store that in a DB or file with more efficiency than storing the video, and you could just feed the natural language description to a model for automation? And the major benefit is it is much easier for humans to review and modify if needed.

from-nibly 3y ago
Selenium scripts are already so flaky. Why would you want to add more ambiguity to them?
MetalGuru 3y ago
Eventually; openai, adept, etc. are working on these types of agents. But currently, name a model that can replace selenium (ie. engage with the browser)

coconutpalms 3y ago

Raise your hand if you know an actual position at an actual company that was made redundant by ChatGPT, DaVinci, Stable Diffusion or related tech

brutal_chaos_ 3y ago
https://in.indeed.com/m/jobs?q=wayfair&l=&sc=0kf%3Afcckey%28364a8702098d2cd6%29occ%28EHPW9%29%3B https://in.indeed.com/m/jobs?q=wayfair&l=&sc=0kf%3Afcckey%28... wayfair laid off a bunch of US IT staff and are trying to build an AI in india to replace workers (I think, based on events and new job opennings). edit:typos adt'l: tbf, AI hasn't actually replaced jobs there yet (afaik)
- from-nibly 3y ago
  Trying doesn't count.
inlined 3y ago
https://arstechnica.com/google/2024/01/google-lays-off-hundreds-more-as-ad-division-switches-to-ai-powered-sales/ https://arstechnica.com/google/2024/01/google-lays-off-hundr...
suddenclarity 3y ago
More and more board games are released with AI generated art. It's still controversial so larger companies are hesitant to join due to the risk of flack on social media but it's definitely a real thing replacing jobs at the moment. Fryx Games that made Terraforming Mars is probably one of the more well-known examples. Art in general seems to be a high risk occupation because the stakes are low. It's a similar case for stock photography. In the gaming world, Embracer recently fired a bunch of artists with the intent of having fewer people be more productive with the use of AI generated content.

karmasimida 3y ago

This is exciting yet a little scary TBH

GaryNumanVevo 3y ago

AI is going to bifurcate the job market. Short of AGI, the current code gen tools seem to be plateauing at a junior SWE level. Any simple tasks that are done by American IT, and couldn't be easily off-shored to India will be AI tasks. Why even bother hiring junior engineers when those tasks will be covered?