9 ms·

What codegen is good for

114 points by tomduncalf 3y ago

mgaunard 3y ago
Nonsense article. A compiler generates code. It's essential to any software development.
- noelwelsh 3y ago
  Agreed. A lot of words and design effort to not say very much.
- deleted 3y ago
  [deleted]
thomashop 3y ago
Instead of thinking about codegen as a replacement for a developer, what if we thought about it as an extension? In my filter bubble most people would already agree with the statement that generative code models are an extension rather than a replacement. It's not that revelatory of a statement. Saying they are a replacement without any evidence to suggest this is already starting to happen seems like it's taking a larger leap of faith.
- diogenes4 3y ago
  This is also a theme with virtually every technology ever made—technology works as leverage for human intention but rarely replaces humans wholesale.
  - toddmorey 3y ago
    I think in most cases, yes. But sometimes it can be very disruptive to the workforce, too. I think of what AI is going to do to the commercial driver, factory / warehouse, and food service industries, for example. That’s a huge labor pool to reallocate. Not to mention ChatGTP is already has most of the skills needed to be a reasonable replacement for most of my customer service interactions. Not all, but somewhere around 70% I’d say. Not trying to be alarmist, but I feel we’re going to have to rethink labor in both large and small ways.
    - diogenes4 3y ago
      Oh, collectively technology definitely replaces workers. As a member of labor it's up to us to seize the productivity margin introduced by technology.
    - anonylizard 3y ago
      Its not going to do anything to the commercial driver, factory/warehouse, and food service industries. Those are physical labour based industries, and dexterity robots are very, very expensive and not cost competitive with humans. Low end, phone/chat only customer service will be replaced very soon. But what's next in line is millions of white collar jobs, from medicine to law to accountancy to teachers. You are like 2 years behind, the 'truck drivers will be replaced first' narrative has long been flipped on its head.
      - toddmorey 3y ago
        Physical labor industries very much at risk, along with white collar jobs. example: Flippy 2 — a robot arm that works the fryer at fast-food restaurants — already deployed at Chipotle, White Castle and Wing Zone. It’s coming fast.
      - jwestbury 3y ago
        > Those are physical labour based industries, and dexterity robots are very, very expensive and not cost competitive with humans. Costs come down over time, generally. This also depends on cheap labour, which is an active area of discussion, I think you'd agree.
    - otteromkram 3y ago
      How did you manage to misspell ChatGPT? Maybe we should take your insight with a giant grain of salt. Especially since it's not groundbreaking. Why doesn't HN have a downvote option?
      - toddmorey 3y ago
        Super fair. Longtime folks on HN learn to take every supposed insight with a giant grain of salt. When it comes to predicting what will change what, we’re all just wildly guessing.
- lifeisstillgood 3y ago
  The "Story of Pixar" documentary gets about 30 minutes in and apparently a Disney exec kicks out the nascent computer animation department with the comment "if computers can't make movies cheaper than animators we don't want them" plus ca change
whywhywhywhy 3y ago
> developers, only 3% say that they highly trust the accuracy of these (AI) tools I don't believe this at all judging from how I've seen developers using them the past few months. Seems a huge amount of trust in them. Although I'd say the trust is earned, it's shocking how often ChatGPT can completely nail a request.
- darknavi 3y ago
  I think the key word is "highly". I won't argue the metrics but while I generally trust Copilot and ChatGPT, I would not say I "highly trust" them.
- jdminhbg 3y ago
  I think they can be useful without saying you "highly trust" them. If you moderately trust them, you can still let them do grunt work that you'll know you have to look over before committing.
- TrueDuality 3y ago
  Use and trust are different things. I use Google Search, I don't trust the first answer it gives me until I confirm it. I use co-pilot, but I don't accept every request. I don't trust them to be right, or even commonly right. Even the code generation benchmarks for language models are sitting between 30-50% correctness for the absolute best models. Would you trust a lawyer that is right 30-50% of the time?
Pannoniae 3y ago
This article is so misleading.... when someone talks about codegen, it usually means AOP, bytecode manipulation, compiler plugins and the sorts. Not "AI tools". What is even worse is that this isn't even a proper article, this is just a thinly concealed advertisement.
- diogenes4 3y ago
  > This article is so misleading.... when someone talks about codegen, it usually means AOP, bytecode manipulation, compiler plugins and the sorts. Not "AI tools". This is an odd take to me when LLMs are very, very good at generating code, which has got a lot of attention recently. Sure it may be a different beast from what we've currently identified as "codegen" but it remains a descriptive term for the code-generating technology.
  - ynx 3y ago
    Words mean things, especially when communicating in professional or semi-professional settings with terminology with longstanding meaning. To not do so is poor communication, and irresponsible depending on the stakes. Codegen's long history of macro-fied (or similar kind of scripting) source writing matters here. It has the very important property of having consistent/deterministic output from a process that can be verified with very high confidence by rudimentary human inspection. It's like calling autocomplete "codegen". I truly love LLM-assisted coding. I would never call it codegen, and think it can even be unethical to do so when the stakes are high, because it gives it the veneer of trustworthiness that lends one to carelessly not audit it. LLM-aided code writing should, IMO, be called "code assist", not "codegen".
    - pvg 3y ago
      I think this is more a matter of familiarity and habit - notice how there are several slightly different definitions of exactly what 'codegen' means in this thread alone. It doesn't really have some super-specific meaning, it's not, dunno 'lexical scoping'. It's still just shorthand for 'code generation' and it's not unreasonable to apply it to, well, most automated processes that generate code.
VyseofArcadia 3y ago
You keep using that word. I don't think it means what you think it means. No, seriously, codegen already has a specific technical meaning. AI generated code vomit needs to be called something else.
- shadowgovt 3y ago
  What is that specific technical meaning? A quick Google search suggests definitions for the term are all over the map; Wikipedia aliases it to "code generation," which is a whole family of processes (including program synthesis and model-driven development). I'm not seeing anything immediately obvious to exclude AI-synthesized code from the "codegen" label.
  - conradludgate 3y ago
    https://en.m.wikipedia.org/wiki/Code_generation_(compiler) https://en.m.wikipedia.org/wiki/Code_generation_(compiler) it's the final stage of compilation. Turning the in abstract machine intermediate representation into machine code
    - verdverm 3y ago
      Most words have multiple definitions, code gen included. Take for example "go gen", React or Markdown to html, Helm templates, generics, Lisp, ... people think of these as code generation. Taken further, both humans and LLMs generate code based on fuzzy statements A transpiler is doing code gen as well, very similar to your link, but a different, typically text based, target
      - aerzen 3y ago
        That's exactly the problem - codegen means a lot of things. You just talk about codegen and expect people to understand that you mean the AI vomit.
        hu3 3y ago
tikkun 3y ago
Did anyone else find the layout of the article to be quite tough to read and confusing?
- the_sleaze9 3y ago
  Absolutely - almost nauseated actually. The actual content felt a bit tortured as well which didn't help anything.
- deleted 3y ago
  [deleted]
- erksa 3y ago
  The font choice in the images made the call outs extremely hard to read for me.
narush 3y ago
We've thought about this question a lot at Mito[1], where we're building a spreadsheet that code-gens Python code for you as you edit it. For us, it's been useful to decompose the question of "what code-gen is good for" into a few sub-questions that help us think about how generative AI approaches effect us: 1. Why is it necessary to generate code in the first place? Can you just skip to the "solution?" 2. Why is just writing the code by the hand not the best solution? 3. So you do want to do code-gen, does it make sense to do it in a chat interface, or can we do better? As a Figma user, I'd answer these in the following way: > Why is it necessary to generate code in the first place? Because mockups aren't your production website, and your production website is written in code. But maybe this is just for now? I'm sure some high-up PM at Figma has this as their goal - mockup the website in Figma, it generates the code for a website (you don't see this code!), and then you can click deploy _so easily_. Who wants to bet that hosting services like Vercel etc reach out to Figma once a week to try and pitch them... In the meantime, while we have websites that don't fit neatly inside Figma constraints, while developers are easier to hire than good designers (in my experience), while no-code tools are continually thought of as limiting and a bad long-term solution -- Figma code export is good. > Why is just writing the code by the hand not the best solution? For the majority of us full-stack devs who have written >0 CSS but are less than masters, I'll leave this as self-evident. > So you do want to do code-gen, does it make sense to do it in a chat interface, or can we do better? In the case of Figma, if they were a new startup with no existing product and they were trying to "automation UI creation" -- v1 of their interface probably would be a "describe your website" and then we'll generate the code for it. This would probably suck. What if you wanted to easily tweak the output? What if you had trouble describing what you wanted, but you could draw it (ok, OpenAI vision might help on this one)? What if you had experience with existing design tools you could use to augment the AI. A chat interface is not the best interface for design work. ChatGPT-style code-generation is like v0.1. Github Copilot is an example of next step - it's not just a chat interface, it's something a bit more integrated into an environment that make sense in the context of the work you're doing. For design work, a canvas (literally! [2]) like Figma is well-suited as an environment for code-gen that can augment (and maybe one day replace) the programmers working on frontend. For tabular data work, we think a spreadsheet is the interface where users want to be, and the interface it makes sense to bring code-gen to. Any thoughts appreciated! [1] https://trymito.io https://trymito.io, https://github.com/mito-ds/mito https://github.com/mito-ds/mito [2] https://www.figma.com/blog/building-a-professional-design-tool-on-the-web/ https://www.figma.com/blog/building-a-professional-design-to...
jackconsidine 3y ago
I thought these were some good ideas. In my experience I vacillate between "omg the singularity is here" and "this actually isn't that good for X specific task". I very much trust the output of LLMs to be well-designed, but I don't trust things to just work, especially if the system is complicated. I experimented a bit the past few days doing a task myself (building an interface in an existing project), using AI assist, and trying to get AI to solve completely (GPT-4). The solve completely pathway failed and I found myself in an interminable loop. AI-assist was a solid experience. Anecdotal but consistent with Figma's observation
- tymscar 3y ago
  I have ran 5 “test cases” where I would jump on a video call with some friends that are not software engineers but are very technically savvy, and would give them a simple-ish task. They are allowed to use ChatGPT as well as Google. None managed to do it in 4 hours each, and that was with me giving them hints when the AI would inevitably get into a loop. The task was to install docker and using docker compose host a reverse proxy with Træfik with self signed SSL, as well as web-server in Rust. All the rust app had to do was read the kernel version of the host machine and return it as html. I have then ran the same exact test, but this time around, all you had to do was install docker, and with docker compose run a Træfik reverse proxy with self signed SSL, and two other containers, UptimeKuma and Audiobookshelf. No dice, nobody managed to do it.
  - jackconsidine 3y ago
    Wow kudos to you for running such a controlled and extensive experiment
  - rolisz 3y ago
    Are they still friends with you? Are they still returning your calls? I've been coding for 15 years, but on my NAS I still use the Synology reverse proxy so I don't have to deal with Traefik and self signed SSL.
    - tymscar 3y ago
pipeline_peak 3y ago
[flagged]
arabello 3y ago
> Instead of thinking about codegen as a replacement for a developer, what if we thought about it as an extension? Who would think otherwise? It seemed pretty clear from the beginning that anything that automatically generates code would act as an assistant rather than a replacement EDIT: Formatting
- danielvaughn 3y ago
  Yeah the article seems to address a question that no one is asking, unless maybe upper management.
- deleted 3y ago
  [deleted]
latenightcoding 3y ago
Never seen anybody refer to chatgpt code as "codegen".
- deleted 3y ago
  [deleted]
- verdverm 3y ago
  Have you tried searching "chatgpt code generation"? Many results We talk about it in our Readme, under features > chat > Combine LLM and Hof code gen for better, scalable results https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof
- hashtag-til 3y ago
  I was puzzled by that as well and was surprised by the employment of the “codegen” terminology here. Well, it seems it will be another round of terminology abuse.
kaycebasques 3y ago
There's a few comments in here along the lines of "codegen is not the correct term for describing what LLMs do". What is the established term?
- verdverm 3y ago
  There is none Many people call it code gen, I would wager that it is at least a plurality that use "code gen" for what LLMs do.
- deleted 3y ago
  [deleted]
- hashtag-til 3y ago
  I guess it should really be text generation or something else. Not codegen please, that’s taken.
emporas 3y ago
I remember back in March, using ChatGPT to generate code, i was not impressed at all. I mean, the code was okey-ish, but nothing that revolutionary. Fast forward 3 months, i learned to use the tool, now i use it all the time to generate code. There are some ways to use it, and really shine as a tool. 1) It works well for CLI tools. It knows thousands of linux commands and can use them flawlessly most of the time. Any recombining of existing command line tools, is easy as a breeze. 2) Try to use it on a language with an as strict compiler as possible. Rust is the most obvious and modern candidate. Untyped or dynamically typed languages like Python and Angular are far from ideal and should be avoided. Typescript is one more obvious language of choice. 3) Jargon of programming, and knowledge about libraries in the programming language are very important. Being as specific as possible about libraries, maybe even modules and functions makes all the difference. Unfortunately the third point rules out any amateur trying to use code generation effectively. The following days, i want to use GPT to create a notification system for HN comments, like a daemon running in the background, which downloads my comment page, saves all comments on a database, and for every reply, it sends a notification, using notify-send with the user and the first 10 words of the reply. Maybe a subtle sound effect as well, like gmail, or facebook. Does a tool like that exist? I have no doubt, GPT will excel at this, not that difficult but still not trivial task.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- fragmede 3y ago
  Soooo I wrote it. Took me 2 hours, and only looks at your first page of comments, but https://github.com/fragmede/hn_reply_notifier https://github.com/fragmede/hn_reply_notifier For not knowing rust, I'm pretty impressed!
  - emporas 3y ago
    Lol, awesome fragmede. The program has some defects though. The username, of which the comments it is looking for, is hardcoded into the code. The standard convention for CLI tools, is to create a .rc file, like .hn_reply_notifierrc and save some configuration in there. The way your program is configured, the next user will have to recompile the code with the new username. That's a big no-no! It is so much fun to use GPT that way, that i cannot resist but generate my own program, and i will share with you on github the resulting code, alongside with the GPT conversation.
IAmGraydon 3y ago
I have to say this kind of wreaks of Figma feeling threatened.
- deleted 3y ago
  [deleted]
KronisLV 3y ago
Even before LLMs, I think that codegen (the model driven variety) was a good thing! For example, you could throw a WSDL file at SoapUI and get an API client that let you play around with it (https://www.soapui.org/docs/soap-and-wsdl/working-with-wsdls/ https://www.soapui.org/docs/soap-and-wsdl/working-with-wsdls...) or client code for some frameworks (https://www.soapui.org/docs/soap-and-wsdl/soap-code-generation/ https://www.soapui.org/docs/soap-and-wsdl/soap-code-generati...), more or less what OpenAPI/Swagger is really nice for nowadays: https://swagger.io/tools/swagger-ui/ https://swagger.io/tools/swagger-ui/ and https://swagger.io/tools/swagger-codegen/ https://swagger.io/tools/swagger-codegen/ You can also do the reverse, say, go from a live MySQL/MariaDB database running somewhere or a script to a model of it by reverse engineering (https://dev.mysql.com/doc/workbench/en/wb-reverse-engineering.html https://dev.mysql.com/doc/workbench/en/wb-reverse-engineerin...), so that you can explore and change it in a visual manner. You can take either that, or a model created from scratch and either synchronize it with an existing schema, or get the full set of SQL migrations for setting it up from scratch: https://dev.mysql.com/doc/workbench/en/wb-design-schema.html https://dev.mysql.com/doc/workbench/en/wb-design-schema.html and https://dev.mysql.com/doc/workbench/en/wb-forward-engineering.html https://dev.mysql.com/doc/workbench/en/wb-forward-engineerin... Codegen like that can even extend across architecture tiers. For example, I can take a live database with said schema, connect JetBrains Rider with an EntityFramework plugin (using ASP.NET and C# here as an example, though similar solutions exist in Java and other tech stacks) and generate a set of entities with mostly correct data types and relation mappings automatically: https://blog.jetbrains.com/dotnet/2022/01/31/entity-framework-core-inside-rider-ui-way/ https://blog.jetbrains.com/dotnet/2022/01/31/entity-framewor... Not only that, but OpenAPI/Swagger codegen is integrated in ASP.NET so I can also end up with a web based UI to test any APIs that I might make, should I opt to create controllers that use those entities. While most of the codegen I've seen in an academic context has been more or less a mess (broken Eclipse plugin based tools), practical approaches like this are wonderful - for the more boring and boilerplate stuff, I can basically draw a few boxes and get bunches of SQL and C# code that I can then change and/or fine tune as necessary, using the generated stuff as a basis (even though re-generating it would probably overwrite the changes). LLMs feel like the logical next step: feed in a bunch of projects and language documentation and you can query the LLM with various questions about how to do something, to at least sometimes send you on the correct search path yourself without having to jump around 15 different documentation pages, maybe only 5 will suffice now. I use ChatGPT fairly liberally for my personal projects and while it's no silver bullet, it feels like a value add to me, since there's a surprising amount of boilerplate out there for the boring problems that I solve, with every framework and language solving the same stuff in slightly different ways.
beaugunderson 3y ago
Entire page crashes with "Application error: a client-side exception has occurred (see the browser console for more information)." if HTML5 autoplay is disabled.
- deleted 3y ago
  [deleted]
galkk 3y ago
I'm quite sure that LLM style things will be smart enough to generate things like complicated decision trees and state machines from human explanations, plus test suites and explaining which cases have ambiguity and/or missing. I'm still quite skeptical about generic codegen.
- packetlost 3y ago
  > I'm quite sure that LLM style things will be smart enough to generate things like complicated decision trees and state machines from human explanations, plus test suites and explaining which cases have ambiguity and/or missing. I'm not.
- ShamelessC 3y ago
  You have a very strange notion of the current strengths of generative AI.

114 points by tomduncalf 3y ago

mgaunard 3y ago

Nonsense article. A compiler generates code. It's essential to any software development.

noelwelsh 3y ago
Agreed. A lot of words and design effort to not say very much.
deleted 3y ago
[deleted]

thomashop 3y ago

Instead of thinking about codegen as a replacement for a developer, what if we thought about it as an extension? In my filter bubble most people would already agree with the statement that generative code models are an extension rather than a replacement. It's not that revelatory of a statement. Saying they are a replacement without any evidence to suggest this is already starting to happen seems like it's taking a larger leap of faith.

diogenes4 3y ago
This is also a theme with virtually every technology ever made—technology works as leverage for human intention but rarely replaces humans wholesale.
- toddmorey 3y ago
  I think in most cases, yes. But sometimes it can be very disruptive to the workforce, too. I think of what AI is going to do to the commercial driver, factory / warehouse, and food service industries, for example. That’s a huge labor pool to reallocate. Not to mention ChatGTP is already has most of the skills needed to be a reasonable replacement for most of my customer service interactions. Not all, but somewhere around 70% I’d say. Not trying to be alarmist, but I feel we’re going to have to rethink labor in both large and small ways.
  - diogenes4 3y ago
    Oh, collectively technology definitely replaces workers. As a member of labor it's up to us to seize the productivity margin introduced by technology.
  - anonylizard 3y ago
    Its not going to do anything to the commercial driver, factory/warehouse, and food service industries. Those are physical labour based industries, and dexterity robots are very, very expensive and not cost competitive with humans. Low end, phone/chat only customer service will be replaced very soon. But what's next in line is millions of white collar jobs, from medicine to law to accountancy to teachers. You are like 2 years behind, the 'truck drivers will be replaced first' narrative has long been flipped on its head.
    - toddmorey 3y ago
      Physical labor industries very much at risk, along with white collar jobs. example: Flippy 2 — a robot arm that works the fryer at fast-food restaurants — already deployed at Chipotle, White Castle and Wing Zone. It’s coming fast.
    - jwestbury 3y ago
      > Those are physical labour based industries, and dexterity robots are very, very expensive and not cost competitive with humans. Costs come down over time, generally. This also depends on cheap labour, which is an active area of discussion, I think you'd agree.
  - otteromkram 3y ago
    How did you manage to misspell ChatGPT? Maybe we should take your insight with a giant grain of salt. Especially since it's not groundbreaking. Why doesn't HN have a downvote option?
    - toddmorey 3y ago
      Super fair. Longtime folks on HN learn to take every supposed insight with a giant grain of salt. When it comes to predicting what will change what, we’re all just wildly guessing.
lifeisstillgood 3y ago
The "Story of Pixar" documentary gets about 30 minutes in and apparently a Disney exec kicks out the nascent computer animation department with the comment "if computers can't make movies cheaper than animators we don't want them" plus ca change

whywhywhywhy 3y ago

> developers, only 3% say that they highly trust the accuracy of these (AI) tools I don't believe this at all judging from how I've seen developers using them the past few months. Seems a huge amount of trust in them. Although I'd say the trust is earned, it's shocking how often ChatGPT can completely nail a request.

darknavi 3y ago
I think the key word is "highly". I won't argue the metrics but while I generally trust Copilot and ChatGPT, I would not say I "highly trust" them.
jdminhbg 3y ago
I think they can be useful without saying you "highly trust" them. If you moderately trust them, you can still let them do grunt work that you'll know you have to look over before committing.
TrueDuality 3y ago
Use and trust are different things. I use Google Search, I don't trust the first answer it gives me until I confirm it. I use co-pilot, but I don't accept every request. I don't trust them to be right, or even commonly right. Even the code generation benchmarks for language models are sitting between 30-50% correctness for the absolute best models. Would you trust a lawyer that is right 30-50% of the time?

Pannoniae 3y ago

This article is so misleading.... when someone talks about codegen, it usually means AOP, bytecode manipulation, compiler plugins and the sorts. Not "AI tools". What is even worse is that this isn't even a proper article, this is just a thinly concealed advertisement.

diogenes4 3y ago
> This article is so misleading.... when someone talks about codegen, it usually means AOP, bytecode manipulation, compiler plugins and the sorts. Not "AI tools". This is an odd take to me when LLMs are very, very good at generating code, which has got a lot of attention recently. Sure it may be a different beast from what we've currently identified as "codegen" but it remains a descriptive term for the code-generating technology.
- ynx 3y ago
  Words mean things, especially when communicating in professional or semi-professional settings with terminology with longstanding meaning. To not do so is poor communication, and irresponsible depending on the stakes. Codegen's long history of macro-fied (or similar kind of scripting) source writing matters here. It has the very important property of having consistent/deterministic output from a process that can be verified with very high confidence by rudimentary human inspection. It's like calling autocomplete "codegen". I truly love LLM-assisted coding. I would never call it codegen, and think it can even be unethical to do so when the stakes are high, because it gives it the veneer of trustworthiness that lends one to carelessly not audit it. LLM-aided code writing should, IMO, be called "code assist", not "codegen".
  - pvg 3y ago
    I think this is more a matter of familiarity and habit - notice how there are several slightly different definitions of exactly what 'codegen' means in this thread alone. It doesn't really have some super-specific meaning, it's not, dunno 'lexical scoping'. It's still just shorthand for 'code generation' and it's not unreasonable to apply it to, well, most automated processes that generate code.

VyseofArcadia 3y ago

You keep using that word. I don't think it means what you think it means. No, seriously, codegen already has a specific technical meaning. AI generated code vomit needs to be called something else.

shadowgovt 3y ago
What is that specific technical meaning? A quick Google search suggests definitions for the term are all over the map; Wikipedia aliases it to "code generation," which is a whole family of processes (including program synthesis and model-driven development). I'm not seeing anything immediately obvious to exclude AI-synthesized code from the "codegen" label.
- conradludgate 3y ago
  https://en.m.wikipedia.org/wiki/Code_generation_(compiler) https://en.m.wikipedia.org/wiki/Code_generation_(compiler) it's the final stage of compilation. Turning the in abstract machine intermediate representation into machine code
  - verdverm 3y ago
    Most words have multiple definitions, code gen included. Take for example "go gen", React or Markdown to html, Helm templates, generics, Lisp, ... people think of these as code generation. Taken further, both humans and LLMs generate code based on fuzzy statements A transpiler is doing code gen as well, very similar to your link, but a different, typically text based, target
    - aerzen 3y ago
      That's exactly the problem - codegen means a lot of things. You just talk about codegen and expect people to understand that you mean the AI vomit.
      - hu3 3y ago

tikkun 3y ago

Did anyone else find the layout of the article to be quite tough to read and confusing?

the_sleaze9 3y ago
Absolutely - almost nauseated actually. The actual content felt a bit tortured as well which didn't help anything.
deleted 3y ago
[deleted]
erksa 3y ago
The font choice in the images made the call outs extremely hard to read for me.

narush 3y ago

We've thought about this question a lot at Mito[1], where we're building a spreadsheet that code-gens Python code for you as you edit it. For us, it's been useful to decompose the question of "what code-gen is good for" into a few sub-questions that help us think about how generative AI approaches effect us: 1. Why is it necessary to generate code in the first place? Can you just skip to the "solution?" 2. Why is just writing the code by the hand not the best solution? 3. So you do want to do code-gen, does it make sense to do it in a chat interface, or can we do better? As a Figma user, I'd answer these in the following way: > Why is it necessary to generate code in the first place? Because mockups aren't your production website, and your production website is written in code. But maybe this is just for now? I'm sure some high-up PM at Figma has this as their goal - mockup the website in Figma, it generates the code for a website (you don't see this code!), and then you can click deploy _so easily_. Who wants to bet that hosting services like Vercel etc reach out to Figma once a week to try and pitch them... In the meantime, while we have websites that don't fit neatly inside Figma constraints, while developers are easier to hire than good designers (in my experience), while no-code tools are continually thought of as limiting and a bad long-term solution -- Figma code export is good. > Why is just writing the code by the hand not the best solution? For the majority of us full-stack devs who have written >0 CSS but are less than masters, I'll leave this as self-evident. > So you do want to do code-gen, does it make sense to do it in a chat interface, or can we do better? In the case of Figma, if they were a new startup with no existing product and they were trying to "automation UI creation" -- v1 of their interface probably would be a "describe your website" and then we'll generate the code for it. This would probably suck. What if you wanted to easily tweak the output? What if you had trouble describing what you wanted, but you could draw it (ok, OpenAI vision might help on this one)? What if you had experience with existing design tools you could use to augment the AI. A chat interface is not the best interface for design work. ChatGPT-style code-generation is like v0.1. Github Copilot is an example of next step - it's not just a chat interface, it's something a bit more integrated into an environment that make sense in the context of the work you're doing. For design work, a canvas (literally! [2]) like Figma is well-suited as an environment for code-gen that can augment (and maybe one day replace) the programmers working on frontend. For tabular data work, we think a spreadsheet is the interface where users want to be, and the interface it makes sense to bring code-gen to. Any thoughts appreciated! [1] https://trymito.io https://trymito.io, https://github.com/mito-ds/mito https://github.com/mito-ds/mito [2] https://www.figma.com/blog/building-a-professional-design-tool-on-the-web/ https://www.figma.com/blog/building-a-professional-design-to...

jackconsidine 3y ago

I thought these were some good ideas. In my experience I vacillate between "omg the singularity is here" and "this actually isn't that good for X specific task". I very much trust the output of LLMs to be well-designed, but I don't trust things to just work, especially if the system is complicated. I experimented a bit the past few days doing a task myself (building an interface in an existing project), using AI assist, and trying to get AI to solve completely (GPT-4). The solve completely pathway failed and I found myself in an interminable loop. AI-assist was a solid experience. Anecdotal but consistent with Figma's observation

tymscar 3y ago
I have ran 5 “test cases” where I would jump on a video call with some friends that are not software engineers but are very technically savvy, and would give them a simple-ish task. They are allowed to use ChatGPT as well as Google. None managed to do it in 4 hours each, and that was with me giving them hints when the AI would inevitably get into a loop. The task was to install docker and using docker compose host a reverse proxy with Træfik with self signed SSL, as well as web-server in Rust. All the rust app had to do was read the kernel version of the host machine and return it as html. I have then ran the same exact test, but this time around, all you had to do was install docker, and with docker compose run a Træfik reverse proxy with self signed SSL, and two other containers, UptimeKuma and Audiobookshelf. No dice, nobody managed to do it.
- jackconsidine 3y ago
  Wow kudos to you for running such a controlled and extensive experiment
- rolisz 3y ago
  Are they still friends with you? Are they still returning your calls? I've been coding for 15 years, but on my NAS I still use the Synology reverse proxy so I don't have to deal with Traefik and self signed SSL.
  - tymscar 3y ago

pipeline_peak 3y ago

[flagged]

arabello 3y ago

> Instead of thinking about codegen as a replacement for a developer, what if we thought about it as an extension? Who would think otherwise? It seemed pretty clear from the beginning that anything that automatically generates code would act as an assistant rather than a replacement EDIT: Formatting

danielvaughn 3y ago
Yeah the article seems to address a question that no one is asking, unless maybe upper management.
deleted 3y ago
[deleted]

latenightcoding 3y ago

Never seen anybody refer to chatgpt code as "codegen".

deleted 3y ago
[deleted]
verdverm 3y ago
Have you tried searching "chatgpt code generation"? Many results We talk about it in our Readme, under features > chat > Combine LLM and Hof code gen for better, scalable results https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof
hashtag-til 3y ago
I was puzzled by that as well and was surprised by the employment of the “codegen” terminology here. Well, it seems it will be another round of terminology abuse.

kaycebasques 3y ago

There's a few comments in here along the lines of "codegen is not the correct term for describing what LLMs do". What is the established term?

verdverm 3y ago
There is none Many people call it code gen, I would wager that it is at least a plurality that use "code gen" for what LLMs do.
deleted 3y ago
[deleted]
hashtag-til 3y ago
I guess it should really be text generation or something else. Not codegen please, that’s taken.

emporas 3y ago

I remember back in March, using ChatGPT to generate code, i was not impressed at all. I mean, the code was okey-ish, but nothing that revolutionary. Fast forward 3 months, i learned to use the tool, now i use it all the time to generate code. There are some ways to use it, and really shine as a tool. 1) It works well for CLI tools. It knows thousands of linux commands and can use them flawlessly most of the time. Any recombining of existing command line tools, is easy as a breeze. 2) Try to use it on a language with an as strict compiler as possible. Rust is the most obvious and modern candidate. Untyped or dynamically typed languages like Python and Angular are far from ideal and should be avoided. Typescript is one more obvious language of choice. 3) Jargon of programming, and knowledge about libraries in the programming language are very important. Being as specific as possible about libraries, maybe even modules and functions makes all the difference. Unfortunately the third point rules out any amateur trying to use code generation effectively. The following days, i want to use GPT to create a notification system for HN comments, like a daemon running in the background, which downloads my comment page, saves all comments on a database, and for every reply, it sends a notification, using notify-send with the user and the first 10 words of the reply. Maybe a subtle sound effect as well, like gmail, or facebook. Does a tool like that exist? I have no doubt, GPT will excel at this, not that difficult but still not trivial task.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
fragmede 3y ago
Soooo I wrote it. Took me 2 hours, and only looks at your first page of comments, but https://github.com/fragmede/hn_reply_notifier https://github.com/fragmede/hn_reply_notifier For not knowing rust, I'm pretty impressed!
- emporas 3y ago
  Lol, awesome fragmede. The program has some defects though. The username, of which the comments it is looking for, is hardcoded into the code. The standard convention for CLI tools, is to create a .rc file, like .hn_reply_notifierrc and save some configuration in there. The way your program is configured, the next user will have to recompile the code with the new username. That's a big no-no! It is so much fun to use GPT that way, that i cannot resist but generate my own program, and i will share with you on github the resulting code, alongside with the GPT conversation.

IAmGraydon 3y ago

I have to say this kind of wreaks of Figma feeling threatened.

deleted 3y ago
[deleted]

KronisLV 3y ago

Even before LLMs, I think that codegen (the model driven variety) was a good thing! For example, you could throw a WSDL file at SoapUI and get an API client that let you play around with it (https://www.soapui.org/docs/soap-and-wsdl/working-with-wsdls/ https://www.soapui.org/docs/soap-and-wsdl/working-with-wsdls...) or client code for some frameworks (https://www.soapui.org/docs/soap-and-wsdl/soap-code-generation/ https://www.soapui.org/docs/soap-and-wsdl/soap-code-generati...), more or less what OpenAPI/Swagger is really nice for nowadays: https://swagger.io/tools/swagger-ui/ https://swagger.io/tools/swagger-ui/ and https://swagger.io/tools/swagger-codegen/ https://swagger.io/tools/swagger-codegen/ You can also do the reverse, say, go from a live MySQL/MariaDB database running somewhere or a script to a model of it by reverse engineering (https://dev.mysql.com/doc/workbench/en/wb-reverse-engineering.html https://dev.mysql.com/doc/workbench/en/wb-reverse-engineerin...), so that you can explore and change it in a visual manner. You can take either that, or a model created from scratch and either synchronize it with an existing schema, or get the full set of SQL migrations for setting it up from scratch: https://dev.mysql.com/doc/workbench/en/wb-design-schema.html https://dev.mysql.com/doc/workbench/en/wb-design-schema.html and https://dev.mysql.com/doc/workbench/en/wb-forward-engineering.html https://dev.mysql.com/doc/workbench/en/wb-forward-engineerin... Codegen like that can even extend across architecture tiers. For example, I can take a live database with said schema, connect JetBrains Rider with an EntityFramework plugin (using ASP.NET and C# here as an example, though similar solutions exist in Java and other tech stacks) and generate a set of entities with mostly correct data types and relation mappings automatically: https://blog.jetbrains.com/dotnet/2022/01/31/entity-framework-core-inside-rider-ui-way/ https://blog.jetbrains.com/dotnet/2022/01/31/entity-framewor... Not only that, but OpenAPI/Swagger codegen is integrated in ASP.NET so I can also end up with a web based UI to test any APIs that I might make, should I opt to create controllers that use those entities. While most of the codegen I've seen in an academic context has been more or less a mess (broken Eclipse plugin based tools), practical approaches like this are wonderful - for the more boring and boilerplate stuff, I can basically draw a few boxes and get bunches of SQL and C# code that I can then change and/or fine tune as necessary, using the generated stuff as a basis (even though re-generating it would probably overwrite the changes). LLMs feel like the logical next step: feed in a bunch of projects and language documentation and you can query the LLM with various questions about how to do something, to at least sometimes send you on the correct search path yourself without having to jump around 15 different documentation pages, maybe only 5 will suffice now. I use ChatGPT fairly liberally for my personal projects and while it's no silver bullet, it feels like a value add to me, since there's a surprising amount of boilerplate out there for the boring problems that I solve, with every framework and language solving the same stuff in slightly different ways.

beaugunderson 3y ago

Entire page crashes with "Application error: a client-side exception has occurred (see the browser console for more information)." if HTML5 autoplay is disabled.

deleted 3y ago
[deleted]

galkk 3y ago

I'm quite sure that LLM style things will be smart enough to generate things like complicated decision trees and state machines from human explanations, plus test suites and explaining which cases have ambiguity and/or missing. I'm still quite skeptical about generic codegen.

packetlost 3y ago
> I'm quite sure that LLM style things will be smart enough to generate things like complicated decision trees and state machines from human explanations, plus test suites and explaining which cases have ambiguity and/or missing. I'm not.
ShamelessC 3y ago
You have a very strange notion of the current strengths of generative AI.