6 ms·

Large Language Models: Compilers for the 4th Gen of Programming Languages? [pdf]

78 points by luu 3y ago

PaulHoule 3y ago
What is it with rude and ignorant people who recycle version numbers? There was Web 3.0 https://en.wikipedia.org/wiki/Semantic_Web https://en.wikipedia.org/wiki/Semantic_Web and the blockheads totally ignored it. (e.g. I had a Web 3.0 site a decade before they did!) Similarly there was a lot of talk about 4GL in the 1980s https://en.wikipedia.org/wiki/Fourth-generation_programming_language https://en.wikipedia.org/wiki/Fourth-generation_programming_... some of which was related to the same symbolic A.I. ideas that were influential to the semantic web.
- linguae 3y ago
  I think a major part of it is a lack of knowledge of history among many people in computer science and software engineering, even researchers. Alan Kay, the creator of the Smalltalk programming language and one of my personal heroes, has lamented many times over the years that a "pop culture" has developed in the software engineering and computer science world. This culture is focused on the present and often disregards the past. It's more than the overloading of terminology like "Web 3.0" and "fourth-generation programming languages." It's an overall attitude that focuses solely on the fad du jour and lacks a deeper understanding of computing. I've noticed a lack of knowledge of computing history among many people who work in computing professions. Personally I think all undergraduate students of computer science and related disciplines would benefit immensely from a one-semester history of computing course. They'd start by learning about early analog computing devices before learning about two very crucial developments in the 1930's and 1940's: (1) the invention of early digital computers such as the ENIAC, and (2) meta-mathematics and how that led to Godel's incompleteness theorem, the lambda calculus, Turing machines, and other fundamental concepts that underpin theoretical computer science. Then they'd learn about the rise of the computer industry, the history of software, the history of high-level programming languages, the history of AI, advances in computer architecture, the birth of the Internet, the GUI, the World Wide Web, social networking, and modern topics such as deep neural networks and large language models. It would be a lot of ground to cover and it would be impossible to cover everything in just a single course, but the goal of such a course is for practitioners in computer science and software engineering to have a basic understanding of the history of their profession and why things are the way that they are today. I also believe that those fortunate enough to work in or visit Silicon Valley should make a visit to the Computer History Museum in Mountain View to get a taste of how far computing has come in just a relatively short amount of time compared to all of human history.
  - Filligree 3y ago
    > I also believe that those fortunate enough to work in or visit Silicon Valley should make a visit to the Computer History Museum in Mountain View to get a taste of how far computing has come in just a relatively short amount of time compared to all of human history. Got to second this. That museum was the highlight of my visit, and I still remember everything in detail.
- IshKebab 3y ago
  "blockheads" seems quite rude to me. In any case the semantic web was ignored by almost everyone because the effort of wading through wanky over-engineered specifications was never worth the approximately zero benefits it actually gave. Presumably there will be a third Web 3.0 by people who also ignore the blockchain nonsense which also never amounted to anything!
- deleted 3y ago
  [deleted]
- seydor 3y ago
  Semantic web was so hated and condemned to damnatio memoriae
- CyberDildonics 3y ago
  It's just nonsense for clicks, don't take it too seriously
- FFP999 3y ago
  But neither SemWeb nor 4GL ever lived up to a fraction of the hype around them, so it's understandable if people have forgotten, or never learned about, these evolutionary dead ends.
isaiahg 3y ago
This is something I’ve been thinking about for some time, a new type of programming that uses an ai model for parsing. It’s a interesting thought that can lead down many rabbit holes. You could have AI interpret a lot of expressiveness into a language while having an optimizing AI turn whatever crazy thing we come up with into fast code. For any prospective language designers out there. Please don’t make human based language programming a thing again. That’s an idea that should stay in the waste bin. A language that describes programs shouldn’t be ambiguous. I also wouldn’t trust any AI—no matter how smart—to create code I’d be happy with, unless I could go into very specific detail. No limits on structure serialization though. Make that a one liner please
random3 3y ago
Oh, so this isn't about these 4GLs https://en.wikipedia.org/wiki/Fourth-generation_programming_language https://en.wikipedia.org/wiki/Fourth-generation_programming_... This said, I'm not sure why we'd call natural language 4th gen it's very much the first gen.
- lifthrasiir 3y ago
  The original concepts of 4GL and 5GL are long gone [1], and I do think it is reasonable to reuse the 4GL label for other possibilities. [1] In the sense that those labels do not describe the actual "generation". There still exist visual programming (4GL) and constraint and logic programming (5GL), but they are no longer considered to delieve the goal of 4GL and 5GL by its own, and thus have failed to generate enough interest to be considered a "generation".
  - mitjam 3y ago
    I think 4gl is alive and delivering on its goal of manipulating and analyzing a lot of data with few statements, for example in the form of SQL, R, Pandas, Mathematica, Wolfram Alpha, Power Query, and many more. Nobody calls it that way, anymore, however.
    - random3 3y ago
      Agreed. SQL with relational algebras is enough to bear the name. This said - less a matter of whether something else should be 4th or fifth, I think the concern is with making everything an idiocracy-like marketing bingo like the 5D cinema https://www.xd-cinema.com/the-difference-between-4d-5d-6d-7d-8d-9d-xd-cinema/ https://www.xd-cinema.com/the-difference-between-4d-5d-6d-7d...
ixaxaar 3y ago
I am not so sure if the determinism offered by programming languages is more meant for a human requirement or not - if it is, making programming languages more "natural" will actually backfire because a human ultimately needs to read it and understand. Code is meant to be mostly read not written. And no, LLM explanations are shit useless, a language model simply cannot handhold a human to make them understand logic line by line, you NEED to read it, and hence code has to be "readable", which to an engineer means "precise".
- anonzzzies 3y ago
  > LLM explanations are shit useless, For you and for me maybe, but by now I know a lot of people I met at meetups who never programmed but who are creating SaaS etc systems now just by having a chat with chatgpt. They cannot code without, they have the llm explain line by line. They don't understand when they read it (mostly Python/React/JS) without the llm explaining. And my worst nightmare (as someone who likes precise, static typing, strong typing and even formal proofs now and then) which seems to rapidly becoming reality; read/see no code at all; it's just nanoservice blackbox (just single endpoints) stuff written by chatgpt and the 'programmer' never sees the code, just the input and output.
  - dkersten 3y ago
    How well does this work for them in practice? Try as I might, the great I could get ChatGPT to generate was what I would consider student quality code. Good enough for learning, maybe ok as a starting point, but would never pass code review at my day job and it also tended to break in subtle and not so subtle ways when additional requirements or bug fixes were prompted for. I did find it was good at explaining code, but I was also able to read the code alongside its explanations since I have been programming for two decades, and I did find it useful for familiarising myself with unknown codebases (although the tooling for this needs to be worked on). I also find it’s quite capable at little self contained code snippets (useful for bits of data mungeing in Python for example, but not quite able to build a full product). Presumably these people are generating snippets of stuff and gluing them together. I guess that can work, but I can’t imagine the output is particularly good or bug free.
canvascritic 3y ago
Haha I love this, great read. The concept of llms serving as a fourth generation compiler is not really novel. I think we saw Databricks in here talking about LLMs as the new compiler and Python as the new bytecode. Anyway if we're to follow this line of thought, it might not be long before large language models begin interfacing directly with machine hardware, bypassing the need for languages like python and java altogether. Like what purpose would they serve? This could streamline computational processes, reducing overheads and inefficiencies. As the distance between the llm and hardware closes, human interaction with these systems might evolve to use even higher-level languages, further abstracting us from the underlying mechanics. it's conceivable that, in the hopefully distant future our interaction could advance to direct neural or brain interfaces, where our thoughts translate into commands without the need for any traditional "coding" at all. Instead the spec becomes the software. These are damn exciting times
richardjam73 3y ago
I don't think that their usage of 4th Generation Programming Languages is incorrect. The paper shows cleary that they understand the generations. The conclusion reads more like LLMs can translate from an human prompt to a 3GL such as python. The prompt is not quite human language but is rather a specific form of writing designed to get the output needed from the LLM, in this way it is a 4GL and the LLM is the compiler.
Smaug123 3y ago
twitch I thought the Queen was greedy and had very little patience - why are we traversing the input list twice in both generated responses?
sheepscreek 3y ago
I’ve been imagining a slightly different hypothetical future. - LLMs and GenAI can potentially replace many OS layers - They can translate text directly to machine code, or a portable representation like LLVM IR/MLIR - They may do this by making system calls directly - or have a lightweight kernel of its own supporting basic GUI primitives and web-views Even better, imagine an LLM fused with a web-browser core. Like Chrome OS, but with LLM powered interaction. Capable of conjuring windows with rich GUIs that are purpose built for the task at hand - using WASM code.
FrustratedMonky 3y ago
I was also thrown off by the use of '4th Gen' since that was the exact term used for new languages in the 90's. I kept trying to fit this into context with the earlier attempts. So this latest should be called '5th-gen' or '6th-gen', if they wanted to relate this to the historical lineage of 'the next great time saving way to generate code'.

78 points by luu 3y ago

What is it with rude and ignorant people who recycle version numbers? There was Web 3.0 https://en.wikipedia.org/wiki/Semantic_Web https://en.wikipedia.org/wiki/Semantic_Web and the blockheads totally ignored it. (e.g. I had a Web 3.0 site a decade before they did!) Similarly there was a lot of talk about 4GL in the 1980s https://en.wikipedia.org/wiki/Fourth-generation_programming_language https://en.wikipedia.org/wiki/Fourth-generation_programming_... some of which was related to the same symbolic A.I. ideas that were influential to the semantic web.

linguae 3y ago
I think a major part of it is a lack of knowledge of history among many people in computer science and software engineering, even researchers. Alan Kay, the creator of the Smalltalk programming language and one of my personal heroes, has lamented many times over the years that a "pop culture" has developed in the software engineering and computer science world. This culture is focused on the present and often disregards the past. It's more than the overloading of terminology like "Web 3.0" and "fourth-generation programming languages." It's an overall attitude that focuses solely on the fad du jour and lacks a deeper understanding of computing. I've noticed a lack of knowledge of computing history among many people who work in computing professions. Personally I think all undergraduate students of computer science and related disciplines would benefit immensely from a one-semester history of computing course. They'd start by learning about early analog computing devices before learning about two very crucial developments in the 1930's and 1940's: (1) the invention of early digital computers such as the ENIAC, and (2) meta-mathematics and how that led to Godel's incompleteness theorem, the lambda calculus, Turing machines, and other fundamental concepts that underpin theoretical computer science. Then they'd learn about the rise of the computer industry, the history of software, the history of high-level programming languages, the history of AI, advances in computer architecture, the birth of the Internet, the GUI, the World Wide Web, social networking, and modern topics such as deep neural networks and large language models. It would be a lot of ground to cover and it would be impossible to cover everything in just a single course, but the goal of such a course is for practitioners in computer science and software engineering to have a basic understanding of the history of their profession and why things are the way that they are today. I also believe that those fortunate enough to work in or visit Silicon Valley should make a visit to the Computer History Museum in Mountain View to get a taste of how far computing has come in just a relatively short amount of time compared to all of human history.
- Filligree 3y ago
  > I also believe that those fortunate enough to work in or visit Silicon Valley should make a visit to the Computer History Museum in Mountain View to get a taste of how far computing has come in just a relatively short amount of time compared to all of human history. Got to second this. That museum was the highlight of my visit, and I still remember everything in detail.
IshKebab 3y ago
"blockheads" seems quite rude to me. In any case the semantic web was ignored by almost everyone because the effort of wading through wanky over-engineered specifications was never worth the approximately zero benefits it actually gave. Presumably there will be a third Web 3.0 by people who also ignore the blockchain nonsense which also never amounted to anything!
deleted 3y ago
[deleted]
seydor 3y ago
Semantic web was so hated and condemned to damnatio memoriae
CyberDildonics 3y ago
It's just nonsense for clicks, don't take it too seriously
FFP999 3y ago
But neither SemWeb nor 4GL ever lived up to a fraction of the hype around them, so it's understandable if people have forgotten, or never learned about, these evolutionary dead ends.

isaiahg 3y ago

This is something I’ve been thinking about for some time, a new type of programming that uses an ai model for parsing. It’s a interesting thought that can lead down many rabbit holes. You could have AI interpret a lot of expressiveness into a language while having an optimizing AI turn whatever crazy thing we come up with into fast code. For any prospective language designers out there. Please don’t make human based language programming a thing again. That’s an idea that should stay in the waste bin. A language that describes programs shouldn’t be ambiguous. I also wouldn’t trust any AI—no matter how smart—to create code I’d be happy with, unless I could go into very specific detail. No limits on structure serialization though. Make that a one liner please

random3 3y ago

Oh, so this isn't about these 4GLs https://en.wikipedia.org/wiki/Fourth-generation_programming_language https://en.wikipedia.org/wiki/Fourth-generation_programming_... This said, I'm not sure why we'd call natural language 4th gen it's very much the first gen.

lifthrasiir 3y ago
The original concepts of 4GL and 5GL are long gone [1], and I do think it is reasonable to reuse the 4GL label for other possibilities. [1] In the sense that those labels do not describe the actual "generation". There still exist visual programming (4GL) and constraint and logic programming (5GL), but they are no longer considered to delieve the goal of 4GL and 5GL by its own, and thus have failed to generate enough interest to be considered a "generation".
- mitjam 3y ago
  I think 4gl is alive and delivering on its goal of manipulating and analyzing a lot of data with few statements, for example in the form of SQL, R, Pandas, Mathematica, Wolfram Alpha, Power Query, and many more. Nobody calls it that way, anymore, however.
  - random3 3y ago
    Agreed. SQL with relational algebras is enough to bear the name. This said - less a matter of whether something else should be 4th or fifth, I think the concern is with making everything an idiocracy-like marketing bingo like the 5D cinema https://www.xd-cinema.com/the-difference-between-4d-5d-6d-7d-8d-9d-xd-cinema/ https://www.xd-cinema.com/the-difference-between-4d-5d-6d-7d...

ixaxaar 3y ago

I am not so sure if the determinism offered by programming languages is more meant for a human requirement or not - if it is, making programming languages more "natural" will actually backfire because a human ultimately needs to read it and understand. Code is meant to be mostly read not written. And no, LLM explanations are shit useless, a language model simply cannot handhold a human to make them understand logic line by line, you NEED to read it, and hence code has to be "readable", which to an engineer means "precise".

anonzzzies 3y ago
> LLM explanations are shit useless, For you and for me maybe, but by now I know a lot of people I met at meetups who never programmed but who are creating SaaS etc systems now just by having a chat with chatgpt. They cannot code without, they have the llm explain line by line. They don't understand when they read it (mostly Python/React/JS) without the llm explaining. And my worst nightmare (as someone who likes precise, static typing, strong typing and even formal proofs now and then) which seems to rapidly becoming reality; read/see no code at all; it's just nanoservice blackbox (just single endpoints) stuff written by chatgpt and the 'programmer' never sees the code, just the input and output.
- dkersten 3y ago
  How well does this work for them in practice? Try as I might, the great I could get ChatGPT to generate was what I would consider student quality code. Good enough for learning, maybe ok as a starting point, but would never pass code review at my day job and it also tended to break in subtle and not so subtle ways when additional requirements or bug fixes were prompted for. I did find it was good at explaining code, but I was also able to read the code alongside its explanations since I have been programming for two decades, and I did find it useful for familiarising myself with unknown codebases (although the tooling for this needs to be worked on). I also find it’s quite capable at little self contained code snippets (useful for bits of data mungeing in Python for example, but not quite able to build a full product). Presumably these people are generating snippets of stuff and gluing them together. I guess that can work, but I can’t imagine the output is particularly good or bug free.

canvascritic 3y ago

Haha I love this, great read. The concept of llms serving as a fourth generation compiler is not really novel. I think we saw Databricks in here talking about LLMs as the new compiler and Python as the new bytecode. Anyway if we're to follow this line of thought, it might not be long before large language models begin interfacing directly with machine hardware, bypassing the need for languages like python and java altogether. Like what purpose would they serve? This could streamline computational processes, reducing overheads and inefficiencies. As the distance between the llm and hardware closes, human interaction with these systems might evolve to use even higher-level languages, further abstracting us from the underlying mechanics. it's conceivable that, in the hopefully distant future our interaction could advance to direct neural or brain interfaces, where our thoughts translate into commands without the need for any traditional "coding" at all. Instead the spec becomes the software. These are damn exciting times

richardjam73 3y ago

I don't think that their usage of 4th Generation Programming Languages is incorrect. The paper shows cleary that they understand the generations. The conclusion reads more like LLMs can translate from an human prompt to a 3GL such as python. The prompt is not quite human language but is rather a specific form of writing designed to get the output needed from the LLM, in this way it is a 4GL and the LLM is the compiler.

Smaug123 3y ago

twitch I thought the Queen was greedy and had very little patience - why are we traversing the input list twice in both generated responses?

sheepscreek 3y ago

I’ve been imagining a slightly different hypothetical future. - LLMs and GenAI can potentially replace many OS layers - They can translate text directly to machine code, or a portable representation like LLVM IR/MLIR - They may do this by making system calls directly - or have a lightweight kernel of its own supporting basic GUI primitives and web-views Even better, imagine an LLM fused with a web-browser core. Like Chrome OS, but with LLM powered interaction. Capable of conjuring windows with rich GUIs that are purpose built for the task at hand - using WASM code.

FrustratedMonky 3y ago

I was also thrown off by the use of '4th Gen' since that was the exact term used for new languages in the 90's. I kept trying to fit this into context with the earlier attempts. So this latest should be called '5th-gen' or '6th-gen', if they wanted to relate this to the historical lineage of 'the next great time saving way to generate code'.