3 ms·

How come it's "not representative for real engineering"? Other than copy-pasting existing code (which is not what an LLM does), I don't see how you can create a

How come it's "not representative for real engineering"? Other than copy-pasting existing code (which is not what an LLM does), I don't see how you can create a space invaders game without applying "engineering".

phkahler 1y ago
>> Other than copy-pasting existing code (which is not what an LLM does) I'd like to see someone try to prove this. How many space invaders projects exist on the internet? I'd be hard to compare model "generated" code to everything out there looking for plagiarism, but I bet there are lots of snippets pulled in. These things are NOT smart, they are huge and articulate information repositories.
- simonw 1y ago
  Go for it. https://www.google.com/search?client=firefox-b-1-d&q=github+space+invaders https://www.google.com/search?client=firefox-b-1-d&q=github+... has a bunch of results. Here's the source code GLM-4.5 Air spat out for me on my laptop: https://github.com/simonw/tools/blob/main/space-invaders-GLM-4.5-Air-3bit.html https://github.com/simonw/tools/blob/main/space-invaders-GLM... Based on my mental model of how these things work I'll be genuinely surprised if you can find even a few lines of code duplicated from one of those projects into the code that GLM-4.5 wrote for me.
  - phkahler 1y ago
    So I scanned the beginning of the generated code, picked line 83: animation: glow 2s ease-in-out infinite; stuffed it verbatim into google and found a stack overflow discussion that contained this: animation: glow .5s infinite alternate; in under one minute. Then I found this page of CSS effects: https://alvarotrigo.com/blog/animated-backgrounds-css/ https://alvarotrigo.com/blog/animated-backgrounds-css/ Another page has examples and contains: animation: float 15s infinite ease-in-out; There is just too much internet to scan for an exact match or a match of larger size.
    - simonw 1y ago
      That's not an example of copying from an existing Space Invaders implementation. That's an LLM using a CSS animation pattern - one that it's seen thousands (probably millions) of times in the training data. That's what I expect these things to do: they break down Space Invaders into the components they need to build, then mix and match thousands of different coding patterns (like "animation: glow 2s ease-in-out infinite;") to implement different aspects of that game. You can see that in the "reasoning" trace here: https://gist.github.com/simonw/9f515c8e32fb791549aeb88304550893#file-space_invaders-txt-L41 https://gist.github.com/simonw/9f515c8e32fb791549aeb88304550... - "I'll use a modern design with smooth animations, particle effects, and a retro-futuristic aesthetic."
      - threeducks 1y ago
        I think LLMs are adapting higher level concepts. For example, the following JavaScript code generated by GLM (https://github.com/simonw/tools/blob/9e04fd9895fae1aa9ac78b8e62d2833831fe0544/space-invaders-GLM-4.5-Air-3bit.html#L387-L405 https://github.com/simonw/tools/blob/9e04fd9895fae1aa9ac78b8...) is clearly inspired by this C++ code (https://github.com/portapack-mayhem/mayhem-firmware/blob/28e2f770e5f9cb60021a1ddfe12cdc1a1f82d38f/firmware/application/external/spaceinv/ui_spaceinv.cpp#L468-L486 https://github.com/portapack-mayhem/mayhem-firmware/blob/28e...), but it is not an exact copy.
        simonw 1y ago
        This is a really good spot. That code certainly looks similar, but I have trouble imagining how else you would implement very basic collision detection between a projectile and a player object in a game of this nature.
        threeducks 1y ago
        A human would likely have refactored the two collision checks between bullet/enemy and enemyBullet/player in the JavaScript code into its own function, perhaps something like "areRectanglesOverlapping". The C++ code only does one collision check like that, so it has not been refactored there, but as a human, I certainly would not want to write that twice. More importantly, it is not just the collision check that is similar. Almost the entire sequence of operations is identical on a higher level: 1. enemyBullet/player collision check 2. same comment "// Player hit!" (this is how I found the code) 3. remove enemy bullet from array 4. decrement lives 5. update lives UI 6. (createParticle only exists in JS code) 7. if lives are <= 0, gameOver
  - deleted 1y ago
    [deleted]
- ben_w 1y ago
  Sorites paradox. Where's the distinction between "snippet" and "a design pattern"? Compressing a few petabytes into a few gigabytes requires that they can't be like this about all of the things they're accused of simply copy-pasting, from code to newspaper articles to novels. There's not enough space.
hbn 1y ago
The prompt was > Write an HTML and JavaScript page implementing space invaders It may not be "copy pasting" but it's generating output as best it can be recreated from its training on looking at Space Invaders source code. The engineers at Taito that originally developed Space Invaders were not told "make Space Invaders" and then did their best to recall all the source code they've looked at in their life to re-type the source code to an existing game. From a logistics standpoint, where the source code already exists and is accessible, you may as well have copy-pasted it and fudged a few things around.
- simonw 1y ago
  The source code for original Space Invaders from 1978 has never been published. The closest to that is disassembled ROMs. I used that prompt because it's the shortest possible prompt that tells the model to build a game with a specific set of features. If I wanted to build a custom game I would have had to write a prompt that was many paragraphs longer than that. The aim of this piece isn't "OMG looks LLMs can build space invaders" - at this point that shouldn't be a surprise to anyone. What's interesting is that my laptop can run a model that is capable of that now.
  - sarchertech 1y ago
    > The source code for original Space Invaders from 1978 has never been published. The closest to that is disassembled ROMs. Sure but that doesn’t impact the OPs point at all because there are numerous copies of reverse engineered source code available. There are numerous copies of the reverse engineered source code already translated to JavaScript in your models training set.
  - nottorp 1y ago
    > What's interesting is that my laptop can run a model that is capable of that now. I'm afraid no one cared much about your point :) You'll only get "OMG look how good LLMs are they'll get us all fired!" comments and "LLMs suck" comments. This is how it goes with religion...
  - hbn 1y ago
sharkjacobs 1y ago
Making a space invaders game is not representative of normal engineering because you're reproducing an existing game with well known specs and requirements. There are probably hundreds of thousands of words describing and discussing Space Invaders in GLM-4.5's training data It's like using an LLM to implement a red black tree. Red black trees are in the training data, so you don't need to explain or describe what you mean beyond naming it. "Real engineering" with LLMs usually requires a bunch of up front work creating specs and outlines and unit tests. "Context engineering"
- jasonvorhe 1y ago
  Smells like moving the goal post. What's real engineering to be in 2028? Implementing Google's infra stack in your homelab?

phkahler 1y ago

>> Other than copy-pasting existing code (which is not what an LLM does) I'd like to see someone try to prove this. How many space invaders projects exist on the internet? I'd be hard to compare model "generated" code to everything out there looking for plagiarism, but I bet there are lots of snippets pulled in. These things are NOT smart, they are huge and articulate information repositories.

simonw 1y ago
Go for it. https://www.google.com/search?client=firefox-b-1-d&q=github+space+invaders https://www.google.com/search?client=firefox-b-1-d&q=github+... has a bunch of results. Here's the source code GLM-4.5 Air spat out for me on my laptop: https://github.com/simonw/tools/blob/main/space-invaders-GLM-4.5-Air-3bit.html https://github.com/simonw/tools/blob/main/space-invaders-GLM... Based on my mental model of how these things work I'll be genuinely surprised if you can find even a few lines of code duplicated from one of those projects into the code that GLM-4.5 wrote for me.
- phkahler 1y ago
  So I scanned the beginning of the generated code, picked line 83: animation: glow 2s ease-in-out infinite; stuffed it verbatim into google and found a stack overflow discussion that contained this: animation: glow .5s infinite alternate; in under one minute. Then I found this page of CSS effects: https://alvarotrigo.com/blog/animated-backgrounds-css/ https://alvarotrigo.com/blog/animated-backgrounds-css/ Another page has examples and contains: animation: float 15s infinite ease-in-out; There is just too much internet to scan for an exact match or a match of larger size.
  - simonw 1y ago
    That's not an example of copying from an existing Space Invaders implementation. That's an LLM using a CSS animation pattern - one that it's seen thousands (probably millions) of times in the training data. That's what I expect these things to do: they break down Space Invaders into the components they need to build, then mix and match thousands of different coding patterns (like "animation: glow 2s ease-in-out infinite;") to implement different aspects of that game. You can see that in the "reasoning" trace here: https://gist.github.com/simonw/9f515c8e32fb791549aeb88304550893#file-space_invaders-txt-L41 https://gist.github.com/simonw/9f515c8e32fb791549aeb88304550... - "I'll use a modern design with smooth animations, particle effects, and a retro-futuristic aesthetic."
    - threeducks 1y ago
      I think LLMs are adapting higher level concepts. For example, the following JavaScript code generated by GLM (https://github.com/simonw/tools/blob/9e04fd9895fae1aa9ac78b8e62d2833831fe0544/space-invaders-GLM-4.5-Air-3bit.html#L387-L405 https://github.com/simonw/tools/blob/9e04fd9895fae1aa9ac78b8...) is clearly inspired by this C++ code (https://github.com/portapack-mayhem/mayhem-firmware/blob/28e2f770e5f9cb60021a1ddfe12cdc1a1f82d38f/firmware/application/external/spaceinv/ui_spaceinv.cpp#L468-L486 https://github.com/portapack-mayhem/mayhem-firmware/blob/28e...), but it is not an exact copy.
      - simonw 1y ago
        This is a really good spot. That code certainly looks similar, but I have trouble imagining how else you would implement very basic collision detection between a projectile and a player object in a game of this nature.
        threeducks 1y ago
        A human would likely have refactored the two collision checks between bullet/enemy and enemyBullet/player in the JavaScript code into its own function, perhaps something like "areRectanglesOverlapping". The C++ code only does one collision check like that, so it has not been refactored there, but as a human, I certainly would not want to write that twice. More importantly, it is not just the collision check that is similar. Almost the entire sequence of operations is identical on a higher level: 1. enemyBullet/player collision check 2. same comment "// Player hit!" (this is how I found the code) 3. remove enemy bullet from array 4. decrement lives 5. update lives UI 6. (createParticle only exists in JS code) 7. if lives are <= 0, gameOver
- deleted 1y ago
  [deleted]
ben_w 1y ago
Sorites paradox. Where's the distinction between "snippet" and "a design pattern"? Compressing a few petabytes into a few gigabytes requires that they can't be like this about all of the things they're accused of simply copy-pasting, from code to newspaper articles to novels. There's not enough space.

hbn 1y ago

The prompt was > Write an HTML and JavaScript page implementing space invaders It may not be "copy pasting" but it's generating output as best it can be recreated from its training on looking at Space Invaders source code. The engineers at Taito that originally developed Space Invaders were not told "make Space Invaders" and then did their best to recall all the source code they've looked at in their life to re-type the source code to an existing game. From a logistics standpoint, where the source code already exists and is accessible, you may as well have copy-pasted it and fudged a few things around.

simonw 1y ago
The source code for original Space Invaders from 1978 has never been published. The closest to that is disassembled ROMs. I used that prompt because it's the shortest possible prompt that tells the model to build a game with a specific set of features. If I wanted to build a custom game I would have had to write a prompt that was many paragraphs longer than that. The aim of this piece isn't "OMG looks LLMs can build space invaders" - at this point that shouldn't be a surprise to anyone. What's interesting is that my laptop can run a model that is capable of that now.
- sarchertech 1y ago
  > The source code for original Space Invaders from 1978 has never been published. The closest to that is disassembled ROMs. Sure but that doesn’t impact the OPs point at all because there are numerous copies of reverse engineered source code available. There are numerous copies of the reverse engineered source code already translated to JavaScript in your models training set.
- nottorp 1y ago
  > What's interesting is that my laptop can run a model that is capable of that now. I'm afraid no one cared much about your point :) You'll only get "OMG look how good LLMs are they'll get us all fired!" comments and "LLMs suck" comments. This is how it goes with religion...
- hbn 1y ago

sharkjacobs 1y ago

Making a space invaders game is not representative of normal engineering because you're reproducing an existing game with well known specs and requirements. There are probably hundreds of thousands of words describing and discussing Space Invaders in GLM-4.5's training data It's like using an LLM to implement a red black tree. Red black trees are in the training data, so you don't need to explain or describe what you mean beyond naming it. "Real engineering" with LLMs usually requires a bunch of up front work creating specs and outlines and unit tests. "Context engineering"

jasonvorhe 1y ago
Smells like moving the goal post. What's real engineering to be in 2028? Implementing Google's infra stack in your homelab?