12 ms·

I tried vibe coding in BASIC and it didn't go well

160 points by ibobev 1y ago

dogman1050 1y ago
It never did draw a blue circle, it's orange or similar but that's never mentioned in the article.
heisenbit 1y ago
Maybe the Basic code out there has a too high similarity with spaghetti making it hard to abstract in latent space. OO and functional patterns are more localized.
firesteelrain 1y ago
Not surprised; there were so many variations of BASIC and unless you train ChatGPT on a bunch of code examples and contexts then it can only get so close. Try a local LLM then train it
- ofrzeta 1y ago
  > ... unless you train ChatGPT on a bunch of code examples and contexts then it can only get so close. How do you do this?
  - sixothree 1y ago
    RAG maybe?
    - firesteelrain 1y ago
      RAG is good suggestion to pull in runtime without weights
  - oharapj 1y ago
    If you're OpenAI you scrape StackOverflow and GitHub and spend billions of dollars on training. If you're a user, you don't
  - firesteelrain 1y ago
    Gist is 1. Gather training data 2. Format it into JSONL or Hugging Face Dataset format 3. Use Axolotl or Hugging Face peft to fine-tune 4. Export model to GGUF or HF format 5. Serve via Ollama https://adithyask.medium.com/axolotl-is-all-you-need-331d5de1b7c0 https://adithyask.medium.com/axolotl-is-all-you-need-331d5de... https://www.philschmid.de/fine-tune-llms-in-2025 https://www.philschmid.de/fine-tune-llms-in-2025 https://blog.devgenius.io/complete-guide-to-model-fine-tuning-from-hugging-face-to-axolotl-dc5cdaf1ae27 https://blog.devgenius.io/complete-guide-to-model-fine-tunin...
    - ofrzeta 1y ago
      So, finetuning? Not so easy with ChatGPT I guess, but thanks for the info anyway.
      - firesteelrain 1y ago
        Yes, it takes an existing model and fine tunes it. ChatGPT would be basically extensively prompt engineering in a session. Maybe using their API? I have never tried it personally. When I fine tuned a Mistral 7B model it took hundreds of examples in Alpaca style It’s a lot of work. Maybe OpenAI has a more efficient way of doing it because in my case I had to manually adjust each prompt
  - Paradigma11 1y ago
    Gemini Pro 2.5 has a context window of 1 million tokens and wants to rise that to 2 million tokens soon. 1 token is approx 0.75 words, so 1 million tokens would be in the ballpark of 3k pages of code. You can add some tutorials/language docs as context without any problem. The bigger your project gets the more context it gets from there. You can also convert apis/documentation to a RAG and expose it as a MCP tool to the LLM.
    - ofrzeta 1y ago
      > Gemini Pro 2.5 has a context window of 1 million tokens and wants to rise that to 2 million tokens soon. 1 token is approx 0.75 words, so 1 million tokens would be in the ballpark of 3k pages of code. You mean around 3000 files with 3000 characters? That is a lot. I've played with some other LLMs in Agentic AIs but at work we are using Copilot, and when I add context through drag and drop it seems to be limited to some dozen files.
      - Paradigma11 1y ago
        I think Copilot has some hardcoded limitations of around a dozen files, like you said. But this stuff changes constantly.
        ofrzeta 1y ago
        Still I don't totally understand how that huge of a context works for Gemini. I guess you don't provide the whole context for every request? So it keeps (but also updates) context for a specific session?
        Paradigma11 1y ago
        I dont know how the massive context works but Caching is certainly a thing and cheaper: https://ai.google.dev/gemini-api/docs/caching?lang=python https://ai.google.dev/gemini-api/docs/caching?lang=python Gemini is better than Sonnet if you have broad questions that concern a large codebase, the context size seems to help there. People also use subagents for specific purposes to keep each context size manageable, if possible. On a related note I think the agent metaphor is a bit harmful because it suggests state while the LLM is stateless.
empressplay 1y ago
We have Claude Code writing Applesoft BASIC fine. It wrote a text adventure (complete with puzzles) and a PONG clone, among other things. Obviously it didn't do it 100% right straight out of the gate, but the hand-holding wasn't extensive. I've been using Grok 4 to write 6502 assembly language and it's been a bit of a slog but honestly the issues I've encountered are due mostly my to naivety. If I'm disciplined and make sure it has all of the relevant information and I'm (very) incremental, I've had some success writing game logic. You can't just tell it to build an entire game in a prompt, but if you're gradual about it you can go places with it. Like any tool, if you understand its idiosyncrasies you can cater for them, and be productive with it. If you're not then yeah, it's not going to go well.
- hammyhavoc 1y ago
  Ah yes, truly impressive, Pong. A game that countless textbooks et al have recreated numerous times. There's a mountain of training data for something so unoriginal.
- deleted 1y ago
  [deleted]
recipe19 1y ago
I work on niche platforms where the amount of example code on Github is minimal, and this definitely aligns with my observations. The error rate is way too high to make "vibe coding" possible. I think it's a good reality check for the claims of impending AGI. The models still depend heavily on being able to transform other people's work.
- empressplay 1y ago
  I don't know if you're working with modern models. Grok 4 doesn't really know much about assembly language on the Apple II but I gave it all of the architectural information it needed in the first prompt of a conversation and it built compilable and executable code. Most of the issues I encountered were due to me asking for too much in a prompt. But it built a complete, albeit simple, assembly language game in a few hours of back and forth with it. Obviously I know enough about the Apple II to steer it when it goes awry, but it's definitely able to write 'original' code in a language / platform it doesn't inherently comprehend.
  - timschmidt 1y ago
    This matches my experience as well. Poor performance usually means I haven't provided enough context or have asked for too much in a single prompt. Modifying the prompt accordingly and iterating usually results in satisfactory output within the next few tries.
- gompertz 1y ago
  Yep I program in some niche languages like Pike, Snobol4, Unicon. Vibe coding is out of the question for these languages. Forced to use my brain!
  - johnisgood 1y ago
    You could always feed it some documentation and example programs. I did it with a niche language and it worked out really well, with Claude. Around 8 months ago.
- 1y ago
docandrew 1y ago
Maybe other folks’ vibe coding experiences are a lot richer than mine have been, but I read the article and reached the opposite conclusion of the author. I was actually pretty impressed that it did as well as it did in a largely forgotten language and outdated platform. Looks like a vibe coding win to me.
- sixothree 1y ago
  Here's an example of a recent experience. I have a web site that is sort of a cms. I wanted users to be able to add a list of external links to their items. When a user adds a link to an entry, the web site should go out and fetch a cached copy of the site. If there are errors, it should retry a few times. It should also capture an mhtml single file as well as a full page screenshot. The user should be able to refresh the cache, and the site should keep all past versions. The cached copy should be viewable in a modal. The task also involves creating database entities, DTOs, CQRS handlers, etc. I asked Claude to implement the feature, went and took a shower, and when I came out it was done.
  - hammyhavoc 1y ago
    Let us know how the security audit by human beings on the output goes.
    - catmanjan 1y ago
      The auditors are using llms too!
    - sixothree 1y ago
      It's really just a personal project for myself. Why else would I add that feature without any guardrails?
  - nico 1y ago
    Im pretty new to CC, been using it in a very interactive way. What settings are you using to get it to just do all of that without your feedback or approval? Are you also running it inside a container, or setting some sort of command restrictions, or just yoloing it on a regular shell?
serf 1y ago
please just include the prompts rather than saying "So I said X.." There is a lot of nuance in how X is said.
pavelstoev 1y ago
I vibe coded a site about vibe 2 code projects. https://builtwithvibe.com/ https://builtwithvibe.com/
- esafak 1y ago
  The "Yo dawg, I heard..." memes are writing themselves today.
clambaker117 1y ago
Wouldn’t it have been better to use Claude 4?
- sixothree 1y ago
  I'm thinking Gemini CLI because of the context. He could add some information about the programming language itself in the project. I think that would help immensely.
  - 4b11b4 1y ago
    Even though the max token limit is higher, it's more complicated than that. As the context length increases, undesirable things happen.
calvinmorrison 1y ago
This is great. I am currently vibecoding a replacement connector for some old EDI software that is written in a business basic kind of language called ProvideX, and a fork of that has undocumented behaviour. It uses some built inet ftp tooling thats terrible and barely works, even internally anymore. We are replacing it with a winscp implementation since winscp can talk over a COM object. unsuprisingly the COM object in basic works great - the problem is that I have no idea what I am doing. I spent hours doing something like WINSCP_SESSION'OPEN(WINSCP_SESSION_OPTIONS) when i needed WINSCP_SESSION'OPEN(*WINSCP_SESSION_OPTIONS) It was obvious after because it was a pointer type of setup, but i didnt find it until pages and pages deep into old PDF manuals. However the vibecode of all the agents did not understand the syntax of the system, it did help me analyse the old code, format it, and at least throw some stuff at the wall. I finished it up friday, hopefully i deploy monday.
deleted 1y ago
[deleted]
CMay 1y ago
This does kind of make me wonder. It's believable that we might either see an increase in the number of new programming languages since making new languages is becoming more accessible, or we could see fewer new languages as the problems of the existing ones are worked around more reliably with LLMs. Yet, what happens to adoption? Perhaps getting people to adopt new languages will be harder as generations come to expect LLM support. Would you almost need to use LLMs to synthesize tons of code examples that convert into the new language to prime the inputs? Once conversational intelligence machines reach a sort of godlike generality, then maybe they could very quickly adapt languages from much fewer examples. That still might not help much with the gotchas of any tooling or other quirks. So maybe we'll all snap to a new LLM super-language in 20 years, or we could be concreting ourselves into the most popular languages of today for the next 50 years.
- hammyhavoc 1y ago
  Fantasy.
  - CMay 1y ago
    Can you elaborate?
    - hammyhavoc 1y ago
      > Once conversational intelligence machines reach a sort of godlike generality, Sums up fantastical inevitablism. Why would they? How could they? The data is telling us that they won't. Anybody who believes otherwise is ignoring the science. The assumed path to "AGI" was more tokens, but more tokens actually means worse output, and if LLMs aren't a total technological dead-end, period, then the data is supporting smaller models meant for more specific things, ergo, LLMs are not giving AGI any time, ever. Pure fantasy. Can't even call it sci-fi when it ignores the science entirely.
      - CMay 1y ago
        Ah, so you misread. I said conversational intelligence machines, but you focused on LLMs. I don't know where LLMs will lead, but I haven't ruled out the possibility of improvements continuing to surprise us. If anything is unscientific, it would be overconfidence in one way or the other.
ilaksh 1y ago
I think it's a fair article. However I will just mention a few things. When you make an article like this please take note of the particular language model used and acknowledge that they aren't all the same. Also realize that the context window is pretty large and you can help it by giving it information from manuals etc. so you don't need to rely on the intrinsic knowledge entirely. If they used o3 or o3 Pro and gave it a few sections of the manual it might have gotten farther. Also if someone finds a way to connect an agent to a retro computer, like an Atari BASIC MCP that can enter text and take screenshots, "vibe coding" can work better as an agent that can see errors and self-correct.
manca 1y ago
I literally had the same experience when I asked the top code LLMs (Claude Code, GPT-4o) to rewrite the code from Erlang/Elixir codebase to Java. It got some things right, but most things wrong and it required a lot of debugging to figure out what went wrong. It's the absolute proof that they are still dumb prediction machines, fully relying on the type of content they've been trained on. They can't generalize (yet) and if you want to use them for novel things, they'll fail miserably.
- hammyhavoc 1y ago
  They'll never be fit for purpose. They're a technological dead-end for anything like what people are usually throwing them at, IMO.
  - zer00eyz 1y ago
    I will give you an example of where you are dead wrong, and one where the article is spot on (without diving into historic artifacts). I run HomeAssistant, I don't get to play/use it every day. Here, LLM's excel at filling in the (legion) of blanks in both the manual and end user devices. There is a large body of work for it to summarize and work against. I also play with SBC's. Many of these are "fringe" at best. LLM's are as you say "not fit for purpose". What kind of development you are using LLM's for will determine your experience with them. The tool may or may not live up to the hype depending how "common", well documented and "frequent" your issue is. Once you start hitting these "walls" you realize that no, real reason, leaps of inference and intelligence are still far away.
    - SecuredMarvin 1y ago
      I also made this experience. As long as the public level of knowledge is high, LLMs are massively helpful. Otherwise not so much and still hallucinating. It does not matter if you think highly of this public knowledge. QFT, QED and Gravity are fine, AD emulation on SAMBA, or Atari Basic not so much. If I would program Atari Basic, after finishing my Atari Emulator on my C64, I would learn the environment and test my assumptions. Single shot LLMs questions won't do it. A strong agent loop could probably. I believe that LLMs are yanking the needle to 80%. This level is easy achievable for professionals of the trade and this level is beyond the ability of beginners. LLMs are really powerful tools here. But if you are trying for 90% LLMs are always trying to keep you down. And if you are trying for 100%, new, fringe or exotic LLMs are a disaster because they do not learn and do not understand, even while being inside the token window. We learn that knowledge, (power) and language proficiency are an indicator for crystalline but not fluid intelligence
ofrzeta 1y ago
It didn't go well? I think it went quite well. It even produced an almost working drawing program.
- abrookewood 1y ago
  Yep, thought the same thing. I guess people have very different expectations.
Radle 1y ago
I had way better results. I'd assume the same would have happened to the author if he provided the LLM with a full documentation on what ATARI BASIC is and some example programs. Especially when asking the LLM to create a drawing program and a game the author would have probably received working code if he supplied the ai with documentation to the graphics function and sprite rendering using ATARI BASIC.
xiphias2 1y ago
4o is not even a coding model and very far from the best coding models OpenAI has, I seriously don't understand why these articles are upvoted so much
- throw101010 1y ago
  > I seriously don't understand why these articles are upvoted so much It confirm a bias for some, it triggers others who might have the opposite position (and maybe have a bias too on the other end). Perfect combo for successful social media posts... literally all about "attention" from start to finish.
- deleted 1y ago
  [deleted]
- timeon 1y ago
  Short while ago it was like "you are still using <X>? Why not the new 4o?!"
- layer8 1y ago
  A coding model doesn't seem to produce better results: https://news.ycombinator.com/item?id=44624222 https://news.ycombinator.com/item?id=44624222
fcatalan 1y ago
I had more luck with a little experiment a few days ago: I took phone pics of one of the shorter BASIC listings from Tim Hartnell's "Giant Book of Computer Games" (I learned to program out of those back in the early 80s, so I treasure my copy) and asked Gemini to translate it to plain C. It compiled and played just fine on the first go.
edent 1y ago
Vibe Coding seems to work best when you are already an experienced programmer. For example "Prompt: Write me an Atari BASIC program that draws a blue circle in graphics mode 7." You need to know that there are various graphics modes and that mode 7 is the best for your use-case. Without that preexisting knowledge, you get stuck very quickly.
- throwawaylaptop 1y ago
  Exactly this. I'm a self taught PHP/jQuery guy that learned it well enough to make an entire saas that enough companies pay for that it's a decent little lifestyle business. I started another project recently basically vibe coding in PHP. Instead of a single page app like I made before, it's just page by page single loading. Which means the AI also only needs to keep a few functions and the database in its head, not constantly work on some crazy ui management framework (what that's called). It's made in a few days what would have taken me weeks as an amateur. Yet I know enough to catch a few 'mistakes' and remind it to do it better. I'm happy enough.
- kqr 1y ago
  Not only is it a useful constraint to ask for mode 7, but making sure the context contains domain-expert technology puts the LLM in a better spot in the sampling space.
- cfn 1y ago
  Just for fun I asked ChatGPT "How would you ask an LLM to write a drawing program for the ATARI?" and it asked back a bunch of details to which I answered "I have no idea, just go with the simplest option". It chose the correct graphics mode and BASIC and created the program (which I didn't test). I still agree with you for large applications but for these simple examples anyone with a basic understanding of vibe coding could wing it.
- baxtr 1y ago
  This is a description of a “tool”. Anyone can use a hammer and chisel to carve out wood, but only an artist with extensive experience will create something truly remarkable. I believe many in this debate are confusing tools with magic wands.
Earw0rm 1y ago
Has anyone tried it on x87 assembly language? For those that don't know. x87 was the FPU for 32-bit x86 architectures. It's not terribly complicated, but it uses stack-based register addressing with a fixed size (eight entry) stack. All operations work on the top-of-stack register and one other register operand, and push the result onto the top of the stack (optionally popping the previous top of stack before the push). It's hard but not horribly so for humans to write.. more a case of annoyingly slow and having to be methodical, because you have to reason about the state of the stack at every step. I'd be very curious as to whether a token-prediction machine can get anywhere with this kind of task, as it requires a strong mental model of what's actually happening, or at least the ability to consistently simulate one as intermediate tokens/words.
- silisili 1y ago
  I'm going to doubt that. I was pushing GPT a couple weeks ago to test its limits. It's 100% unable to write compilable Go ASM syntax. In fairness it's slightly oddball, but enough exists that it's not esoteric. In the error feedback cycle, it kept blaming Go, not itself. A bit eye opening.
  - messe 1y ago
    I'm comfortable writing asm for quite a few architectures, but Go's assembler... When I struggle to write Go ASM, I also blame Go and not myself.
  - Earw0rm 1y ago
    The thing with x87 is that it's easy to write compilable, correct-looking code, and much harder to write correct compilable code, even for trivial sequences of a dozen or so operations. Whereas in most asm dialects, register AX is always register AX (word length aliasing aside), that's not the case for x87: the object/value at ST3 in one operation may be ST1 or ST5 in a couple of instructions' time.
- userbinator 1y ago
  If you are familiar with HP's calculators, x87 Asm isn't that difficult. Also noteworthy is that its density makes it a common choice for tiny demoscene productions.
nine_k 1y ago
All these stories about vibe coding going well or wrong remind me of an old joke. A man visits his friend's house. There is a dog in the house. The friend says that the dog can play poker. The man is incredulous, but they sit at a table and have a game of poker; the dog actually can play! The man says: "Wow! Your dog is incredibly, fantastically smart!" The friend answers: "Oh, well, no, he's a naïve fool. Every time he gets a good hand, he starts wagging his tail." Whether you see LLMs impressively smart or annoyingly foolish depends on your expectations. Currently they are very smart talking dogs.
- kqr 1y ago
  Somehow this also reminds me of http://raisingtalentthebook.com/wp-content/uploads/2014/04/taughtmydog.jpg http://raisingtalentthebook.com/wp-content/uploads/2014/04/t... "I taught my dog to whistle!" "Really? I don't hear him whistling." "I said I taught him, not that he learnt it."
  - pyman 1y ago
    A person is flying a hot air balloon and realises he’s lost. He lowers the balloon and spots a man down below. He shouts: “Excuse me! Can you help me? I promised a friend I’d meet him, but I have no idea where I am.” The man replies, “You’re in a hot air balloon, hovering 30 feet above the ground, somewhere between 40 and 41 degrees north latitude and between 59 and 60 degrees west longitude.” “You must be a Prompt Engineer,” says the balloonist. “I am,” replies the man. “How did you know?” “Well,” says the balloonist, “everything you told me is technically correct, but it’s of no use to me and I still have no idea where I am.” The man below replies, “You must be a Vibe Coder.” “I am,” says the balloonist. “How did you know?” "Because you don’t know where you are or where you’re going. You made a promise you can’t keep, and now you expect me to solve your problem. The fact is, you’re in the same position you were in before we met, but now it’s somehow my fault!"
    - johnisgood 1y ago
      This is really good, saving it. :D
Yokolos 1y ago
When I start getting nonsense back and forth prompting, I've found it best to just start a new chat/context with the latest working version and then try again with a slightly more detailed prompt that tries to avoid the issues encountered in the previous chat. It usually helps. AI generally quickly gets itself lost, which can be annoying.
danjc 1y ago
What's missing here is tool use. For example, if the llm had a compile tool it would likely have been able to correct syntax errors. Similarly, visual errors may also have been caught if it were able to run the program and capture screens.
- wbolt 1y ago
  Exactly! The way in which the LLM is used here is very, very basic and outdated. This experiment should be redone in a proper „agentic” setup where there is a feedback loop between the model and the runtime plus access to documentation / internet. The goal now is not to encapsulate all the knowledge inside single LLM - this is too problematic and costly. LLM is a language model not knowledge database. It allows to interpret and interact with knowledge and text data from multiple sources.
486sx33 1y ago
[dead]
Mikhail_Edoshin 1y ago
I don't know if anyone notices that, but LLMs are very much like what you see in dreams when you reflect on them when awake. When you asleep a dream feels very coherent; but when you're awake, you see wild gaps and jumps and the overall impression of many dreams' plot is that it is pure nonsense. I once heard advice on trying to re-read the text you see in dreams. And I did that once. It was a phrase where one of words referred to a city. The first time I read that city was "London". I remembered the advice and re-read the phrase and the word changed to the name of an old Russian city "Rostov". Yet the phrase was "same", that is it felt same in the dream, even though the city was different. LLMs are like that dream mechanics. It is how something else is reflected in what we know (e.g. an image of a city is rendered as a name of a city, just any city). So, on one hand, we do have a similar mechanism in our minds. But on another hand our normal reasoning is very much unlike that. It would be a very wild stretch to believe that reasoning somehow stems from dreaming. I'd say reasoning is opposite to dreaming. If we amplify our dreaming mechanics we won't get a genius; more likely we'll get a schizophrenic.
- neom 1y ago
  I think that is a cool way of looking at it - build on what you said, to me it might be a bit like in both instances (dreaming and llms) - it's maybe a bit related to what they are trying to do (presuming dreams have a purpose) + the resource they have to do it in + the context they have to couple in to get the point across + something related to the abilities of the user? Lets for fun say there is a subsystem that understands and runs dreaming, you only have so much time, plus it's a weird modality, and you're trying to do something... maybe it's fine enough to just serve up the story no matter how muddled it is, what you might be talking about in LLMs is a similar thing? Dreams have a ticking clock where the brain chemistry is literally changing and the opportunity for that type of processing is about to disappear, and eventually the human will awaken, LLMs have a context window size. Fun thinking, anyway.
- Fade_Dance 1y ago
  Well it's neither schizophrenic nor using human reasoning. It's a machine using transformers no reason to overly anthropomorphize it, but sure the parallels do exist to some degree. These tools are obviously not reasoning tools in the classic sense, as they're not building off of core axiomic truths like classic logic. These truths may be largely embedded in the probabilistic output, due to the input universe that we feed it being based on human reason, but it's certainly not in a power of these tools. That said, of course we are attacking on more and more of this ability (ability to check databases of facts, iterative "reasoning" models, etc) so they are becoming "adequate" in these respects. The dreaming comparison seems quite apt though. I entirely get what you mean by rereading the dream word and seeing it suddenly transformed to another city name, yet somehow also "fitting". For some reason I'm keenly aware of these sorts of relationships when I think about my dreams. I will think of a situation and immediately be able to identify the input memories that "built" the dream scenario. Usually they involve overlapping themes and concepts, as well as some human specific common targets like "unresolved, emotionally charged, danger, etc" (presumably running through these types of brain neurons provides some sort of advantage for mammals, which makes sense to me). What an LLM does is essentially create a huge interconnected conceptual web of the universe which is fed, and then uses probably ballistic models to travel through these chains, much like how dreaming does a trance like dance through these conceptual connections. In the former example though, we have optimized the traversal to be as close to a "mock awake human" as possible. If the dream poem is dreary in nature, and Rostov sounds dreary, and you were hearing about the dreary London rain earlier in the day, and you have a childhood memory of reading through a dreary poem that made you very sad, that's the perfect sort of overlapping set of memory synapses for a dream to light up. And when looking back, all you'll see is a strange fantasmic and (usually, not always) frustratingly inaccessible conglomeration of these inputs. This sort of traversal isn't just used in dreaming though. To some degree we're doing similar things when we do things like creative thinking. The difference is, and this is especially so in "modern" life, we're strongly filtering our thoughts through language first and foremost (and indeed there's a lot of philosophical and scientific work done about how extremely important language is to humanness), but also through basic logic. LLM's inherent some of the power/magic of language, in that it is deconstructing the relationships between the concepts behind language that contains so much embedded meaning. But they aren't filtering through logic like we do. Well, reasoning models do to some degree, but it is obviously quite rudimentary. I think it's a good analogy.
register 1y ago
I'll tell you a secret. The same happens also with mainstream languages
ghuntley 1y ago
It's not really 'vibe coding' if you're copying and pasting from ChatGPT by hand...
alfiedotwtf 1y ago
I’ve been using my AirPods to talk to ChatGPT while I drive into work.. not coding talk though, using that time to pick up particle physics. So far, nothing looks hallucinated, though I’m only touching the surface and haven’t looked at equations yet. .. either way, I’m super happy that it has kept my drives to work very interesting!
ofrzeta 1y ago
Even though I was impressed with the original article (so kind of contrary to the author) in the meantime I tried the same thing with Claude Sonnet 4 (because some people here criticized the approach due to not using a proper "coding model") and got no better results. Now I tried about a dozen iterations but it did not manage to create a "BASIC programm for Atari 800XL that makes use of display list interrupts to draw rainbow-like colored horizontal stripes", although this is like a "hello world" for that technique and there should be plenty of samples on the Internet. I am curious to see if anyone can make that work with an LLM.
- JKCalhoun 1y ago
  Are there really plenty of examples on the internet? My first thought reading the article was that Atari BASIC is kind of a little specialized. If BASIC is kind of an under-represented language in general on the internet (you know, compared to Javascript, for example) then Atari BASIC has to be a white whale.
  - ofrzeta 1y ago
    Yeah, when you google for "atari display list interrupts" a lot of similar code comes up. Here's a complete example https://makezine.com/article/technology/generate-the-atari-rainbow-eff/ https://makezine.com/article/technology/generate-the-atari-r... Also, sure, compared to Javascript it's a small amount.
j45 1y ago
It can only code to the average level of content that it's trained on.
bethekidyouwant 1y ago
It literally worked fine but then he called it a bust for no reason. Also free version of chatgpt is a bold choice
benbristow 1y ago
Annoying website. Cookie policy banner at the start with the usual 'essential' or 'all', then get about a fifth down the article and the page fades out making the article unreadable with a box asking me to subscribe.
rzz3 1y ago
> With the rise of LLM systems (or “AI” as they are annoyingly called), I made it this far and realized the rest wasn’t worth reading. Language evolves, words change, and AI means what it means now. It turns out it’s actually really useful to have an abstraction above the concept of LLMs to talk about the broader set of these types of technologies, and generally speaking I find that these very pedantic types of people don’t bring me useful new perspectives.
- andriamanitra 1y ago
  > Language evolves, words change, and AI means what it means now. It turns out it’s actually really useful to have an abstraction above the concept of LLMs to talk about the broader set of these types of technologies I agree that "AI" can be useful as an umbrella term, but using it when referring specifically to the "LLM" subset of AI technologies is not useful. A ton of information about the capabilities and limitations of the system is lost when making that substitution. I understand why marketing departments are pushing everything as "AI" to sell a product but as consumers we should be fighting against that.
  - rzz3 1y ago
    People who aren’t software engineers need to be able to converse about these technologies, and no one outside of our world cares about the difference between LLMs and other types of AI.
    - andriamanitra 1y ago
      [dead]

160 points by ibobev 1y ago

It never did draw a blue circle, it's orange or similar but that's never mentioned in the article.

Maybe the Basic code out there has a too high similarity with spaghetti making it hard to abstract in latent space. OO and functional patterns are more localized.

firesteelrain 1y ago

Not surprised; there were so many variations of BASIC and unless you train ChatGPT on a bunch of code examples and contexts then it can only get so close. Try a local LLM then train it

ofrzeta 1y ago
> ... unless you train ChatGPT on a bunch of code examples and contexts then it can only get so close. How do you do this?
- sixothree 1y ago
  RAG maybe?
  - firesteelrain 1y ago
    RAG is good suggestion to pull in runtime without weights
- oharapj 1y ago
  If you're OpenAI you scrape StackOverflow and GitHub and spend billions of dollars on training. If you're a user, you don't
- firesteelrain 1y ago
  Gist is 1. Gather training data 2. Format it into JSONL or Hugging Face Dataset format 3. Use Axolotl or Hugging Face peft to fine-tune 4. Export model to GGUF or HF format 5. Serve via Ollama https://adithyask.medium.com/axolotl-is-all-you-need-331d5de1b7c0 https://adithyask.medium.com/axolotl-is-all-you-need-331d5de... https://www.philschmid.de/fine-tune-llms-in-2025 https://www.philschmid.de/fine-tune-llms-in-2025 https://blog.devgenius.io/complete-guide-to-model-fine-tuning-from-hugging-face-to-axolotl-dc5cdaf1ae27 https://blog.devgenius.io/complete-guide-to-model-fine-tunin...
  - ofrzeta 1y ago
    So, finetuning? Not so easy with ChatGPT I guess, but thanks for the info anyway.
    - firesteelrain 1y ago
      Yes, it takes an existing model and fine tunes it. ChatGPT would be basically extensively prompt engineering in a session. Maybe using their API? I have never tried it personally. When I fine tuned a Mistral 7B model it took hundreds of examples in Alpaca style It’s a lot of work. Maybe OpenAI has a more efficient way of doing it because in my case I had to manually adjust each prompt
- Paradigma11 1y ago
  Gemini Pro 2.5 has a context window of 1 million tokens and wants to rise that to 2 million tokens soon. 1 token is approx 0.75 words, so 1 million tokens would be in the ballpark of 3k pages of code. You can add some tutorials/language docs as context without any problem. The bigger your project gets the more context it gets from there. You can also convert apis/documentation to a RAG and expose it as a MCP tool to the LLM.
  - ofrzeta 1y ago
    > Gemini Pro 2.5 has a context window of 1 million tokens and wants to rise that to 2 million tokens soon. 1 token is approx 0.75 words, so 1 million tokens would be in the ballpark of 3k pages of code. You mean around 3000 files with 3000 characters? That is a lot. I've played with some other LLMs in Agentic AIs but at work we are using Copilot, and when I add context through drag and drop it seems to be limited to some dozen files.
    - Paradigma11 1y ago
      I think Copilot has some hardcoded limitations of around a dozen files, like you said. But this stuff changes constantly.
      - ofrzeta 1y ago
        Still I don't totally understand how that huge of a context works for Gemini. I guess you don't provide the whole context for every request? So it keeps (but also updates) context for a specific session?
        Paradigma11 1y ago
        I dont know how the massive context works but Caching is certainly a thing and cheaper: https://ai.google.dev/gemini-api/docs/caching?lang=python https://ai.google.dev/gemini-api/docs/caching?lang=python Gemini is better than Sonnet if you have broad questions that concern a large codebase, the context size seems to help there. People also use subagents for specific purposes to keep each context size manageable, if possible. On a related note I think the agent metaphor is a bit harmful because it suggests state while the LLM is stateless.

empressplay 1y ago

We have Claude Code writing Applesoft BASIC fine. It wrote a text adventure (complete with puzzles) and a PONG clone, among other things. Obviously it didn't do it 100% right straight out of the gate, but the hand-holding wasn't extensive. I've been using Grok 4 to write 6502 assembly language and it's been a bit of a slog but honestly the issues I've encountered are due mostly my to naivety. If I'm disciplined and make sure it has all of the relevant information and I'm (very) incremental, I've had some success writing game logic. You can't just tell it to build an entire game in a prompt, but if you're gradual about it you can go places with it. Like any tool, if you understand its idiosyncrasies you can cater for them, and be productive with it. If you're not then yeah, it's not going to go well.

hammyhavoc 1y ago
Ah yes, truly impressive, Pong. A game that countless textbooks et al have recreated numerous times. There's a mountain of training data for something so unoriginal.
deleted 1y ago
[deleted]

recipe19 1y ago

I work on niche platforms where the amount of example code on Github is minimal, and this definitely aligns with my observations. The error rate is way too high to make "vibe coding" possible. I think it's a good reality check for the claims of impending AGI. The models still depend heavily on being able to transform other people's work.

empressplay 1y ago
I don't know if you're working with modern models. Grok 4 doesn't really know much about assembly language on the Apple II but I gave it all of the architectural information it needed in the first prompt of a conversation and it built compilable and executable code. Most of the issues I encountered were due to me asking for too much in a prompt. But it built a complete, albeit simple, assembly language game in a few hours of back and forth with it. Obviously I know enough about the Apple II to steer it when it goes awry, but it's definitely able to write 'original' code in a language / platform it doesn't inherently comprehend.
- timschmidt 1y ago
  This matches my experience as well. Poor performance usually means I haven't provided enough context or have asked for too much in a single prompt. Modifying the prompt accordingly and iterating usually results in satisfactory output within the next few tries.
gompertz 1y ago
Yep I program in some niche languages like Pike, Snobol4, Unicon. Vibe coding is out of the question for these languages. Forced to use my brain!
- johnisgood 1y ago
  You could always feed it some documentation and example programs. I did it with a niche language and it worked out really well, with Claude. Around 8 months ago.
1y ago

docandrew 1y ago

Maybe other folks’ vibe coding experiences are a lot richer than mine have been, but I read the article and reached the opposite conclusion of the author. I was actually pretty impressed that it did as well as it did in a largely forgotten language and outdated platform. Looks like a vibe coding win to me.

sixothree 1y ago
Here's an example of a recent experience. I have a web site that is sort of a cms. I wanted users to be able to add a list of external links to their items. When a user adds a link to an entry, the web site should go out and fetch a cached copy of the site. If there are errors, it should retry a few times. It should also capture an mhtml single file as well as a full page screenshot. The user should be able to refresh the cache, and the site should keep all past versions. The cached copy should be viewable in a modal. The task also involves creating database entities, DTOs, CQRS handlers, etc. I asked Claude to implement the feature, went and took a shower, and when I came out it was done.
- hammyhavoc 1y ago
  Let us know how the security audit by human beings on the output goes.
  - catmanjan 1y ago
    The auditors are using llms too!
  - sixothree 1y ago
    It's really just a personal project for myself. Why else would I add that feature without any guardrails?
- nico 1y ago
  Im pretty new to CC, been using it in a very interactive way. What settings are you using to get it to just do all of that without your feedback or approval? Are you also running it inside a container, or setting some sort of command restrictions, or just yoloing it on a regular shell?

serf 1y ago

please just include the prompts rather than saying "So I said X.." There is a lot of nuance in how X is said.

pavelstoev 1y ago

I vibe coded a site about vibe 2 code projects. https://builtwithvibe.com/ https://builtwithvibe.com/

esafak 1y ago
The "Yo dawg, I heard..." memes are writing themselves today.

clambaker117 1y ago

Wouldn’t it have been better to use Claude 4?

sixothree 1y ago
I'm thinking Gemini CLI because of the context. He could add some information about the programming language itself in the project. I think that would help immensely.
- 4b11b4 1y ago
  Even though the max token limit is higher, it's more complicated than that. As the context length increases, undesirable things happen.

calvinmorrison 1y ago

This is great. I am currently vibecoding a replacement connector for some old EDI software that is written in a business basic kind of language called ProvideX, and a fork of that has undocumented behaviour. It uses some built inet ftp tooling thats terrible and barely works, even internally anymore. We are replacing it with a winscp implementation since winscp can talk over a COM object. unsuprisingly the COM object in basic works great - the problem is that I have no idea what I am doing. I spent hours doing something like WINSCP_SESSION'OPEN(WINSCP_SESSION_OPTIONS) when i needed WINSCP_SESSION'OPEN(*WINSCP_SESSION_OPTIONS) It was obvious after because it was a pointer type of setup, but i didnt find it until pages and pages deep into old PDF manuals. However the vibecode of all the agents did not understand the syntax of the system, it did help me analyse the old code, format it, and at least throw some stuff at the wall. I finished it up friday, hopefully i deploy monday.

deleted 1y ago

[deleted]

CMay 1y ago

This does kind of make me wonder. It's believable that we might either see an increase in the number of new programming languages since making new languages is becoming more accessible, or we could see fewer new languages as the problems of the existing ones are worked around more reliably with LLMs. Yet, what happens to adoption? Perhaps getting people to adopt new languages will be harder as generations come to expect LLM support. Would you almost need to use LLMs to synthesize tons of code examples that convert into the new language to prime the inputs? Once conversational intelligence machines reach a sort of godlike generality, then maybe they could very quickly adapt languages from much fewer examples. That still might not help much with the gotchas of any tooling or other quirks. So maybe we'll all snap to a new LLM super-language in 20 years, or we could be concreting ourselves into the most popular languages of today for the next 50 years.

hammyhavoc 1y ago
Fantasy.
- CMay 1y ago
  Can you elaborate?
  - hammyhavoc 1y ago
    > Once conversational intelligence machines reach a sort of godlike generality, Sums up fantastical inevitablism. Why would they? How could they? The data is telling us that they won't. Anybody who believes otherwise is ignoring the science. The assumed path to "AGI" was more tokens, but more tokens actually means worse output, and if LLMs aren't a total technological dead-end, period, then the data is supporting smaller models meant for more specific things, ergo, LLMs are not giving AGI any time, ever. Pure fantasy. Can't even call it sci-fi when it ignores the science entirely.
    - CMay 1y ago
      Ah, so you misread. I said conversational intelligence machines, but you focused on LLMs. I don't know where LLMs will lead, but I haven't ruled out the possibility of improvements continuing to surprise us. If anything is unscientific, it would be overconfidence in one way or the other.

ilaksh 1y ago

I think it's a fair article. However I will just mention a few things. When you make an article like this please take note of the particular language model used and acknowledge that they aren't all the same. Also realize that the context window is pretty large and you can help it by giving it information from manuals etc. so you don't need to rely on the intrinsic knowledge entirely. If they used o3 or o3 Pro and gave it a few sections of the manual it might have gotten farther. Also if someone finds a way to connect an agent to a retro computer, like an Atari BASIC MCP that can enter text and take screenshots, "vibe coding" can work better as an agent that can see errors and self-correct.

manca 1y ago

I literally had the same experience when I asked the top code LLMs (Claude Code, GPT-4o) to rewrite the code from Erlang/Elixir codebase to Java. It got some things right, but most things wrong and it required a lot of debugging to figure out what went wrong. It's the absolute proof that they are still dumb prediction machines, fully relying on the type of content they've been trained on. They can't generalize (yet) and if you want to use them for novel things, they'll fail miserably.

hammyhavoc 1y ago
They'll never be fit for purpose. They're a technological dead-end for anything like what people are usually throwing them at, IMO.
- zer00eyz 1y ago
  I will give you an example of where you are dead wrong, and one where the article is spot on (without diving into historic artifacts). I run HomeAssistant, I don't get to play/use it every day. Here, LLM's excel at filling in the (legion) of blanks in both the manual and end user devices. There is a large body of work for it to summarize and work against. I also play with SBC's. Many of these are "fringe" at best. LLM's are as you say "not fit for purpose". What kind of development you are using LLM's for will determine your experience with them. The tool may or may not live up to the hype depending how "common", well documented and "frequent" your issue is. Once you start hitting these "walls" you realize that no, real reason, leaps of inference and intelligence are still far away.
  - SecuredMarvin 1y ago
    I also made this experience. As long as the public level of knowledge is high, LLMs are massively helpful. Otherwise not so much and still hallucinating. It does not matter if you think highly of this public knowledge. QFT, QED and Gravity are fine, AD emulation on SAMBA, or Atari Basic not so much. If I would program Atari Basic, after finishing my Atari Emulator on my C64, I would learn the environment and test my assumptions. Single shot LLMs questions won't do it. A strong agent loop could probably. I believe that LLMs are yanking the needle to 80%. This level is easy achievable for professionals of the trade and this level is beyond the ability of beginners. LLMs are really powerful tools here. But if you are trying for 90% LLMs are always trying to keep you down. And if you are trying for 100%, new, fringe or exotic LLMs are a disaster because they do not learn and do not understand, even while being inside the token window. We learn that knowledge, (power) and language proficiency are an indicator for crystalline but not fluid intelligence

ofrzeta 1y ago

It didn't go well? I think it went quite well. It even produced an almost working drawing program.

abrookewood 1y ago
Yep, thought the same thing. I guess people have very different expectations.

Radle 1y ago

I had way better results. I'd assume the same would have happened to the author if he provided the LLM with a full documentation on what ATARI BASIC is and some example programs. Especially when asking the LLM to create a drawing program and a game the author would have probably received working code if he supplied the ai with documentation to the graphics function and sprite rendering using ATARI BASIC.

xiphias2 1y ago

4o is not even a coding model and very far from the best coding models OpenAI has, I seriously don't understand why these articles are upvoted so much

throw101010 1y ago
> I seriously don't understand why these articles are upvoted so much It confirm a bias for some, it triggers others who might have the opposite position (and maybe have a bias too on the other end). Perfect combo for successful social media posts... literally all about "attention" from start to finish.
deleted 1y ago
[deleted]
timeon 1y ago
Short while ago it was like "you are still using <X>? Why not the new 4o?!"
layer8 1y ago
A coding model doesn't seem to produce better results: https://news.ycombinator.com/item?id=44624222 https://news.ycombinator.com/item?id=44624222

fcatalan 1y ago

I had more luck with a little experiment a few days ago: I took phone pics of one of the shorter BASIC listings from Tim Hartnell's "Giant Book of Computer Games" (I learned to program out of those back in the early 80s, so I treasure my copy) and asked Gemini to translate it to plain C. It compiled and played just fine on the first go.

edent 1y ago

Vibe Coding seems to work best when you are already an experienced programmer. For example "Prompt: Write me an Atari BASIC program that draws a blue circle in graphics mode 7." You need to know that there are various graphics modes and that mode 7 is the best for your use-case. Without that preexisting knowledge, you get stuck very quickly.

throwawaylaptop 1y ago
Exactly this. I'm a self taught PHP/jQuery guy that learned it well enough to make an entire saas that enough companies pay for that it's a decent little lifestyle business. I started another project recently basically vibe coding in PHP. Instead of a single page app like I made before, it's just page by page single loading. Which means the AI also only needs to keep a few functions and the database in its head, not constantly work on some crazy ui management framework (what that's called). It's made in a few days what would have taken me weeks as an amateur. Yet I know enough to catch a few 'mistakes' and remind it to do it better. I'm happy enough.
kqr 1y ago
Not only is it a useful constraint to ask for mode 7, but making sure the context contains domain-expert technology puts the LLM in a better spot in the sampling space.
cfn 1y ago
Just for fun I asked ChatGPT "How would you ask an LLM to write a drawing program for the ATARI?" and it asked back a bunch of details to which I answered "I have no idea, just go with the simplest option". It chose the correct graphics mode and BASIC and created the program (which I didn't test). I still agree with you for large applications but for these simple examples anyone with a basic understanding of vibe coding could wing it.
baxtr 1y ago
This is a description of a “tool”. Anyone can use a hammer and chisel to carve out wood, but only an artist with extensive experience will create something truly remarkable. I believe many in this debate are confusing tools with magic wands.

Earw0rm 1y ago

Has anyone tried it on x87 assembly language? For those that don't know. x87 was the FPU for 32-bit x86 architectures. It's not terribly complicated, but it uses stack-based register addressing with a fixed size (eight entry) stack. All operations work on the top-of-stack register and one other register operand, and push the result onto the top of the stack (optionally popping the previous top of stack before the push). It's hard but not horribly so for humans to write.. more a case of annoyingly slow and having to be methodical, because you have to reason about the state of the stack at every step. I'd be very curious as to whether a token-prediction machine can get anywhere with this kind of task, as it requires a strong mental model of what's actually happening, or at least the ability to consistently simulate one as intermediate tokens/words.

silisili 1y ago
I'm going to doubt that. I was pushing GPT a couple weeks ago to test its limits. It's 100% unable to write compilable Go ASM syntax. In fairness it's slightly oddball, but enough exists that it's not esoteric. In the error feedback cycle, it kept blaming Go, not itself. A bit eye opening.
- messe 1y ago
  I'm comfortable writing asm for quite a few architectures, but Go's assembler... When I struggle to write Go ASM, I also blame Go and not myself.
- Earw0rm 1y ago
  The thing with x87 is that it's easy to write compilable, correct-looking code, and much harder to write correct compilable code, even for trivial sequences of a dozen or so operations. Whereas in most asm dialects, register AX is always register AX (word length aliasing aside), that's not the case for x87: the object/value at ST3 in one operation may be ST1 or ST5 in a couple of instructions' time.
userbinator 1y ago
If you are familiar with HP's calculators, x87 Asm isn't that difficult. Also noteworthy is that its density makes it a common choice for tiny demoscene productions.

nine_k 1y ago

All these stories about vibe coding going well or wrong remind me of an old joke. A man visits his friend's house. There is a dog in the house. The friend says that the dog can play poker. The man is incredulous, but they sit at a table and have a game of poker; the dog actually can play! The man says: "Wow! Your dog is incredibly, fantastically smart!" The friend answers: "Oh, well, no, he's a naïve fool. Every time he gets a good hand, he starts wagging his tail." Whether you see LLMs impressively smart or annoyingly foolish depends on your expectations. Currently they are very smart talking dogs.

kqr 1y ago
Somehow this also reminds me of http://raisingtalentthebook.com/wp-content/uploads/2014/04/taughtmydog.jpg http://raisingtalentthebook.com/wp-content/uploads/2014/04/t... "I taught my dog to whistle!" "Really? I don't hear him whistling." "I said I taught him, not that he learnt it."
- pyman 1y ago
  A person is flying a hot air balloon and realises he’s lost. He lowers the balloon and spots a man down below. He shouts: “Excuse me! Can you help me? I promised a friend I’d meet him, but I have no idea where I am.” The man replies, “You’re in a hot air balloon, hovering 30 feet above the ground, somewhere between 40 and 41 degrees north latitude and between 59 and 60 degrees west longitude.” “You must be a Prompt Engineer,” says the balloonist. “I am,” replies the man. “How did you know?” “Well,” says the balloonist, “everything you told me is technically correct, but it’s of no use to me and I still have no idea where I am.” The man below replies, “You must be a Vibe Coder.” “I am,” says the balloonist. “How did you know?” "Because you don’t know where you are or where you’re going. You made a promise you can’t keep, and now you expect me to solve your problem. The fact is, you’re in the same position you were in before we met, but now it’s somehow my fault!"
  - johnisgood 1y ago
    This is really good, saving it. :D

Yokolos 1y ago

When I start getting nonsense back and forth prompting, I've found it best to just start a new chat/context with the latest working version and then try again with a slightly more detailed prompt that tries to avoid the issues encountered in the previous chat. It usually helps. AI generally quickly gets itself lost, which can be annoying.

danjc 1y ago

What's missing here is tool use. For example, if the llm had a compile tool it would likely have been able to correct syntax errors. Similarly, visual errors may also have been caught if it were able to run the program and capture screens.

wbolt 1y ago
Exactly! The way in which the LLM is used here is very, very basic and outdated. This experiment should be redone in a proper „agentic” setup where there is a feedback loop between the model and the runtime plus access to documentation / internet. The goal now is not to encapsulate all the knowledge inside single LLM - this is too problematic and costly. LLM is a language model not knowledge database. It allows to interpret and interact with knowledge and text data from multiple sources.

486sx33 1y ago

[dead]

Mikhail_Edoshin 1y ago

I don't know if anyone notices that, but LLMs are very much like what you see in dreams when you reflect on them when awake. When you asleep a dream feels very coherent; but when you're awake, you see wild gaps and jumps and the overall impression of many dreams' plot is that it is pure nonsense. I once heard advice on trying to re-read the text you see in dreams. And I did that once. It was a phrase where one of words referred to a city. The first time I read that city was "London". I remembered the advice and re-read the phrase and the word changed to the name of an old Russian city "Rostov". Yet the phrase was "same", that is it felt same in the dream, even though the city was different. LLMs are like that dream mechanics. It is how something else is reflected in what we know (e.g. an image of a city is rendered as a name of a city, just any city). So, on one hand, we do have a similar mechanism in our minds. But on another hand our normal reasoning is very much unlike that. It would be a very wild stretch to believe that reasoning somehow stems from dreaming. I'd say reasoning is opposite to dreaming. If we amplify our dreaming mechanics we won't get a genius; more likely we'll get a schizophrenic.

neom 1y ago
I think that is a cool way of looking at it - build on what you said, to me it might be a bit like in both instances (dreaming and llms) - it's maybe a bit related to what they are trying to do (presuming dreams have a purpose) + the resource they have to do it in + the context they have to couple in to get the point across + something related to the abilities of the user? Lets for fun say there is a subsystem that understands and runs dreaming, you only have so much time, plus it's a weird modality, and you're trying to do something... maybe it's fine enough to just serve up the story no matter how muddled it is, what you might be talking about in LLMs is a similar thing? Dreams have a ticking clock where the brain chemistry is literally changing and the opportunity for that type of processing is about to disappear, and eventually the human will awaken, LLMs have a context window size. Fun thinking, anyway.
Fade_Dance 1y ago
Well it's neither schizophrenic nor using human reasoning. It's a machine using transformers no reason to overly anthropomorphize it, but sure the parallels do exist to some degree. These tools are obviously not reasoning tools in the classic sense, as they're not building off of core axiomic truths like classic logic. These truths may be largely embedded in the probabilistic output, due to the input universe that we feed it being based on human reason, but it's certainly not in a power of these tools. That said, of course we are attacking on more and more of this ability (ability to check databases of facts, iterative "reasoning" models, etc) so they are becoming "adequate" in these respects. The dreaming comparison seems quite apt though. I entirely get what you mean by rereading the dream word and seeing it suddenly transformed to another city name, yet somehow also "fitting". For some reason I'm keenly aware of these sorts of relationships when I think about my dreams. I will think of a situation and immediately be able to identify the input memories that "built" the dream scenario. Usually they involve overlapping themes and concepts, as well as some human specific common targets like "unresolved, emotionally charged, danger, etc" (presumably running through these types of brain neurons provides some sort of advantage for mammals, which makes sense to me). What an LLM does is essentially create a huge interconnected conceptual web of the universe which is fed, and then uses probably ballistic models to travel through these chains, much like how dreaming does a trance like dance through these conceptual connections. In the former example though, we have optimized the traversal to be as close to a "mock awake human" as possible. If the dream poem is dreary in nature, and Rostov sounds dreary, and you were hearing about the dreary London rain earlier in the day, and you have a childhood memory of reading through a dreary poem that made you very sad, that's the perfect sort of overlapping set of memory synapses for a dream to light up. And when looking back, all you'll see is a strange fantasmic and (usually, not always) frustratingly inaccessible conglomeration of these inputs. This sort of traversal isn't just used in dreaming though. To some degree we're doing similar things when we do things like creative thinking. The difference is, and this is especially so in "modern" life, we're strongly filtering our thoughts through language first and foremost (and indeed there's a lot of philosophical and scientific work done about how extremely important language is to humanness), but also through basic logic. LLM's inherent some of the power/magic of language, in that it is deconstructing the relationships between the concepts behind language that contains so much embedded meaning. But they aren't filtering through logic like we do. Well, reasoning models do to some degree, but it is obviously quite rudimentary. I think it's a good analogy.

I'll tell you a secret. The same happens also with mainstream languages

ghuntley 1y ago

It's not really 'vibe coding' if you're copying and pasting from ChatGPT by hand...

alfiedotwtf 1y ago

I’ve been using my AirPods to talk to ChatGPT while I drive into work.. not coding talk though, using that time to pick up particle physics. So far, nothing looks hallucinated, though I’m only touching the surface and haven’t looked at equations yet. .. either way, I’m super happy that it has kept my drives to work very interesting!

ofrzeta 1y ago

Even though I was impressed with the original article (so kind of contrary to the author) in the meantime I tried the same thing with Claude Sonnet 4 (because some people here criticized the approach due to not using a proper "coding model") and got no better results. Now I tried about a dozen iterations but it did not manage to create a "BASIC programm for Atari 800XL that makes use of display list interrupts to draw rainbow-like colored horizontal stripes", although this is like a "hello world" for that technique and there should be plenty of samples on the Internet. I am curious to see if anyone can make that work with an LLM.

JKCalhoun 1y ago
Are there really plenty of examples on the internet? My first thought reading the article was that Atari BASIC is kind of a little specialized. If BASIC is kind of an under-represented language in general on the internet (you know, compared to Javascript, for example) then Atari BASIC has to be a white whale.
- ofrzeta 1y ago
  Yeah, when you google for "atari display list interrupts" a lot of similar code comes up. Here's a complete example https://makezine.com/article/technology/generate-the-atari-rainbow-eff/ https://makezine.com/article/technology/generate-the-atari-r... Also, sure, compared to Javascript it's a small amount.

j45 1y ago

It can only code to the average level of content that it's trained on.

bethekidyouwant 1y ago

It literally worked fine but then he called it a bust for no reason. Also free version of chatgpt is a bold choice

benbristow 1y ago

Annoying website. Cookie policy banner at the start with the usual 'essential' or 'all', then get about a fifth down the article and the page fades out making the article unreadable with a box asking me to subscribe.

rzz3 1y ago

> With the rise of LLM systems (or “AI” as they are annoyingly called), I made it this far and realized the rest wasn’t worth reading. Language evolves, words change, and AI means what it means now. It turns out it’s actually really useful to have an abstraction above the concept of LLMs to talk about the broader set of these types of technologies, and generally speaking I find that these very pedantic types of people don’t bring me useful new perspectives.

andriamanitra 1y ago
> Language evolves, words change, and AI means what it means now. It turns out it’s actually really useful to have an abstraction above the concept of LLMs to talk about the broader set of these types of technologies I agree that "AI" can be useful as an umbrella term, but using it when referring specifically to the "LLM" subset of AI technologies is not useful. A ton of information about the capabilities and limitations of the system is lost when making that substitution. I understand why marketing departments are pushing everything as "AI" to sell a product but as consumers we should be fighting against that.
- rzz3 1y ago
  People who aren’t software engineers need to be able to converse about these technologies, and no one outside of our world cares about the difference between LLMs and other types of AI.
  - andriamanitra 1y ago
    [dead]