4 ms·

I had a fun day with replit yesterday. I signed up to try it out after someone at work mentioned it. I gave it a really simple challenge: a note taking app. Al

I had a fun day with replit yesterday. I signed up to try it out after someone at work mentioned it. I gave it a really simple challenge: a note taking app. Allow the user to write a sentence of plain text, and add it to an array with a time stamp. Show the text on screen. Save it in local storage. I could hand code this in about ten minutes. Replit went off and built a truly impressive UI that looked like something that would take me a couple of evenings to pull together. It was professional and polished, but it didn't work. Hitting enter to add the item did nothing. I chatted to it and explained. It confidently did a few revisions but didn't fix it. It started adding logging and asked me to paste in the logs. After each subsequent paste it would say some variation of "ah I see the problem" and revise again. After the third time, I didn't reply with the console logs and instead suggested we change tack. Replit completely ignored my input and carried on talking as if I was following instructions "yes I see the issue now, ok I've fixed it" It was just a string of hallucinations, pretending to take user input but it fact not doing so. In the end I started asking it completely off topic questions but it had an imaginary convo with me instead. I gave up. I've had much more success with non agentic LLMs like Gemini or Claude but they all are great at the initial response and slowly degrade add the conversation continues

stirfish 1y ago
>they all are great at the initial response and slowly degrade add the conversation continues I've noticed this too. I give the llm 3 or 4 prompts before a new session is opened and code is repasted. My work flow is a lot like 1. Here's a schema, what would you change? I assume nothing, because my schemas are perfect. 2. Given this schema, make me some structs 3. Given these structs, make me some endpoints 1.(New session) Given these endpoints that return data like this, make me a ui 1. while true: (New session), An llm wrote me this ui, what would you change? If I try to steamroll ahead from schema to ui, I find it starts having trouble with context, like hallucinating functions it meant to write but didn't, or just straight up dropping functions it wrote before.
- nopelynopington 1y ago
  It makes me think back to "attention is all you need". Most of the LLMs seem to have quite limited attention, at least while coding. I suspect replit probably had similar issues while talking to itself, before it even gave me the first response.

stirfish 1y ago

>they all are great at the initial response and slowly degrade add the conversation continues I've noticed this too. I give the llm 3 or 4 prompts before a new session is opened and code is repasted. My work flow is a lot like 1. Here's a schema, what would you change? I assume nothing, because my schemas are perfect. 2. Given this schema, make me some structs 3. Given these structs, make me some endpoints 1.(New session) Given these endpoints that return data like this, make me a ui 1. while true: (New session), An llm wrote me this ui, what would you change? If I try to steamroll ahead from schema to ui, I find it starts having trouble with context, like hallucinating functions it meant to write but didn't, or just straight up dropping functions it wrote before.

nopelynopington 1y ago
It makes me think back to "attention is all you need". Most of the LLMs seem to have quite limited attention, at least while coding. I suspect replit probably had similar issues while talking to itself, before it even gave me the first response.