4 ms·

AI deleted my most tests, and said "All Tests Pass"

27 points by autobe 5mo ago

deleted 5mo ago
[deleted]
benchwright 5mo ago
Tends to be a problem. I've tried to mitigate these problems by using either external harnesses (aka GitHub actions that are "fixed" based on known-good) or by using n-number of witness agents (e.g. Kimi/Qwen/whatever <=> Claude/OpenAI/Google). Generally sucks more time and energy (and now token/$). that being said, I still have a "fix the code, not the test" line somewhere in here...
cyanydeez 5mo ago
believing that oneshot prompt is enough specification is pretty delusional. I keep seeing people talk about the power of these SOTA models, yet keep reading the types of prompts that make no sense to anyone who understands the ludicrous number of decisions that would need to be made.
- brianwawok 5mo ago
  Yah it’s a weird thing to one prompt. I’m porting some code (Python to go in my case), and I’m pretty specific about doing it sessions by session. Port one file, hit a new wall, break into another session to fix it, and return to my original chat with the problem fixed.
- veunes 5mo ago
  [flagged]
OMGWTF 5mo ago
8 billion tokens? What did that cost?
- autobe 5mo ago
  No additional cost occured, but just weekly limit exhausted
patrakov 5mo ago
I confirm the same experience. I tried to port the Dynamic Range Optimization code in OpenCamera from Java to mathematical formulas (i.e., the spec), with the intention to translate that into Python with NumPy, so that I could run it on my own images not coming from my phone camera sensor. Tool used: just a chat with ChatGPT with the relevant files uploaded, research activated, and questions asked where I did not understand or doubted something in the response. Result: ChatGPT faithfully and correctly reverse-engineered the initial highlight pre-compression step and then said that the rest (the real thing!) is too complex and not important anyway. I did not pursue it further.
valunlabs 5mo ago
Then, when it’s time for the monthly payment, let’s write “Paid” in the chat and request the token. AI tools are great, but they require a lot of oversight. Developing a product is a process. People are generating so much code that they can’t keep up with code reviews. What they really need to do is spread the work out over several days. If your project is solid, you’ll get positive results anyway. (I’m not talking about the MVP stage here.)
smrtinsert 5mo ago
Did the author say which model and harness handled the first attempts? Codex at the end ok but what did he try the rest with?
- veunes 5mo ago
  [dead]
- subscribed 5mo ago
  They specifically refused to do it, which is very meh and in retrospect reads like Codex astroturfing because of that ("all AI bad except for Codex"?
veunes 5mo ago
[flagged]
GoToRO 5mo ago
Very human
tim-projects 5mo ago
Not had deleted tests but had a lot of "this isn't working I'll insert a skip". I gues a fix for that is a hard check for skips
butlike 5mo ago
There's multiple ways to skin a cat, human.

27 points by autobe 5mo ago

deleted 5mo ago

[deleted]

Tends to be a problem. I've tried to mitigate these problems by using either external harnesses (aka GitHub actions that are "fixed" based on known-good) or by using n-number of witness agents (e.g. Kimi/Qwen/whatever <=> Claude/OpenAI/Google). Generally sucks more time and energy (and now token/$). that being said, I still have a "fix the code, not the test" line somewhere in here...

cyanydeez 5mo ago

believing that oneshot prompt is enough specification is pretty delusional. I keep seeing people talk about the power of these SOTA models, yet keep reading the types of prompts that make no sense to anyone who understands the ludicrous number of decisions that would need to be made.

brianwawok 5mo ago
Yah it’s a weird thing to one prompt. I’m porting some code (Python to go in my case), and I’m pretty specific about doing it sessions by session. Port one file, hit a new wall, break into another session to fix it, and return to my original chat with the problem fixed.
veunes 5mo ago
[flagged]

OMGWTF 5mo ago

8 billion tokens? What did that cost?

autobe 5mo ago
No additional cost occured, but just weekly limit exhausted

patrakov 5mo ago

I confirm the same experience. I tried to port the Dynamic Range Optimization code in OpenCamera from Java to mathematical formulas (i.e., the spec), with the intention to translate that into Python with NumPy, so that I could run it on my own images not coming from my phone camera sensor. Tool used: just a chat with ChatGPT with the relevant files uploaded, research activated, and questions asked where I did not understand or doubted something in the response. Result: ChatGPT faithfully and correctly reverse-engineered the initial highlight pre-compression step and then said that the rest (the real thing!) is too complex and not important anyway. I did not pursue it further.

valunlabs 5mo ago

Then, when it’s time for the monthly payment, let’s write “Paid” in the chat and request the token. AI tools are great, but they require a lot of oversight. Developing a product is a process. People are generating so much code that they can’t keep up with code reviews. What they really need to do is spread the work out over several days. If your project is solid, you’ll get positive results anyway. (I’m not talking about the MVP stage here.)

smrtinsert 5mo ago

Did the author say which model and harness handled the first attempts? Codex at the end ok but what did he try the rest with?

veunes 5mo ago
[dead]
subscribed 5mo ago
They specifically refused to do it, which is very meh and in retrospect reads like Codex astroturfing because of that ("all AI bad except for Codex"?

veunes 5mo ago

[flagged]

GoToRO 5mo ago

Very human

tim-projects 5mo ago

Not had deleted tests but had a lot of "this isn't working I'll insert a skip". I gues a fix for that is a hard check for skips

butlike 5mo ago

There's multiple ways to skin a cat, human.