9 ms·

An AI Vibe Coding Horror Story

213 points by teichmann 6mo ago

dubeye 6mo ago
The person at the desk told the author this? Interesting how unquestioning the responses are that this isn’t engagement bait
sjamaan 6mo ago
So much is missing from this story. Did they report it to the relevant data authority? Did the fix they said they applied actually fix anything? Etc.
direwolf20 6mo ago
Some people only care about actual consequences. Download all the data and send it, in the post on a flash drive, to the GDPR regulator's office and another copy to the medical licensing board because why not.
- krater23 6mo ago
  Hopefully. And I hope he wasn't dumb enough to remove himself.
websap 6mo ago
Do you think if the agency hired a consultant to build this , a consultant couldn’t have made the same mistakes? Lack of security theater is a good thing for most businesses
- grey-area 6mo ago
  Usually they would just use an off the shelf product and extend it, so they wouldn’t produce the absolute horror story described in the article, no. I’m not even sure what your last comment means, are you contending that it is a good thing this company violated multiple laws with sensitive patient data?
  - trick-or-treat 6mo ago
    > Usually they would just use an off the shelf product and extend it AI does the same thing an agency or dev would do. Those vibe coding platforms have a template for these things which is usually Vite + React with Supabase for the backend, the same as a dev might use because surprise the LLM trained on the dev's work. OP's point is that you're not guaranteed a good outcome hiring an agency or solo dev either, in fact I would say you're almost guaranteed a bad outcome either way.
    - grey-area 6mo ago
      Apparently your assumptions about AI are completely wrong, if you read the article it produced terrible code.
      - trick-or-treat 6mo ago
        Right, because the user provided terrible prompts. A dev / agency wouldn't have done much better.
- ginko 6mo ago
  There's lack of security theater and there's: > All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one curl command away from anyone who looked. They are not the same thing.
  - chrisjj 6mo ago
    You've got to wonder from where did the "AI" parroted that.
    - chrisjj 6mo ago
      A Stackoverflow wrong answer?
- voidUpdate 6mo ago
  It is physically possible for a consultant to write bad code. But you'd hope that a consultant could understand that medical data is extremely important to keep secure, and actually write it to have some level of security
  - trick-or-treat 6mo ago
    Sure, but you'd hope that the LLM could understand that too.
    - voidUpdate 6mo ago
      And yet it seems it didn't
      - trick-or-treat 6mo ago
        I'll bet you anything it did. If you ask any recent coding model to do this project is will strongly advise you to give up.
- rightofcourse 6mo ago
  It is possible. If you select consulting that you know nothing about, and they know nothing about programming and vibe coded it for you... and maybe you dont even have a contract to held them responsible and maybe they dont really have a company either... Then I can imagine something like this.
- miningape 6mo ago
  If a consultant made the same mistakes I'd expect the consultant to be held accountable, not the client business that hired the consultancy - they knew they didn't have the requisite skills and so outsourced to an "expert" (and therefore can't be judged for not knowing how to secure their software since they did everything possible) In this case the "client" is fully liable for the security issues.
delis-thumbs-7e 6mo ago
Meanwhile on Linkedin… Every sales bozo with zero technical understanding is screaming top of their virtual lungs that evrything must be done with AI and it is solution to every layoff, economic problem, everything. It is just a matter of time when something really really bad happens.
- Gigachad 6mo ago
  [dead]
- funkyfourier 6mo ago
  The Hindenburg of coding.
  - m4rtink 6mo ago
    That worked for years & traveled for tens of thousands of kilometers until the disaster. They were also quite aware of the risks associated with a hydrogen airship and did all sorts of mitigations to avoid them. Compared to that vibe coding has no such qualities.
  - intended 6mo ago
    Nice. Another one I read on HN is “we are building hell on earth.”
- monkeydust 6mo ago
  Looks like bad stuff is happening, really bad is a bit scary if you qualify that as threat to life or livelihood. Let's see what the next generation of models bring to this equation.
  - animuchan 6mo ago
    The next generation of the user is what I think will be the factor for AI-assisted loss of life, moreso than the models themselves. It's the trust we put in them. The dinosaurs of the before-LLM age will increasingly struggle to convince young people e.g. to seek competent medical consultation, because ChatGPT can do that much better (it's always supportive, available at any hour, meets you where you are...) -- this aspect alone is bound to have a death toll associated with it. Or imagine asking ChatGPT to map an Alpine climbing route, -- anything with a bad failure mode.
andai 6mo ago
Archived version: https://archive.ph/GsLvt https://archive.ph/GsLvt https://web.archive.org/web/20260331184500/https://www.tobru.ch/an-ai-vibe-coding-horror-story/ https://web.archive.org/web/20260331184500/https://www.tobru...
- ramon156 6mo ago
  502, and the site is getting hugged to death it seems Edit: the archive.ph one works for me :)
mikojan 6mo ago
Hard to believe... This activity should certainly land you in a German prison?!
- VanTodi 6mo ago
  since its a .ch domain, i believe its in swiss. In germany we have our DSGVO (GDPR), and you can report it too. If a breach happen, you have to inform all your customers. if its a first time and you tried to your own best, the punishment is not that hard, but since these are medical infos they should have known better. Lets really hope they learned from their mistakes
  - piokoch 6mo ago
    Switzerland is very liberal in terms of business-oriented regulations to the point that you could crate a new year party in a closed cellar without emergency exists, not to mention anti-fire installation and burn people alive there.
spaniard89277 6mo ago
I did something similar to a local company here in Spain. Not medical, but a small insurance company. Believe it or not, yes, they vibecoded their CRM. I sent them an email and they threatened to sue me. I was a bit in shock from such dumb response, but I guess some people only learn the hard way, so I filed a report to the AEPD (Data protection agency in Spain) for starters, known to be brutal. I've also sent them a burofax demanding the removal of my data on their systems just last friday.
- petesergeant 6mo ago
  > [burofax is] a service that allows you to send a document with certified proof of delivery and confirmation of the date of receipt, and this confirmation has legal validity
- ramon156 6mo ago
  You only burn your hand once, unless you're a company, then you never learn.
  - dwaltrip 6mo ago
    Ah yes, companies don't have hands. Nothing to burn!
- sixtyj 6mo ago
  They should give you a chocolate at least. I think that having paper documentation will be safer very soon :)
- ramon156 6mo ago
  I'm also curious how much effort it would be to setup some OWASP tools with an agent and crawl for company tools. I'm sure I'm not the first one to think of this, but for local businesses it would give a solid rep, I suppose. I have a feeling that next year's theme will be security. People have turned off their brain when it comes to tech.
- fainpul 6mo ago
  > AEPD […] known to be brutal. Nice. I wish more countries had something like that. Many of these organizations are lethargic and have to be forced into action by civilian efforts or the press.
junofan 6mo ago
Kinda crazy but hopefully the future holds a Clippy-esque thing for people who don’t know to set up CI, checkpoints, reviews, environments, etc. that just takes care of all that. It sorta should do this anyway given that the user intent probably wasn’t to dump everyone’s data into Firebase or whatever. I personally would like this as well since it gets tiring specifying all the guardrails and double-checking myself. Using this stuff feels too much like developing a skill I shouldn’t need while not focusing on real user problems.
- grey-area 6mo ago
  This problem is unrelated to CI and dev practices etc, this is about trusting the output of generative AI without reading it, then using it to handle patient data. Vibe coding is just a bad idea, unless you’re willing and able to vet the output, which most people doing it are not.
  - dgb23 6mo ago
    Fully agentic development is neat for scripts and utilities that you wouldn‘t have the time to do otherwise, where you can treat it as intput/output and check both. In these cases you don’t necessarily care too much about the code itself, as long as it looks reasonable at a glance.
  - edwinjm 6mo ago
    It is related to CI and dev practices etc. A experienced developer using AI would add security/data protection, even when vibe coding.
    - grey-area 6mo ago
      An experienced developer would not have created this mess nor 'vibe-coded' (i.e. used AI without checking), but this person probably didn't know what they didn't know and believed the AI when it confidently asserted this mess was the correct way to do this. None of that is related to the practice of Continuous Integration.
  - 6mo ago
consumer451 6mo ago
What would a responsible on-boarding flow for all of these tools look like? > Welcome to VibeToolX. > By pressing Confirm you accept all responsibility for user data stewardship as regulated in every country where your users reside. Would that be scary enough to nudge some risk analysis on the user's part? I am sure that would drop adoption by a lot, so I don't see it happening voluntarily.
- jgrizou 6mo ago
  Would it? Feels a bit like when you use Facebook and handover all your data.
  - consumer451 6mo ago
    Yeah, fair. I am just thinking out loud here. What is a decent solution to this problem? Is there one?
- sigseg1v 6mo ago
  We require someone with a professional engineering designation from an accredited engineering body to sign off and approve before a building can be built. If it is found to have structural issues later, that person can be directly liable and can lose their license to operate. Why this is not the case with health software I cannot explain. Every time I propose this the only argument I recieve against it is people who are mad that their field might dare to apply the same regulation every other field has.
  - deleted 6mo ago
    [deleted]
  - consumer451 6mo ago
    Oh man, I have gone off on rants about software "engineering" here in the past. My first office job was as an AutoCAD/network admin at a large Civil and Structural engineering firm. I saw how seriously real engineering is taken. When I brought up your argument to my FAANG employed sibling, he said "well, what would it take to be a real software engineer in your mind!??" My response was, and always will be: "When there is a path to a software Professional Engineer stamp, with the engineer's name on it, which carries legal liability for gross negligence, then I will call them Software Engineers."
repeekad 6mo ago
A perfect example of why a product like Medplum exists, as opposed to completely reinventing the wheel from scratch
BrissyCoder 6mo ago
This reads like internet fiction to me. Very vague and short.
- spacebacon 6mo ago
  [flagged]
- abrookewood 6mo ago
  Given the subject matter, it would be highly unethical to reveal the name of the company before verifying it was indeed fixed. I'd be wary of getting sued.
  - croemer 6mo ago
    Company should just take down the whole thing. One vuln might be fixed but how many others might be there.
  - mijoharas 6mo ago
    The first time I stumbled onto a big security vulnerability (exposed stripe/aws/play store keys. I was poking around an API a web app was using, and instead of hitting /api/v1, if you just hit /api it served them. I wasn't trying to do anything malicious), the very first thing I did was contacted a security researcher friend to ask about covering my ass while performing responsible disclosure. You hear too much about people being persecuted for trying to point out security vulnerabilities. (Guess they haven't heard about "don't shoot the messenger"). (It turned out fine after finally managing to speak with someone. Had to ring up customer service and say "look, here are the last digits of your stripe private key. Please speak with an engineer". Figuring out how to talk with someone was the difficult thing)
- yawniek 6mo ago
  fwiw i know tobias and its very very unlikely he made this up. my guess its intentionally vague to not leak any information about the culprit which i guess is fair.
  - 6mo ago
agos 6mo ago
I really hope OP also contacted their relevant national privacy authority, this is a giant violation
zoobab 6mo ago
Avoid javascript like plague, it can be overwritten at the client side.
krater23 6mo ago
The only thing what helps is deleting the database. Every day. Until the thing goes down because the 'developer' thinks he has a bug that he can't find.
- rubzah 6mo ago
  https://www.myvibesite.com/?id=10; DROP TABLE customer;--
BitsAndObjects 6mo ago
Software engineering is looking more and more like it needs a professional body in each country, and accreditation and standards. Ie it needs to grow up and become like every other strand of engineering. Gone should be the days of “I taught myself so now I can [design software in a professional setting / design a bridge in a professional setting].” I’m not advocating gatekeeping - if you want to build a small bridge at the end of your garden for personal use, go for it. If you want to build a bridge in your local town over a river, you’re gonna need professional accreditation. Same should be true for software engineering now.
- cik 6mo ago
  Professional bodies act as nothing more then gatekeepers and rent seekers for things of this nature. Anyone can write software, but not everyone writes security minded software. We already have laws in place, and certifications that help someone understand if a given organization adheres to given standards. We can argue over their validity, efficacy, or value. The infrastructure, laws, and framework exist for this. More regulation and beaurocracy doesn't help when current state isn't enforced.
  - BitsAndObjects 6mo ago
    There’s a reason why many professions have professional bodies and consolidated standards - from medicine to accountancy, actuarial work, civil engineering, aerospace, electronic and electrical engineering, law, surveying, and so many more. In most of those professions, it is a crime or a civil violation to offer services without the proper qualifications, experience and accreditation from one of the appropriate professional bodies. We DO NOT have this in software engineering. At all. Anyone can teach themselves a bit of coding and start using it in their professional life. Analogous to law, you can draft a contract by yourself, but if it goes wrong you have a major headache. You cannot, however, offer services as a solicitor without proper qualifications and accreditation (at least in the UK). Yet in software engineering, not only can we teach ourselves and then write small bits of software for ourselves, we can then offer professional services with no further barriers or steps. The mishmash of laws we have around data and privacy are not professional standards, nor are they accreditation. We don’t have the framework or laws around this. And I am not aware of the USA (federal level) or Europe (or member states) or China or Russia or India or etc having this. For example, the BCS in the UK is so weak that although it exists, exceedingly few professional software engineers are even registered with them. They have no teeth. There’s no laws covering any of this stuff. Just good-ol’ GDPR and some sector-specific laws here and there trying to keep people mildly safe.
ionwake 6mo ago
Anyone else read the title on HN and shudder not wanting to actually click it?
- Hendrikto 6mo ago
  Every time I see “AI”, “LLM”, or “vibe conding” in the title. And then half the submissions not having it in the title are that anyways.
GistNoesis 6mo ago
Who should get jailed ? Does the company which willingly sells the polymorphic virus editor bear any responsibility, or should the unaware vibe coder be incumbent ?
- BitsAndObjects 6mo ago
  We don’t blame companies selling 3D Design software or 3D printers or mortar and cement, or graph paper and pencils. When people abuse those tools and build huts or houses or bridges that fall down, we usually blame the user for not having appropriate professional qualifications, accreditation, and experience. (Very occasionally we blame bugs in simulation software tools). AI is a tool. It’s not intelligent, and it works at a much bigger scale than bricks and mortar, but it’s still just a tool. There’s lots we can blame AI companies for, but abuse of the tool isn’t a clear-cut situation. We should blame them for misleading marketing. But we should also blame users (who are often highly intelligent - eg doctors) for using it outside their ability. Much like doctors are fed up of patients using AI to try to act like doctors, software engineers are now finding out what it’s like when clients try to use AI to act like software engineers.
  - Hendrikto 6mo ago
    I largely agree, but if a company sold cement explicitly claiming that they will replace every job in the entire construction industry, that the cement is able to plan, verify, and build on its own, without supervision, and that any layperson can now create PhD level bridges with that cement without any input from or verification by professionals, some liability would definitely fall on the company selling that cement under these pretenses.
    - BitsAndObjects 6mo ago
      > We should blame them for misleading marketing.
faangguyindia 6mo ago
It's nothing new, dunning kruger existing long before AI entered coding realm. Several years ago ran into one american company which consulted with me. They had 4000 paying customers and they rolled out their billing solution which accept crypto, paypal and stripe. They had problem with payment going missing, i migrated them to WHMCs with hardening and they never had any issues after. Now people may laugh at whmcs but use the right tool for job U need battle tested billing solution then whmcs does count it can support VAT, taxes, reporting/accounting and pretty all which you'll error while you try to do it all yourself. Too bad there aren't battle tested opensource solution for this
- t43562 6mo ago
  AI empowers bullshitters but for sure they existed before. The guys who do something quickly and are gone before it starts to fall over. It often works because everyone is impressed with them and the problems that arise are seen as the fault of whoever is left to clean up the mess. You can probably detect my bitterness :-D
jillesvangurp 6mo ago
I think the issue here is less about AI misbehaving and more about people doing things they should not be doing without thinking too hard about the consequences. There are going to be a lot of accidents like this because it's just really easy to do. And some people are inevitably going to do silly things. But it's not that different from people doing stupid things with Visual Basic back in the day. Or responding to friendly worded emails with the subject "I love you". Putting CDs/USB drives in work PCs with viruses, worms, etc. That's what people do when you give the useful tools with sharp edges.
- BitsAndObjects 6mo ago
  Particularly ironic for a doctor to have done this, given all the complaints about patients using Google (even pre-AI)!
- sersi 6mo ago
  I'd argue that back in the visual basic/Delphi day, there was a minimum level of competence needed AND, more importantly, apps didn't have as much surface area because they weren't exposed to internet
aledevv 6mo ago
> All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one command away from anyone who looked This is the top! This is a typical example of someone using Coding Agents without being a developer: AI that isn't used knowingly can be a huge risk if you don't know what you're doing. AI used for professional purposes (not experiments) should NOT be used haphazardly. And this also opens up a serious liability issue: the developer has the perception of being exempt from responsibility and this also leads to enormous risks for the business.
- anal_reactor 6mo ago
  The problem isn't AI, the problem is lack of an intelligent person somewhere in this whole situation. Way before AI I've seen a medical company create a service where frontend would tell backend what SQL queries to execute.
  - bootsmann 6mo ago
    “You’re just holding it wrong”
- dgb23 6mo ago
  Also it’s the wrong tool for this kind of work. Claude, opencode etc. Are brute force coding harnesses that literally use bash tools plus a whole bunch of vague prompting (skills, AGENT.md, MCP and all that stuff) to nudge them probabilistically into desirable behavior. Without engineering specialized harnesses that control workflows and validate output, this issue won‘t go away. We‘re in the wild west phase of LLM usage now, where problems emerge that shouldn’t exist in the first place and are being solved at the entirely wrong layer (outside of the harness) or with the entirely wrong tools (prompts).
rubzah 6mo ago
I know, through personal acquaintance, of at least one boutique accounting firm that is currently vibe-building their own CRM with Lovable. They have no technical staff. I can't begin to comprehend the disasters that are in store.
- antupis 6mo ago
  Generally why build your own CRM? ERP and other resource planning systems I get becouse you can tailor made those to your back office. But for CRM you need mostly reliability.
  - nslsm 6mo ago
    Because CRMs are very expensive, and they get much more expensive if you need custom development (which you usually need)
high_byte 6mo ago
this is exactly the kind of vibe coding horror stories I asked for just few days ago :) https://news.ycombinator.com/item?id=47707681 https://news.ycombinator.com/item?id=47707681
sajithdilshan 6mo ago
Don't blame the AI for what is clearly gross human negligence. It's like renovating your entire house and then acting surprised when the pipes burst because you used duct tape as a permanent fix.
- t43562 6mo ago
  At least part of the negligence is about the people who knowingly promote AI without also promoting knowledge of the limitations. Those who post stories about vibe-coding XXX in a week and don't bother to point out that they have no idea if it's not a piece of crap, waiting to explode, because there's no way they could have tested it properly in a week let alone read the mountains of code produced. There's a hype machine working and lots of people riding on it.
  - sajithdilshan 6mo ago
    That's what is meant by human negligence. There will always be a hype about something and that is not an excuse to have a devil may care attitude on any work being done
    - t43562 6mo ago
      Negligence depends on what you believe to be true. If you're being told "this is possible and the AI will do it properly you don't have to worry" then it's not negligence really - on the part of the person who believes what they are told. For the rest of us it is about being put under pressure by managers who don't understand whether to believe what you say or what they read about vibe coding on some linked-in post. As far as they are concerned you're not the authority and some hype-ster is.
      - sajithdilshan 6mo ago
        > "this is possible and the AI will do it properly you don't have to worry" then it's not negligence really Then that's lack of due diligence and and any manager is forcing you to ignore that, you should report them to compliance team. You cannot blame everyone else and bear no responsibility for your actions. If you decide to vibe code blindly and ignore all the laws and standards, then that was your decision and you decided to turn a blind eye.
freakynit 6mo ago
I think vibe-coding is cool, but it runs into limits pretty fast (at least right now). It kinda falls apart once you get past a few thousand lines of code... and real systems aren't just big, they're actually messy...shit loads of components, services, edge cases, things breaking in weird ways. Getting all of that to work together reliably is a different game altogether. And you still need solid software engineering fundamentals. Without understanding architecture, debugging, tradeoffs, and failure modes, it's hard to guide or even evaluate what's being generated. Vibe-coding feels great for prototypes, hobby projects, or just messing around, or even some internal tools in a handful of cases. But for actual production systems, you still need real engineering behind it. As of now, I'm 100% hesitant to pay for, or put my data on systems that are vibe-coded without the knowledge of what's been built and how it's been built.
- Shorel 6mo ago
  It absolutely falls apart more often than not. And requires even better engineering practices than before, because people are just accepting the code changes without understanding the technical debt created by them. On this I agree. There are models that can be run locally, this morning I tested Gemma 4 running on 128 GB of RAM. It was very slow, like 20 minutes to refactor something instead of 20 seconds, but it seems to be as capable as the paid models that run on an expensive cloud subscription on one of these hated data centers. And no data is uploaded to them.
- simianwords 6mo ago
  I suggest actually using Claude code and make a sample app using it. It absolutely can make apps even if you don’t know any fundamentals. I think it can work up to 20k LOC from my experience. You do need a human to give feedback but not someone who understands software principles.
  - Anamon 6mo ago
    That's a one-shot, though. Now try making something you want to build on. It falls apart very, very quickly. Not to mention that these prototypes and samples all turn out more or less the same. Obviously, given how LLMs work. Like LLM prose, LLM-generated web apps have a distinctive, samey look and feel.
mnls 6mo ago
Damn!!! And I keep hardening my RSS app which was partly vibe coded and not exposed to the WAN while "professionals" give data away.
aitchnyu 6mo ago
Is there anybody making some framework where you declare the security intentions as code (for each CRUD action) and which agents can correctly do and unit test? I have seen a Lovable competitor's system prompt have 24 lines of "please consider security when generating select statements, please consider security when generating update statements..." since it expects to dump queries here and there.
CrzyLngPwd 6mo ago
I think it is wonderful. It's reminiscent of the 90s, where every middle manager had dragged and dropped some boxes on some forms, and could get a salesman to sell it, without a care in the world for what was going on behind the scenes. Until something crashed and recovery was needed, of course. The piper always needs to be paid.
- Steve16384 6mo ago
  Or someone starts with an Excel spreadsheet just to "keep track of a few things". Then before they know it, it has become a critical part of the business but too monolithic and unorganised to be usable.
debarshri 6mo ago
I believe there are various dimensions to vibe coding. If you work with an existing codebase, it is a tool to increase productivity. If you have domain specific knowledge, in this case - patient management system, you can build better systems. Otherwise, you endup simulating the production. Lot of the non technical folks building products with AI Vibe coding are basically building Product Simulations. It looks like a product, functions like a product but behind the scene, you can poke holes.
vedant_awasthi 6mo ago
[flagged]
fakedang 6mo ago
Report them - that right there is 5+ different violations. Only then will they realize their stupidity.
seethishat 6mo ago
I saw something very similar a few months ago. It was a web app vibe coded by a surgeon. It worked, but they did not have an index .html file in the root web directory and they would routinely zip up all of the source code which contained all the database connection strings, API credentials, AWS credentials, etc.) and place the backup in the root web directory. They would also dump the database to that folder (for backup). So web browsers that went to https://example.com/ https://example.com/ could see and download all the backups. The quick fix was a simple, empty index.html file (or setting the -Indexes option in the apache config). The surgeon had no idea what this meant or why it was important. And the AI bots didn't either. The odd part of this to me was that the AI had made good choices (strong password hashes, reasonable DB schema, etc.) and the app itself worked well. Honestly, it was impressive. But at the same time, they made some very basic deployment/security mistakes that were trivial. They just needed a bit of guidance from an experienced devops security guy to make it Internet worthy, but no one bothered to do that. Edit: I do not recommend backing up web apps on the web server itself. That's another basic mistake. But they (or the AI) decided to do that and no one with experience was consulted.
- Arch-TK 6mo ago
  The fix is to not let users download the credentials. In fact, ideally the web server wouldn't have access to files containing credentials, it would handle serving and caching static content and offloading requests for dynamic content to the web application's code. Disabling auto-indexing just makes it harder to spot the issue. (To clarify, also not a bad idea in principle, just not _the_ solution.) If the file is still there and can be downloaded, that's strictly something which should not be possible in the first place.
- simianwords 6mo ago
  Agent-Native DevOps tools are probably necessary. There should be no reason they would do it manually. How I see it happening: agents like CC have in built skills for deployment and uses building blocks from either AWS or other simpler providers. Payment through OAuth and seamless checkout. This should be standardised
- shivaniShimpi_ 6mo ago
cmiles8 6mo ago
There’s another version of the Mythos narrative that reads like: AI companies realized that all this vibe coding has released a shitstorm of security vulnerabilities into the wild and so unless they release a much better model to fix that mess they’ll be found out and nobody will touch AI coding with a 100ft pole for the next 15 years. This article points more towards this narrative.
coopykins 6mo ago
I interviewed some years ago for an AI related startup. After looking at the live product, first thing I see is their prod dB credentials and openAI api key publicly send in some requests... Bad actors will be having a lot of fun these days
jseabra 6mo ago
[dead]
shivaniShimpi_ 6mo ago
Every other field that's figured out high stakes failure models eventually landed on the same solution - make sure two people that understand the details are looking at it - pilots have copilots surgeons with checklists and nuclear plants have independent verification. Software was always the exception, cause when it broke it mostly just broke for you, vibe coding is not going to change the equation, it barely removes one check that existed before is that the people who wrote the code understood what was going on, but now that's gone too
- Ekaros 6mo ago
  We do have code reviews for pull requests. But on average I would guess there is great amount of complacency there. I suppose old proper QA phase was best answer we had. But that is expensive and slow.
  - NoGravitas 6mo ago
    Maybe expensive and slow is actually an improvement.
    - shivaniShimpi_ 6mo ago
      i mean i don't disagree with you, but then there's at least unit test cases that we write as devs in a certain way that ai just doesn't understand quite well yet. we worked so much on speed that we forgot along the way that slow was a safety net. we tend to let our qa teams leverage ai as much as they want to automate test cases but i don't feel super comfortable getting rid of it just yet
  - deleted 6mo ago
    [deleted]
avazhi 6mo ago
You guys realise this is AI slop on AI slop, right?
- krapp 6mo ago
  This is reality now, what do you want?
  - avazhi 6mo ago
    HN should just have a rule that all content should be human-generated. This post is literally an LLM writing about something an LLM did; it's a bot botting about a bot. Aside from how funny and dystopian that is - just ban it? Just make a rule that submissions require a human author. I don't think solving this is all that complicated, at least for now. It isn't like it's currently difficult to tell what is and isn't LLM word salad, though that will likely change in the future, but by then the argument will involve whether it really matters or not. But for now, when 80% of the submissions are LLM garbage and it really is garbage, it's pretty jarring.
    - krapp 6mo ago
      Almost everything of interest to HN is written by LLM now, at least partially, and if it isn't it will be soon. There already is a rule against comments being LLM generated (which gets ignored all the time) but if we extended that to articles there would be almost nothing to post.
crvst 6mo ago
Cool story bro. Of course it’s true if it made it to HN. Who needs proofs.
zkmon 6mo ago
Technology for greed vs technology for need. Greed has its cost.
erelong 6mo ago
To me it just sounds like eventually someone will figure out how to make vibecoding more reasonably secure (with prompts to have apps be looked at for security practices?) unless cybersecurity is such a dynamic practice that we can't create automated processes that are secured Essentially a question of what can be done to make vibecoding "secure enough"
TeMPOraL 6mo ago
I have my doubts on the story. I consulted on a medtech project in the recent past in similar space, and at various points different individuals vibe-coded[0] not one but three distinct, independent prototypes of a system like the article describes, and neither of them was anywhere near that bad. On the frontend, you'd have to work pretty hard to force SOTA LLMs to give you what is being reported here. Backend-side, there's plenty of proper turn-key systems to get you started, including OSS servers you can just run locally, and even a year ago, SOTA LLMs knew about them and could find them (and would suggest some of them). I might be biased by my experience, because we actually cared about GDPR and AI act and proper medical data processing, and I've spent my fair share of time investigating the options that exist. Still, I'm struggling to imagine how one could possibly screw it up anywhere near as what the article described. Like, I can't think of a way to do it, to the point I might need to ask an LLM to explain it to me. -- [0] - Not as a means of developing an actual product, but solely to see if we can, plus it was easier to discuss product ideas while having some prototypes to click around.
hamasho 6mo ago
The worst blunder I made was when I explored cloud resources to improve the product's performance. I created a GCP project (my-app-dev) for exploring how to scale up the cloud service. I added several resources to mock the production, like compute instances/cloud SQL/etc, then populated the data and run several benchmarks. I changed the specs, number of instances and replicas, and configs through gcloud command. $ gcloud compute instances stop instance-1 --project=my-app-dev $ gcloud compute instances set-machine-type instance-1 --machine-type=c3-highcpu-176 --project=my-app-dev $ gcloud sql instances patch db-1 --tier=db-custom-32-131072 --project=my-app-dev But for some reason, at one point codex asked to list all projects; I couldn't understand the reason, but it seemed harmless so I approved the command. $ gcloud projects list PROJECT_ID NAME PROJECT_NUMBER my-app-test my app 123456789012 my-app-dev my app 234567890123 <- the dev project I was working on my-app my app 345678901234 <- the production (I know it's a bad name) And after this, for whatever reason it changed the target project from the dev (my-app-dev) to the production (my-app) without asking or me realizing. Of course I checked every commands. I couldn't YOLO while working on cloud resources, even in dev environment. But I focused on the subommands and its content and didn't even think it had changed the project ID along the way. It continued to suggest more and more aggressive commands for testing, and I approved them brain-deadly... $ gcloud sql instances patch db-1 --database-flags=max_connections=500 --project=my-app $ gcloud compute instances delete instance-1 --project=my-app $ echo 'DELETE FROM users WHERE username="test";' \ | gcloud sql connect my-db --user=user --database=my-db --project=my-app $ wrk -t4 -c200 -d30s \ "http://$(gcloud compute instances describe instance-1 \ --project=my-app \ --format='get(networkInterfaces[0].accessConfigs[0].natIP)')" It took a shamefully long time to realize codex was actually operating on production, so I DDoSed and SQL-injected to the production... Fortunately, it didn't do anything irreversible. But it was one of the most terrifying moments in my career.
- BitsAndObjects 6mo ago
  This is part of the reason deployments to production cloud environments should: 1. Only be allowed via CI/CD 2. All infra should be defined as code 3. Any deployment to production should be a delayed process that also has a human-approval step in the workflow (at least one, if not more) (Exactly where that review step is placed depends on your organisation - culture, size, etc.) And anyone that does need to touch production should do so from an isolated VM with temporary credentials. Developers shouldn't routinely have production access from their terminal. This last aspect is easy and cheap to set up on AWS. I presume it's also possible in Google Cloud.
keysersoze33 6mo ago
The takeaway is to vet new companies one is dealing with - even just calling them up and asking if they've AI generated any system which deals with customer/patient data. This is going to get more common (state sponsored hackers are going to have a field day)
0-bad-sectors 6mo ago
I think AI will be too expensive soon for normal/non technical people to tinker with and this kind of vibe coding stories will disappear.
Garlef 6mo ago
If this happened in Germany, this is most likely not only a breach of some contract but actually a criminal offense. (In not a lawyer and so I might be mistaken about this; Especially the level of intentionality might be a factor)
zzyzxd 6mo ago
Vibe coding is fun, but I can't trust it to make any serious decisions. Like, it knows what's the best way to do a thing, but when encounters challenges, it started to make all kinds of excuses to cut corners, just like humans. "but honestly, it's cluster internal traffic so unencrypted traffic is fine". "Given the urgency and tight timeline, your best option is bypassing the pipeline and deploying it manually". "Per my research, XXX also did this so you are fine". If I don't have disciplines or principles, or if I am just technically incompetent, its suggestions would sound so reasonable.
oxag3n 6mo ago
What if that AI was trained to hide implementation as much as possible? E.g. by making the client as thin as possible, using OAuth for authentication, following some robust template, would it be better? Avoiding such low-hanging-fruit disasters is doable and major players have incentives to fix it. This however wouldn't change the goal - replace all software engineers with DIY nephews with LLM in their hands. Even worse - people start thinking it's infeasible to learn hard things, because it prevents you from moving fast, from one domain to another. My area of expertise is cryptography-adjacent. Non-trivial fraction of students think that cryptography and security is a dead-end skill as all companies in this area are being replaced with AI. I asked them to implement web-bot-auth as simple as possible, because I know how AI can read specification and follow it.
MajorLettuce 5mo ago
"That sparked an idea: why use an industry-proven solution when you could just build your own patient management system?" lol I wonder if it is an actual dunning kruger effect at play here (confidence issue) or the lack of critical thinking. The first one could be excused somewhat, but I am afraid it is more of the second one
fredericgalline 5mo ago
[dead]

213 points by teichmann 6mo ago

The person at the desk told the author this? Interesting how unquestioning the responses are that this isn’t engagement bait

sjamaan 6mo ago

So much is missing from this story. Did they report it to the relevant data authority? Did the fix they said they applied actually fix anything? Etc.

direwolf20 6mo ago

Some people only care about actual consequences. Download all the data and send it, in the post on a flash drive, to the GDPR regulator's office and another copy to the medical licensing board because why not.

krater23 6mo ago
Hopefully. And I hope he wasn't dumb enough to remove himself.

websap 6mo ago

Do you think if the agency hired a consultant to build this , a consultant couldn’t have made the same mistakes? Lack of security theater is a good thing for most businesses

grey-area 6mo ago
Usually they would just use an off the shelf product and extend it, so they wouldn’t produce the absolute horror story described in the article, no. I’m not even sure what your last comment means, are you contending that it is a good thing this company violated multiple laws with sensitive patient data?
- trick-or-treat 6mo ago
  > Usually they would just use an off the shelf product and extend it AI does the same thing an agency or dev would do. Those vibe coding platforms have a template for these things which is usually Vite + React with Supabase for the backend, the same as a dev might use because surprise the LLM trained on the dev's work. OP's point is that you're not guaranteed a good outcome hiring an agency or solo dev either, in fact I would say you're almost guaranteed a bad outcome either way.
  - grey-area 6mo ago
    Apparently your assumptions about AI are completely wrong, if you read the article it produced terrible code.
    - trick-or-treat 6mo ago
      Right, because the user provided terrible prompts. A dev / agency wouldn't have done much better.
ginko 6mo ago
There's lack of security theater and there's: > All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one curl command away from anyone who looked. They are not the same thing.
- chrisjj 6mo ago
  You've got to wonder from where did the "AI" parroted that.
  - chrisjj 6mo ago
    A Stackoverflow wrong answer?
voidUpdate 6mo ago
It is physically possible for a consultant to write bad code. But you'd hope that a consultant could understand that medical data is extremely important to keep secure, and actually write it to have some level of security
- trick-or-treat 6mo ago
  Sure, but you'd hope that the LLM could understand that too.
  - voidUpdate 6mo ago
    And yet it seems it didn't
    - trick-or-treat 6mo ago
      I'll bet you anything it did. If you ask any recent coding model to do this project is will strongly advise you to give up.
rightofcourse 6mo ago
It is possible. If you select consulting that you know nothing about, and they know nothing about programming and vibe coded it for you... and maybe you dont even have a contract to held them responsible and maybe they dont really have a company either... Then I can imagine something like this.
miningape 6mo ago
If a consultant made the same mistakes I'd expect the consultant to be held accountable, not the client business that hired the consultancy - they knew they didn't have the requisite skills and so outsourced to an "expert" (and therefore can't be judged for not knowing how to secure their software since they did everything possible) In this case the "client" is fully liable for the security issues.

delis-thumbs-7e 6mo ago

Meanwhile on Linkedin… Every sales bozo with zero technical understanding is screaming top of their virtual lungs that evrything must be done with AI and it is solution to every layoff, economic problem, everything. It is just a matter of time when something really really bad happens.

Gigachad 6mo ago
[dead]
funkyfourier 6mo ago
The Hindenburg of coding.
- m4rtink 6mo ago
  That worked for years & traveled for tens of thousands of kilometers until the disaster. They were also quite aware of the risks associated with a hydrogen airship and did all sorts of mitigations to avoid them. Compared to that vibe coding has no such qualities.
- intended 6mo ago
  Nice. Another one I read on HN is “we are building hell on earth.”
monkeydust 6mo ago
Looks like bad stuff is happening, really bad is a bit scary if you qualify that as threat to life or livelihood. Let's see what the next generation of models bring to this equation.
- animuchan 6mo ago
  The next generation of the user is what I think will be the factor for AI-assisted loss of life, moreso than the models themselves. It's the trust we put in them. The dinosaurs of the before-LLM age will increasingly struggle to convince young people e.g. to seek competent medical consultation, because ChatGPT can do that much better (it's always supportive, available at any hour, meets you where you are...) -- this aspect alone is bound to have a death toll associated with it. Or imagine asking ChatGPT to map an Alpine climbing route, -- anything with a bad failure mode.

andai 6mo ago

Archived version: https://archive.ph/GsLvt https://archive.ph/GsLvt https://web.archive.org/web/20260331184500/https://www.tobru.ch/an-ai-vibe-coding-horror-story/ https://web.archive.org/web/20260331184500/https://www.tobru...

ramon156 6mo ago
502, and the site is getting hugged to death it seems Edit: the archive.ph one works for me :)

mikojan 6mo ago

Hard to believe... This activity should certainly land you in a German prison?!

VanTodi 6mo ago
since its a .ch domain, i believe its in swiss. In germany we have our DSGVO (GDPR), and you can report it too. If a breach happen, you have to inform all your customers. if its a first time and you tried to your own best, the punishment is not that hard, but since these are medical infos they should have known better. Lets really hope they learned from their mistakes
- piokoch 6mo ago
  Switzerland is very liberal in terms of business-oriented regulations to the point that you could crate a new year party in a closed cellar without emergency exists, not to mention anti-fire installation and burn people alive there.

spaniard89277 6mo ago

I did something similar to a local company here in Spain. Not medical, but a small insurance company. Believe it or not, yes, they vibecoded their CRM. I sent them an email and they threatened to sue me. I was a bit in shock from such dumb response, but I guess some people only learn the hard way, so I filed a report to the AEPD (Data protection agency in Spain) for starters, known to be brutal. I've also sent them a burofax demanding the removal of my data on their systems just last friday.

petesergeant 6mo ago
> [burofax is] a service that allows you to send a document with certified proof of delivery and confirmation of the date of receipt, and this confirmation has legal validity
ramon156 6mo ago
You only burn your hand once, unless you're a company, then you never learn.
- dwaltrip 6mo ago
  Ah yes, companies don't have hands. Nothing to burn!
sixtyj 6mo ago
They should give you a chocolate at least. I think that having paper documentation will be safer very soon :)
ramon156 6mo ago
I'm also curious how much effort it would be to setup some OWASP tools with an agent and crawl for company tools. I'm sure I'm not the first one to think of this, but for local businesses it would give a solid rep, I suppose. I have a feeling that next year's theme will be security. People have turned off their brain when it comes to tech.
fainpul 6mo ago
> AEPD […] known to be brutal. Nice. I wish more countries had something like that. Many of these organizations are lethargic and have to be forced into action by civilian efforts or the press.

junofan 6mo ago

Kinda crazy but hopefully the future holds a Clippy-esque thing for people who don’t know to set up CI, checkpoints, reviews, environments, etc. that just takes care of all that. It sorta should do this anyway given that the user intent probably wasn’t to dump everyone’s data into Firebase or whatever. I personally would like this as well since it gets tiring specifying all the guardrails and double-checking myself. Using this stuff feels too much like developing a skill I shouldn’t need while not focusing on real user problems.

grey-area 6mo ago
This problem is unrelated to CI and dev practices etc, this is about trusting the output of generative AI without reading it, then using it to handle patient data. Vibe coding is just a bad idea, unless you’re willing and able to vet the output, which most people doing it are not.
- dgb23 6mo ago
  Fully agentic development is neat for scripts and utilities that you wouldn‘t have the time to do otherwise, where you can treat it as intput/output and check both. In these cases you don’t necessarily care too much about the code itself, as long as it looks reasonable at a glance.
- edwinjm 6mo ago
  It is related to CI and dev practices etc. A experienced developer using AI would add security/data protection, even when vibe coding.
  - grey-area 6mo ago
    An experienced developer would not have created this mess nor 'vibe-coded' (i.e. used AI without checking), but this person probably didn't know what they didn't know and believed the AI when it confidently asserted this mess was the correct way to do this. None of that is related to the practice of Continuous Integration.
- 6mo ago

consumer451 6mo ago

What would a responsible on-boarding flow for all of these tools look like? > Welcome to VibeToolX. > By pressing Confirm you accept all responsibility for user data stewardship as regulated in every country where your users reside. Would that be scary enough to nudge some risk analysis on the user's part? I am sure that would drop adoption by a lot, so I don't see it happening voluntarily.

jgrizou 6mo ago
Would it? Feels a bit like when you use Facebook and handover all your data.
- consumer451 6mo ago
  Yeah, fair. I am just thinking out loud here. What is a decent solution to this problem? Is there one?
sigseg1v 6mo ago
We require someone with a professional engineering designation from an accredited engineering body to sign off and approve before a building can be built. If it is found to have structural issues later, that person can be directly liable and can lose their license to operate. Why this is not the case with health software I cannot explain. Every time I propose this the only argument I recieve against it is people who are mad that their field might dare to apply the same regulation every other field has.
- deleted 6mo ago
  [deleted]
- consumer451 6mo ago
  Oh man, I have gone off on rants about software "engineering" here in the past. My first office job was as an AutoCAD/network admin at a large Civil and Structural engineering firm. I saw how seriously real engineering is taken. When I brought up your argument to my FAANG employed sibling, he said "well, what would it take to be a real software engineer in your mind!??" My response was, and always will be: "When there is a path to a software Professional Engineer stamp, with the engineer's name on it, which carries legal liability for gross negligence, then I will call them Software Engineers."

repeekad 6mo ago

A perfect example of why a product like Medplum exists, as opposed to completely reinventing the wheel from scratch

BrissyCoder 6mo ago

This reads like internet fiction to me. Very vague and short.

spacebacon 6mo ago
[flagged]
abrookewood 6mo ago
Given the subject matter, it would be highly unethical to reveal the name of the company before verifying it was indeed fixed. I'd be wary of getting sued.
- croemer 6mo ago
  Company should just take down the whole thing. One vuln might be fixed but how many others might be there.
- mijoharas 6mo ago
  The first time I stumbled onto a big security vulnerability (exposed stripe/aws/play store keys. I was poking around an API a web app was using, and instead of hitting /api/v1, if you just hit /api it served them. I wasn't trying to do anything malicious), the very first thing I did was contacted a security researcher friend to ask about covering my ass while performing responsible disclosure. You hear too much about people being persecuted for trying to point out security vulnerabilities. (Guess they haven't heard about "don't shoot the messenger"). (It turned out fine after finally managing to speak with someone. Had to ring up customer service and say "look, here are the last digits of your stripe private key. Please speak with an engineer". Figuring out how to talk with someone was the difficult thing)
yawniek 6mo ago
fwiw i know tobias and its very very unlikely he made this up. my guess its intentionally vague to not leak any information about the culprit which i guess is fair.
- 6mo ago

agos 6mo ago

I really hope OP also contacted their relevant national privacy authority, this is a giant violation

zoobab 6mo ago

Avoid javascript like plague, it can be overwritten at the client side.

krater23 6mo ago

The only thing what helps is deleting the database. Every day. Until the thing goes down because the 'developer' thinks he has a bug that he can't find.

rubzah 6mo ago
https://www.myvibesite.com/?id=10; DROP TABLE customer;--

BitsAndObjects 6mo ago

Software engineering is looking more and more like it needs a professional body in each country, and accreditation and standards. Ie it needs to grow up and become like every other strand of engineering. Gone should be the days of “I taught myself so now I can [design software in a professional setting / design a bridge in a professional setting].” I’m not advocating gatekeeping - if you want to build a small bridge at the end of your garden for personal use, go for it. If you want to build a bridge in your local town over a river, you’re gonna need professional accreditation. Same should be true for software engineering now.

cik 6mo ago
Professional bodies act as nothing more then gatekeepers and rent seekers for things of this nature. Anyone can write software, but not everyone writes security minded software. We already have laws in place, and certifications that help someone understand if a given organization adheres to given standards. We can argue over their validity, efficacy, or value. The infrastructure, laws, and framework exist for this. More regulation and beaurocracy doesn't help when current state isn't enforced.
- BitsAndObjects 6mo ago
  There’s a reason why many professions have professional bodies and consolidated standards - from medicine to accountancy, actuarial work, civil engineering, aerospace, electronic and electrical engineering, law, surveying, and so many more. In most of those professions, it is a crime or a civil violation to offer services without the proper qualifications, experience and accreditation from one of the appropriate professional bodies. We DO NOT have this in software engineering. At all. Anyone can teach themselves a bit of coding and start using it in their professional life. Analogous to law, you can draft a contract by yourself, but if it goes wrong you have a major headache. You cannot, however, offer services as a solicitor without proper qualifications and accreditation (at least in the UK). Yet in software engineering, not only can we teach ourselves and then write small bits of software for ourselves, we can then offer professional services with no further barriers or steps. The mishmash of laws we have around data and privacy are not professional standards, nor are they accreditation. We don’t have the framework or laws around this. And I am not aware of the USA (federal level) or Europe (or member states) or China or Russia or India or etc having this. For example, the BCS in the UK is so weak that although it exists, exceedingly few professional software engineers are even registered with them. They have no teeth. There’s no laws covering any of this stuff. Just good-ol’ GDPR and some sector-specific laws here and there trying to keep people mildly safe.

ionwake 6mo ago

Anyone else read the title on HN and shudder not wanting to actually click it?

Hendrikto 6mo ago
Every time I see “AI”, “LLM”, or “vibe conding” in the title. And then half the submissions not having it in the title are that anyways.

GistNoesis 6mo ago

Who should get jailed ? Does the company which willingly sells the polymorphic virus editor bear any responsibility, or should the unaware vibe coder be incumbent ?

BitsAndObjects 6mo ago
We don’t blame companies selling 3D Design software or 3D printers or mortar and cement, or graph paper and pencils. When people abuse those tools and build huts or houses or bridges that fall down, we usually blame the user for not having appropriate professional qualifications, accreditation, and experience. (Very occasionally we blame bugs in simulation software tools). AI is a tool. It’s not intelligent, and it works at a much bigger scale than bricks and mortar, but it’s still just a tool. There’s lots we can blame AI companies for, but abuse of the tool isn’t a clear-cut situation. We should blame them for misleading marketing. But we should also blame users (who are often highly intelligent - eg doctors) for using it outside their ability. Much like doctors are fed up of patients using AI to try to act like doctors, software engineers are now finding out what it’s like when clients try to use AI to act like software engineers.
- Hendrikto 6mo ago
  I largely agree, but if a company sold cement explicitly claiming that they will replace every job in the entire construction industry, that the cement is able to plan, verify, and build on its own, without supervision, and that any layperson can now create PhD level bridges with that cement without any input from or verification by professionals, some liability would definitely fall on the company selling that cement under these pretenses.
  - BitsAndObjects 6mo ago
    > We should blame them for misleading marketing.

faangguyindia 6mo ago

It's nothing new, dunning kruger existing long before AI entered coding realm. Several years ago ran into one american company which consulted with me. They had 4000 paying customers and they rolled out their billing solution which accept crypto, paypal and stripe. They had problem with payment going missing, i migrated them to WHMCs with hardening and they never had any issues after. Now people may laugh at whmcs but use the right tool for job U need battle tested billing solution then whmcs does count it can support VAT, taxes, reporting/accounting and pretty all which you'll error while you try to do it all yourself. Too bad there aren't battle tested opensource solution for this

t43562 6mo ago
AI empowers bullshitters but for sure they existed before. The guys who do something quickly and are gone before it starts to fall over. It often works because everyone is impressed with them and the problems that arise are seen as the fault of whoever is left to clean up the mess. You can probably detect my bitterness :-D

jillesvangurp 6mo ago

I think the issue here is less about AI misbehaving and more about people doing things they should not be doing without thinking too hard about the consequences. There are going to be a lot of accidents like this because it's just really easy to do. And some people are inevitably going to do silly things. But it's not that different from people doing stupid things with Visual Basic back in the day. Or responding to friendly worded emails with the subject "I love you". Putting CDs/USB drives in work PCs with viruses, worms, etc. That's what people do when you give the useful tools with sharp edges.

BitsAndObjects 6mo ago
Particularly ironic for a doctor to have done this, given all the complaints about patients using Google (even pre-AI)!
sersi 6mo ago
I'd argue that back in the visual basic/Delphi day, there was a minimum level of competence needed AND, more importantly, apps didn't have as much surface area because they weren't exposed to internet

aledevv 6mo ago

> All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one command away from anyone who looked This is the top! This is a typical example of someone using Coding Agents without being a developer: AI that isn't used knowingly can be a huge risk if you don't know what you're doing. AI used for professional purposes (not experiments) should NOT be used haphazardly. And this also opens up a serious liability issue: the developer has the perception of being exempt from responsibility and this also leads to enormous risks for the business.

anal_reactor 6mo ago
The problem isn't AI, the problem is lack of an intelligent person somewhere in this whole situation. Way before AI I've seen a medical company create a service where frontend would tell backend what SQL queries to execute.
- bootsmann 6mo ago
  “You’re just holding it wrong”
dgb23 6mo ago
Also it’s the wrong tool for this kind of work. Claude, opencode etc. Are brute force coding harnesses that literally use bash tools plus a whole bunch of vague prompting (skills, AGENT.md, MCP and all that stuff) to nudge them probabilistically into desirable behavior. Without engineering specialized harnesses that control workflows and validate output, this issue won‘t go away. We‘re in the wild west phase of LLM usage now, where problems emerge that shouldn’t exist in the first place and are being solved at the entirely wrong layer (outside of the harness) or with the entirely wrong tools (prompts).

rubzah 6mo ago

I know, through personal acquaintance, of at least one boutique accounting firm that is currently vibe-building their own CRM with Lovable. They have no technical staff. I can't begin to comprehend the disasters that are in store.

antupis 6mo ago
Generally why build your own CRM? ERP and other resource planning systems I get becouse you can tailor made those to your back office. But for CRM you need mostly reliability.
- nslsm 6mo ago
  Because CRMs are very expensive, and they get much more expensive if you need custom development (which you usually need)

high_byte 6mo ago

this is exactly the kind of vibe coding horror stories I asked for just few days ago :) https://news.ycombinator.com/item?id=47707681 https://news.ycombinator.com/item?id=47707681

sajithdilshan 6mo ago

Don't blame the AI for what is clearly gross human negligence. It's like renovating your entire house and then acting surprised when the pipes burst because you used duct tape as a permanent fix.

t43562 6mo ago
At least part of the negligence is about the people who knowingly promote AI without also promoting knowledge of the limitations. Those who post stories about vibe-coding XXX in a week and don't bother to point out that they have no idea if it's not a piece of crap, waiting to explode, because there's no way they could have tested it properly in a week let alone read the mountains of code produced. There's a hype machine working and lots of people riding on it.
- sajithdilshan 6mo ago
  That's what is meant by human negligence. There will always be a hype about something and that is not an excuse to have a devil may care attitude on any work being done
  - t43562 6mo ago
    Negligence depends on what you believe to be true. If you're being told "this is possible and the AI will do it properly you don't have to worry" then it's not negligence really - on the part of the person who believes what they are told. For the rest of us it is about being put under pressure by managers who don't understand whether to believe what you say or what they read about vibe coding on some linked-in post. As far as they are concerned you're not the authority and some hype-ster is.
    - sajithdilshan 6mo ago
      > "this is possible and the AI will do it properly you don't have to worry" then it's not negligence really Then that's lack of due diligence and and any manager is forcing you to ignore that, you should report them to compliance team. You cannot blame everyone else and bear no responsibility for your actions. If you decide to vibe code blindly and ignore all the laws and standards, then that was your decision and you decided to turn a blind eye.

freakynit 6mo ago

I think vibe-coding is cool, but it runs into limits pretty fast (at least right now). It kinda falls apart once you get past a few thousand lines of code... and real systems aren't just big, they're actually messy...shit loads of components, services, edge cases, things breaking in weird ways. Getting all of that to work together reliably is a different game altogether. And you still need solid software engineering fundamentals. Without understanding architecture, debugging, tradeoffs, and failure modes, it's hard to guide or even evaluate what's being generated. Vibe-coding feels great for prototypes, hobby projects, or just messing around, or even some internal tools in a handful of cases. But for actual production systems, you still need real engineering behind it. As of now, I'm 100% hesitant to pay for, or put my data on systems that are vibe-coded without the knowledge of what's been built and how it's been built.

Shorel 6mo ago
It absolutely falls apart more often than not. And requires even better engineering practices than before, because people are just accepting the code changes without understanding the technical debt created by them. On this I agree. There are models that can be run locally, this morning I tested Gemma 4 running on 128 GB of RAM. It was very slow, like 20 minutes to refactor something instead of 20 seconds, but it seems to be as capable as the paid models that run on an expensive cloud subscription on one of these hated data centers. And no data is uploaded to them.
simianwords 6mo ago
I suggest actually using Claude code and make a sample app using it. It absolutely can make apps even if you don’t know any fundamentals. I think it can work up to 20k LOC from my experience. You do need a human to give feedback but not someone who understands software principles.
- Anamon 6mo ago
  That's a one-shot, though. Now try making something you want to build on. It falls apart very, very quickly. Not to mention that these prototypes and samples all turn out more or less the same. Obviously, given how LLMs work. Like LLM prose, LLM-generated web apps have a distinctive, samey look and feel.

mnls 6mo ago

Damn!!! And I keep hardening my RSS app which was partly vibe coded and not exposed to the WAN while "professionals" give data away.

aitchnyu 6mo ago

Is there anybody making some framework where you declare the security intentions as code (for each CRUD action) and which agents can correctly do and unit test? I have seen a Lovable competitor's system prompt have 24 lines of "please consider security when generating select statements, please consider security when generating update statements..." since it expects to dump queries here and there.

CrzyLngPwd 6mo ago

I think it is wonderful. It's reminiscent of the 90s, where every middle manager had dragged and dropped some boxes on some forms, and could get a salesman to sell it, without a care in the world for what was going on behind the scenes. Until something crashed and recovery was needed, of course. The piper always needs to be paid.

Steve16384 6mo ago
Or someone starts with an Excel spreadsheet just to "keep track of a few things". Then before they know it, it has become a critical part of the business but too monolithic and unorganised to be usable.

debarshri 6mo ago

I believe there are various dimensions to vibe coding. If you work with an existing codebase, it is a tool to increase productivity. If you have domain specific knowledge, in this case - patient management system, you can build better systems. Otherwise, you endup simulating the production. Lot of the non technical folks building products with AI Vibe coding are basically building Product Simulations. It looks like a product, functions like a product but behind the scene, you can poke holes.

vedant_awasthi 6mo ago

[flagged]

fakedang 6mo ago

Report them - that right there is 5+ different violations. Only then will they realize their stupidity.

seethishat 6mo ago

I saw something very similar a few months ago. It was a web app vibe coded by a surgeon. It worked, but they did not have an index .html file in the root web directory and they would routinely zip up all of the source code which contained all the database connection strings, API credentials, AWS credentials, etc.) and place the backup in the root web directory. They would also dump the database to that folder (for backup). So web browsers that went to https://example.com/ https://example.com/ could see and download all the backups. The quick fix was a simple, empty index.html file (or setting the -Indexes option in the apache config). The surgeon had no idea what this meant or why it was important. And the AI bots didn't either. The odd part of this to me was that the AI had made good choices (strong password hashes, reasonable DB schema, etc.) and the app itself worked well. Honestly, it was impressive. But at the same time, they made some very basic deployment/security mistakes that were trivial. They just needed a bit of guidance from an experienced devops security guy to make it Internet worthy, but no one bothered to do that. Edit: I do not recommend backing up web apps on the web server itself. That's another basic mistake. But they (or the AI) decided to do that and no one with experience was consulted.

Arch-TK 6mo ago
The fix is to not let users download the credentials. In fact, ideally the web server wouldn't have access to files containing credentials, it would handle serving and caching static content and offloading requests for dynamic content to the web application's code. Disabling auto-indexing just makes it harder to spot the issue. (To clarify, also not a bad idea in principle, just not _the_ solution.) If the file is still there and can be downloaded, that's strictly something which should not be possible in the first place.
simianwords 6mo ago
Agent-Native DevOps tools are probably necessary. There should be no reason they would do it manually. How I see it happening: agents like CC have in built skills for deployment and uses building blocks from either AWS or other simpler providers. Payment through OAuth and seamless checkout. This should be standardised
shivaniShimpi_ 6mo ago

cmiles8 6mo ago

There’s another version of the Mythos narrative that reads like: AI companies realized that all this vibe coding has released a shitstorm of security vulnerabilities into the wild and so unless they release a much better model to fix that mess they’ll be found out and nobody will touch AI coding with a 100ft pole for the next 15 years. This article points more towards this narrative.

coopykins 6mo ago

I interviewed some years ago for an AI related startup. After looking at the live product, first thing I see is their prod dB credentials and openAI api key publicly send in some requests... Bad actors will be having a lot of fun these days

jseabra 6mo ago

[dead]

shivaniShimpi_ 6mo ago

Every other field that's figured out high stakes failure models eventually landed on the same solution - make sure two people that understand the details are looking at it - pilots have copilots surgeons with checklists and nuclear plants have independent verification. Software was always the exception, cause when it broke it mostly just broke for you, vibe coding is not going to change the equation, it barely removes one check that existed before is that the people who wrote the code understood what was going on, but now that's gone too

Ekaros 6mo ago
We do have code reviews for pull requests. But on average I would guess there is great amount of complacency there. I suppose old proper QA phase was best answer we had. But that is expensive and slow.
- NoGravitas 6mo ago
  Maybe expensive and slow is actually an improvement.
  - shivaniShimpi_ 6mo ago
    i mean i don't disagree with you, but then there's at least unit test cases that we write as devs in a certain way that ai just doesn't understand quite well yet. we worked so much on speed that we forgot along the way that slow was a safety net. we tend to let our qa teams leverage ai as much as they want to automate test cases but i don't feel super comfortable getting rid of it just yet
- deleted 6mo ago
  [deleted]

avazhi 6mo ago

You guys realise this is AI slop on AI slop, right?

krapp 6mo ago
This is reality now, what do you want?
- avazhi 6mo ago
  HN should just have a rule that all content should be human-generated. This post is literally an LLM writing about something an LLM did; it's a bot botting about a bot. Aside from how funny and dystopian that is - just ban it? Just make a rule that submissions require a human author. I don't think solving this is all that complicated, at least for now. It isn't like it's currently difficult to tell what is and isn't LLM word salad, though that will likely change in the future, but by then the argument will involve whether it really matters or not. But for now, when 80% of the submissions are LLM garbage and it really is garbage, it's pretty jarring.
  - krapp 6mo ago
    Almost everything of interest to HN is written by LLM now, at least partially, and if it isn't it will be soon. There already is a rule against comments being LLM generated (which gets ignored all the time) but if we extended that to articles there would be almost nothing to post.

crvst 6mo ago

Cool story bro. Of course it’s true if it made it to HN. Who needs proofs.

zkmon 6mo ago

Technology for greed vs technology for need. Greed has its cost.

erelong 6mo ago

To me it just sounds like eventually someone will figure out how to make vibecoding more reasonably secure (with prompts to have apps be looked at for security practices?) unless cybersecurity is such a dynamic practice that we can't create automated processes that are secured Essentially a question of what can be done to make vibecoding "secure enough"

TeMPOraL 6mo ago

I have my doubts on the story. I consulted on a medtech project in the recent past in similar space, and at various points different individuals vibe-coded[0] not one but three distinct, independent prototypes of a system like the article describes, and neither of them was anywhere near that bad. On the frontend, you'd have to work pretty hard to force SOTA LLMs to give you what is being reported here. Backend-side, there's plenty of proper turn-key systems to get you started, including OSS servers you can just run locally, and even a year ago, SOTA LLMs knew about them and could find them (and would suggest some of them). I might be biased by my experience, because we actually cared about GDPR and AI act and proper medical data processing, and I've spent my fair share of time investigating the options that exist. Still, I'm struggling to imagine how one could possibly screw it up anywhere near as what the article described. Like, I can't think of a way to do it, to the point I might need to ask an LLM to explain it to me. -- [0] - Not as a means of developing an actual product, but solely to see if we can, plus it was easier to discuss product ideas while having some prototypes to click around.

hamasho 6mo ago

The worst blunder I made was when I explored cloud resources to improve the product's performance. I created a GCP project (my-app-dev) for exploring how to scale up the cloud service. I added several resources to mock the production, like compute instances/cloud SQL/etc, then populated the data and run several benchmarks. I changed the specs, number of instances and replicas, and configs through gcloud command. $ gcloud compute instances stop instance-1 --project=my-app-dev $ gcloud compute instances set-machine-type instance-1 --machine-type=c3-highcpu-176 --project=my-app-dev $ gcloud sql instances patch db-1 --tier=db-custom-32-131072 --project=my-app-dev But for some reason, at one point codex asked to list all projects; I couldn't understand the reason, but it seemed harmless so I approved the command. $ gcloud projects list PROJECT_ID NAME PROJECT_NUMBER my-app-test my app 123456789012 my-app-dev my app 234567890123 <- the dev project I was working on my-app my app 345678901234 <- the production (I know it's a bad name) And after this, for whatever reason it changed the target project from the dev (my-app-dev) to the production (my-app) without asking or me realizing. Of course I checked every commands. I couldn't YOLO while working on cloud resources, even in dev environment. But I focused on the subommands and its content and didn't even think it had changed the project ID along the way. It continued to suggest more and more aggressive commands for testing, and I approved them brain-deadly... $ gcloud sql instances patch db-1 --database-flags=max_connections=500 --project=my-app $ gcloud compute instances delete instance-1 --project=my-app $ echo 'DELETE FROM users WHERE username="test";' \ | gcloud sql connect my-db --user=user --database=my-db --project=my-app $ wrk -t4 -c200 -d30s \ "http://$(gcloud compute instances describe instance-1 \ --project=my-app \ --format='get(networkInterfaces[0].accessConfigs[0].natIP)')" It took a shamefully long time to realize codex was actually operating on production, so I DDoSed and SQL-injected to the production... Fortunately, it didn't do anything irreversible. But it was one of the most terrifying moments in my career.

BitsAndObjects 6mo ago
This is part of the reason deployments to production cloud environments should: 1. Only be allowed via CI/CD 2. All infra should be defined as code 3. Any deployment to production should be a delayed process that also has a human-approval step in the workflow (at least one, if not more) (Exactly where that review step is placed depends on your organisation - culture, size, etc.) And anyone that does need to touch production should do so from an isolated VM with temporary credentials. Developers shouldn't routinely have production access from their terminal. This last aspect is easy and cheap to set up on AWS. I presume it's also possible in Google Cloud.

keysersoze33 6mo ago

The takeaway is to vet new companies one is dealing with - even just calling them up and asking if they've AI generated any system which deals with customer/patient data. This is going to get more common (state sponsored hackers are going to have a field day)

0-bad-sectors 6mo ago

I think AI will be too expensive soon for normal/non technical people to tinker with and this kind of vibe coding stories will disappear.

Garlef 6mo ago

If this happened in Germany, this is most likely not only a breach of some contract but actually a criminal offense. (In not a lawyer and so I might be mistaken about this; Especially the level of intentionality might be a factor)

zzyzxd 6mo ago

Vibe coding is fun, but I can't trust it to make any serious decisions. Like, it knows what's the best way to do a thing, but when encounters challenges, it started to make all kinds of excuses to cut corners, just like humans. "but honestly, it's cluster internal traffic so unencrypted traffic is fine". "Given the urgency and tight timeline, your best option is bypassing the pipeline and deploying it manually". "Per my research, XXX also did this so you are fine". If I don't have disciplines or principles, or if I am just technically incompetent, its suggestions would sound so reasonable.

oxag3n 6mo ago

What if that AI was trained to hide implementation as much as possible? E.g. by making the client as thin as possible, using OAuth for authentication, following some robust template, would it be better? Avoiding such low-hanging-fruit disasters is doable and major players have incentives to fix it. This however wouldn't change the goal - replace all software engineers with DIY nephews with LLM in their hands. Even worse - people start thinking it's infeasible to learn hard things, because it prevents you from moving fast, from one domain to another. My area of expertise is cryptography-adjacent. Non-trivial fraction of students think that cryptography and security is a dead-end skill as all companies in this area are being replaced with AI. I asked them to implement web-bot-auth as simple as possible, because I know how AI can read specification and follow it.

MajorLettuce 5mo ago

"That sparked an idea: why use an industry-proven solution when you could just build your own patient management system?" lol I wonder if it is an actual dunning kruger effect at play here (confidence issue) or the lack of critical thinking. The first one could be excused somewhat, but I am afraid it is more of the second one

fredericgalline 5mo ago

[dead]