7 ms·
New York could prohibit chatbot medical, legal, engineering advice
- phishin 7mo agoLawyers protecting lawyers. The one thing AI could help the ordinary people fight back against corporations.
- bluepeter 7mo agoSame as it ever was. Honestly, I think we've probably passed peak consumer AI with with all the "guardrails" that regulations will require.
- bklosky 7mo agoRent seeking via occupational licensing, there is nothing new under the sun
- terminalshort 7mo agoThat's a lot of words to say "cartel"
- Esophagus4 7mo agoA cartel of lawyers would be like… the most boring cartel
- bigbadfeline 7mo ago>> OP: New York could prohibit chatbot medical, legal, engineering advice Isn't software engineering "engineering" too? Why split hairs, prohibit all or nothing. Of course it's not about logic or safety, it's about social engineering.
- tzs 7mo ago> Isn't software engineering "engineering" too? No. FTA:>> Important nuance: "engineering" here means New York Education Law Article 145 professions (professional engineering, land surveying, and geology), not software engineering. What the law is basically saying is that in fields where it would by a crime for a random human to give any substantive response, information, or advice chatbots also should not do so. Software engineering is not one of those fields. The law does not make it a crime for a chatbot to do so, but if it does and the person it advises suffers damages it makes it so the injured person can sue the chatbot operator for those damages (and for attorney fees if the chatbot operator willfully allowed the chatbot to give such advice).
- capital_guy 7mo agoThe person that introduced this bill is not a lawyer. A former tech worker, actually.
- bluepeter 7mo agoWhats at least somewhat humorous is the disclaimer requirement that "[t]he text of the notice shall [be] no smaller than the largest font size of other text appearing on the website on which the chatbot is utilized." H1 hero font size here we come for disclaimers! (Which don't do anything, per the bill, anyway.) But also is the fancy thought that chatbots only appear on websites.
- neonnoodle 7mo agoWe're bringing back the <blink/> tag very strongly, some say more than ever before
- terminalshort 7mo ago1. Put very large font size title on the main page. 2. Display the disclaimer in the same font size to comply. 3. Disclaimer is now completely unreadable because it appears in such a large font size that it is one or two words per line.
- fwip 7mo agoSeems like a good bill, at least directionally. If it's a crime to provide advice of this nature without a license, then chat bots shouldn't be dispensing it either.
- bluepeter 7mo agoThis is not directionally good because NY already has laws against unauthorized professional practice and deceptive conduct, and S7263 mainly replaces regulator-led enforcement with a vague, fee-shifting private cause of action that is likely to drive serial plaintiff litigation while chilling useful consumer guidance.
- tekne 7mo agoYou've got a license for looking up the law/engineering textbooks/your symptoms, pal?
- kakacik 7mo ago[flagged]
- operatingthetan 7mo agoThat is not remotely ad hominem. I'd suggest you refresh your understanding.
- alwa 7mo agoIt seems to me that @tekne is comparing the LLM to a reference source. I took them to be pointing out that unlicensed-practice laws don’t crack down on textbooks, or reading the law for yourself (or even going jailhouse-lawyer or trying to defend yourself in court). Rather, that the laws aim to keep the professional title commercially reliable, so that it indicates to the public that the person using it has proven some minimum level of expertise. So the analysis would turn on whether a reasonable person would confuse ChatGPT for a practicing lawyer, or doctor, or whatever—not whether it communicated legal or medical facts. Now, to my mind, the facts are the least interesting part of those professions—I pay those professionals precisely for their nuance and judgment and experience beyond the bare facts of a situation. And I think the ChatGPTs of the world do embellish their responses with the kind of confidence and tone that implies nuance/judgment/experience they don’t have. But I do think @tekne was making a valid point.
- paxys 7mo ago"Hey ChatGPT, my NYC landlord is raising my rent by $500, and says I must pay by Monday or leave. What do I do?" ChatGPT - This is very likely illegal under Housing Stability and Tenant Protection Act of 2019 (HSTPA), specifically New York Real Property Law § 226-c (Notice required for rent increases), RPL § 232-a / § 232-b (Month-to-month termination), RPL § 232-c (Fixed-term lease protections), RPAPL § 711 (Legal eviction procedure) and NYC Admin Code § 26-501+ (Rent stabilization). Here's what you should reply with... And here are some city resources you can contact... ChatGPT now - IDK, pay a lawyer. So under the guise of "protection" you are taking away the strongest knowledge tool common people have had at their disposal in a generation, probably ever.
- cm2012 7mo agoAgreed. This law would have awful outcomes.
- bluepeter 7mo ago100% this.
- terminalshort 7mo agoTheir protection, not yours. Hopefully this will draw public backlash just like when they tried to ban Uber and make everybody go back to cabs. Fuck the entire credential cartel based system of societal organization. Burn it to the ground.
- theturtletalks 7mo agoThey will come for medical advice provided by AI as well. Doctors have been gate keeping that forever and they want you to have to go thru them instead of diagnosing yourself thru AI. Yes, there are people that will misdiagnose themselves, but I’ve read stories where doctors ignore patients symptoms or wave them off, and ChatGPT helps them find the underlying issue and actually improve their lives. Even if doctors and the medical field can’t handicap AI giving medical advice, I’m sure they are going to make it much harder for patients to get their hands on their own scans and bloodwork.
- htrp 7mo agoloopholes wide enough to drive a truck through
- moomoo11 7mo ago[flagged]
- cindyllm 7mo ago[dead]
- HanClinto 7mo agoIt's for your own good! Think of the children! You don't want puppies to DIE, do you? [0] [0] - https://www.youtube.com/watch?v=eXWhbUUE4ko https://www.youtube.com/watch?v=eXWhbUUE4ko
- hypeatei 7mo agoYes, there's a lot of naivety in the "regulate everything" camps that believe you can simply solve problems, including the ones caused by legislation, with more legislation.
- tim-tday 7mo agoNow do target acquisition for lethal munitions.
- cm2012 7mo agoBad law. I have gotten better advice from modern llms than from most of the professional categories above.
- terminalshort 7mo agoI have narcolepsy. It took a dozen or so doctors and years of suffering before I got a correct diagnosis, and even then it was only because I diagnosed myself with Google and then specifically made an appointment with a doctor who specializes in it. Gemini nails it when I put in my symptoms.
- leptons 7mo agoI've been given absolutely wrong medical advice from Gemini, specifically about medication I was taking, so I know it was wrong. So, YMMV.
- freejazz 7mo agoDo share with the class
- OutOfHere 7mo agoNew York residents who opposite this bill can go to https://www.nysenate.gov/legislation/bills/2025/S7263 https://www.nysenate.gov/legislation/bills/2025/S7263 , register and sign in, click Nay on the bill's page, and submit feedback for their vote. --- ChatGPT> Before I answer your question, which state are you a resident of? Human> Not New York. Continue! ChatGPT> Alrighty then! Here you go...
- cm2012 7mo agoGreatly appreciated! I used this link to message my State Senator, I didn't even know her name before.
- deleted 7mo ago[deleted]
- deleted 7mo ago[deleted]
- Esophagus4 7mo agoThe disclosure requirement is probably a decent thing (you have no idea how many people come into the ER and say, “But ChatGPT told me to do [dumb thing].”) But preventing it from answering at all is absurd. Make responsible disclosure absolve AI providers of legal responsibility (not legal advice lol). That way if users ever sue OpenAI for giving them bad advice, OpenAI can say “no way man, you read the disclosure!” I’m usually in favor of giving people the best info they can and letting them make their own decisions. This could just be like those terms of service things everyone clicks “agree” to and I’d be fine with that.
- GuinansEyebrows 7mo ago> Make responsible disclosure absolve AI providers of legal responsibility (not legal advice lol). disclaimer: OSTENSIBLY if the sole aim was to reduce AI provider culpability, then a disclaimer would meet that requirement. humans famously suck at acting within rational self-interest; therefore, this isn't trying to protect AI providers of legal responsibility. it's trying to mitigate unwanted results from actions taken based on decisions informed by unverified LLM output.
- Esophagus4 7mo agoMakes sense. But protecting people from themselves is hard to legislate :)
- terminalshort 7mo agoI am skeptical of this claim. What are some of the dumb things that people do on ChatGPTs advice that puts them in the ER?
- Esophagus4 7mo agoTalk to your doctor friends. Edit: sorry, that was a rude way for me to respond. But this is pretty googleable, and I’m going off of war stories two doctor friends of mine have shared. Eg https://nypost.com/2025/10/24/health/real-life-ways-bad-advice-from-ai-is-sending-people-to-the-er/ https://nypost.com/2025/10/24/health/real-life-ways-bad-advi... To explain, I reacted strongly because there’s a style of HN comment that is basically “source?” But it’s a trap, because they are trying to discredit and dismiss your point, not understand it. You can provide the best sources for something that is simply fact, but it won’t change their mind - they will just find pedantic way to further discredit or dodge anything you put in front of them. I once got into this pattern explaining to someone that, yes, stress actually does cause physical ailments and provided like 5 NIH papers out of the thousands that support that fact, but the commenter just tried to further discredit each study for meaningless and pedantic reasons. So I didn’t want to go down that road again. If you were seeking to understand, I’m sorry for jumping down your throat. If you were seeking to discredit and be the contrarian… stop.
- iamnothere 7mo agoDownload one of the freely available models and use that, if you have the hardware for it. It’s not a good idea to ask sensitive questions on these nontransparent chatbot platforms. (FWIW I also think this is a bad law. Why not improve privacy protections instead? Why not allow nonprofessional use with a disclaimer?)
- mbgerring 7mo agoYes, that’s correct, I do not want a vibe-coded freeway overpass, thanks. We all need to get serious about the unavoidable, unsolvable fact that these tools produce output of unknowable accuracy. Some things require such accuracy, precision, and, importantly, accountability. LLMs are capable of none of these things. Refusing to be honest about this and take appropriate precautions will lead to disaster.
- tantalor 7mo ago> I do not want a vibe-coded freeway overpass I do. One of the reasons our infrastructure is so expensive is planning & design. For a single freeway overpass, you could be looking at $3M (25% of the total budget) before you have even broken ground. That covers feasibility studies, traffic modeling, rough layout, environmental studies, permitting, structural engineering, blueprints, bidding, contracts, community outreach, and the list goes on. If AI can reduce the cost of that by even 10%, that would be huge.
- mbgerring 7mo agoCool, we agree, and if you think the place to cut corners on that is the engineering calcs, you have lost your mind. If you do that, not only will people die, you will drastically increase costs because the infrastructure project you built will collapse. Europe and Asia both have reliable, modern infrastructure that’s decades ahead of the United States and they did not need the million-monkeys-on-typewriters machine to accomplish that.
- arionhardison 7mo agoI don't understand how anyone can rationalize this bill in the face of what OpenAI just agreed to with the DoD. AI can surveil and direct munitions but it cant answer legal questions. Wouldn't this also violate the "no state my limit or restrict the use of AI" that the current administration is pushing?
- bee_rider 7mo ago> I don't understand how anyone can rationalize this bill in the face of what OpenAI just agreed to with the DoD. NY doesn’t have any obligation to agree with the DoD. Also the applications seems quite different, although I don’t think AI should actually be relied on for either one! > Wouldn't this also violate the "no state my limit or restrict the use of AI" that the current administration is pushing? No, it doesn’t violate it. States can’t violate executive orders, because executive orders aren’t instructions for the states. The instructions are for the executive branch, for example, if this becomes law the US Attorney General will try to find some way to fight against it.
- arionhardison 7mo agoI did not understand this, thank you for clarifying.
- chimeracoder 7mo ago> I don't understand how anyone can rationalize this bill in the face of what OpenAI just agreed to with the DoD. > AI can surveil and direct munitions but it cant answer legal questions. There's no contradiction. The people sponsoring this bill don't think that AI should be used for either of those purposes.
- arionhardison 7mo agoUnderstood, I was under the impression that they were supporting this use and opposed to the DoD usage. Thank you for the clarification.
- francisofascii 7mo agoThe other professions are creating lawful protections for themselves in the upcoming AI revolution. Software engineers have no such protection.
- aetherspawn 7mo agoGreat, electrical and mechanical engineers are already underpaid, under appreciated and overworked. I’ve always found it amusing that lawyers and accountants flash their license around with pride, put it in their email signatures, etc. and it provides authority for them. When people see chartered lawyer or accountant, they respect that person and take their advice. An engineering license, on the other hand, is so rarely talked about and never quoted in email signatures and the like. And even as a chartered engineer, people really just treat you like a mechanic or a trade and mostly ignore your advice anyway. Yet, it takes the longest to get, and has the most exams/hardest subjects, except for Doctors. Anything to make an Engineering license worth more is good in my books. Besides, in my experience ChatGPT gives wrong advice for engineering around 50% of the time and therefore probably has no business giving it.
- josefritzishere 7mo agoNew York really should. AI is literally killing people. This is not OK.
- tzs 7mo agoThe only penalty for violating this law is that if someone is injured by your chatbot giving bad advice they can sue you for the actual damages they suffered, and if your violation of the law was willful they can also sue for attorney fees. This only applies to advice that would have illegal for a human to give who is not licensed in the relevant field.
- thrill 7mo ago“Hey Chat, I feel pretty bad, here’s my symptoms. It’s expensive to go to ER, but should I see a doctor?” Can’t advise you buddy, but here’s some OTC meds that have paid for placement. Been nice knowing you and good luck!
- gdevenyi 7mo agoThe medieval guilds will return.
- cybercatgurrl 7mo agothey see the writing on the wall and know their time is coming to an end eventually. this feels like the dying last breath of a rigid system unwilling to accept change and acknowledge that the power is now with the people
- sayhisam1 7mo agoMore slopulist policy. Why do we care about saving long-tailed distribution of idiots from themselves at the expense of everyone else? And is this even a real demonstrable issue (in terms of percent of harmful responses to total number of responses)