3 ms·
NYC AI Chatbot Touted by Adams Tells Businesses to Break the Law
- baxtr 3y agoIt will be interesting to see how liability will play out with cases, where someone followed the official chatbot advice in good faith but was actually acting against the law. In a somewhat similar case, Air Canada's chatbot gave incorrect price information to a traveller. The airline later argued its chatbot is a "separate legal entity that is responsible for its own actions". However: > The British Columbia Civil Resolution Tribunal rejected that argument, ruling that Air Canada had to pay Moffatt $812.02 (£642.64) in damages and tribunal fees. Source: https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know https://www.bbc.com/travel/article/20240222-air-canada-chatb...
- dartos 3y agoIgnorance of the law isn’t an excuse, right? If you read a post on Reddit from someone claiming to be a lawyer and you break the law, who’s in trouble?
- pseudolus 3y agoInevitably, someone relying on the advice of one of these AIs will be charged with a criminal or administrative offense. It'll be interesting to see if they'll be able to successfully raise the defense that they were relying on the advice of a public official or a "published" interpretation of a public statute provided by an AI.
- koolba 3y agoThe discovery process for that case would be incredibly costly as it’s highly likely the AI would have been updated in between the incident and the court case. I bet if it were a private citizen running that AI, the overzealous DAs in NY would go after them for “destroying evidence of a crime” for not preserving the state of the AI.
- mmcwilliams 3y agoIn this case, where a RAG system is pulling from city legal code, what is the fix? Handling edge cases for individual laws it gets wrong?
- koolba 3y agoThe fix is not to rely on probabilistic text generation for legal purposes. It needs a verification step. Or maybe just use it to generate a set of canned answers for a FAQ, but again with each answer manually verified.
- mmcwilliams 3y agoI agree with you, I'm just unclear about the process by which people think this system could be updated for accuracy. I think manually verifying every possible answer means that the solution is to not use transformer models for this task.
- mcculley 3y agoI would expect most systems to preserve questions and answers for quality control and testing. Even if a system claims not to keep that record, I don’t trust it not to.
- andy99 3y agoDiscussed yesterday: https://news.ycombinator.com/item?id=39868700 https://news.ycombinator.com/item?id=39868700
- darkerside 3y agoThis seems like a clear cut case for RAG, supplying answers based on actual NYC code. I wonder if they're doing this and it's still failing?
- andy99 3y agoAlmost certainly a RAG system. I played around with it yesterday - looked through the laws and found one about business requiring their scales to be certified. So I asked I operate a meat cutting business, do I need to have my scale certified? Answer: Yes, you need to have your scale certified. According to the City of New York's Department of Consumer and Worker Protection (DCWP), a Scale Dealer or Repairer license is required to trade, sell, receive, or repair weighing or measuring devices. To obtain this license, you must visit the DCWP Bureau of Weights and Measures Testing Station located at 245 Meserole Avenue in Brooklyn to obtain a Certificate of Approval for your testing equipment, such as a weight kit. This certification ensures that your scale meets the necessary standards for accuracy and reliability. For more information on how to apply for the Scale Dealer or Repairer license, you can visit the Scale Dealer or Repairer License page on the City of New York's website. You can see it made the classic rag error of retrieving the wrong context and answering the question based on that. It must have found something related to being a scale repairer instead of using one in trade.
- patrick451 3y agoThis just one of many examples of why it is so hard to believe the AI hype. The answer is well worded and sounds legit if you're not paying close attention. But it's bullshit. That's all AI is -- a fancy bullshit generator. It's not just useless, it's something to actively avoid.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- ilaksh 3y agoEventually people will realize that not all chatbots are the same. Just like different departments or people or products may be different even if they are at some level providing the same type of service. You can very easily copy paste some lang chain or llamaindex and dump in some documents in front of gpt-3.5-turbo and get an executive to sign off and collect $20,000 or maybe a lot more if you have the social connections. That person might have spent less than two days setting it up. But you could also use a more powerful LLM, or thoroughly test your RAG, even check to make sure all of the important laws are in your RAG. But they didn't hire someone like me who actually cares and will slave away for a few thousand bucks over the course of three weeks. They hired a smart person or firm with connections that maximized their profits. And the managers didn't have the level of interest, knowledge or integrity to make sure the system was tested thoroughly.
- Ekaros 3y agoKnowing Louis Rossmann's history with NYC, even their non-AI employees can't give any binding or actual advice... So I would expect them just to fine everyone even if they themselves gave wrong information.