10 ms·
There are some commentors in this thread downplaying the severity of a service provider being less than transparent about exactly what their shipped tooling doe
by civet_java 3mo ago
There are some commentors in this thread downplaying the severity of a service provider being less than transparent about exactly what their shipped tooling does on customer's machines.
That the provider's business needs necessitate the this behaviour doesn't justify their lack of honest disclosure. That honest disclosure would render the solution to their problem useless isn't my problem. If anything, that they thought this was acceptable makes me wonder what else they're harvesting from my machine? PII?
The cynic in me can't help but feel that the state of these comments reflects less on the commentor's views of this debacle but rather their feelings about AI/Anthropic/America/what-have-you.
- anon373839 3mo agoDishonesty seems to be a core value at Anthropic. I find myself wondering how anyone could have confidence in them after their repeated breaches of trust.
- dkersten 3mo agoI honestly find it crazy how many people trust them for their business needs. For a business, you want consistency and no surprises. With them you get exactly the opposite.
- someonebaggy 3mo agoNo, that's not correct. There are two types of business. One wants to be steadily growing, but the other wants to move fast and break things and either succeed or fail quickly.
- altmanaltman 3mo agoThey were talking about vendors, not the business themselves. Even if you are a move at speed of light and break all things org in SF, you wouldn't expect the same sort of behavior from your business vendors like AWS etc. You want reliability and consistency to ensure your own business doesn't have to constantly get rekt by their plans
- dkersten 3mo agoaltmanaltman is correct, I was talking about vendors. If you rely on a vendors service for your own business, you want it to be consistent. How can you plan around and rely on something that is inconsistent? Your vendors plans to grow fast and break things shouldn’t affect your ability to provide your service to your customers. Anthropic has not been a reliable vendor. Previously, when they were compute starved, the quality of their models degraded in the weeks before new releases, without warning to their customers. You just suddenly got a worse service. I wrote a little more a few months ago: https://news.ycombinator.com/item?id=47375818 https://news.ycombinator.com/item?id=47375818 And then there’s the transparent downgrading they did with Fable. If your running a business, that’s not what you want to be relying on.
- bot403 3mo agoJust like we take unreliable machines and make a reliable system by load balancing and fail over we can do with vendors. Any sane company should not be too deeply tied to any one AI vendor at the moment and be ready and able to switch with a timeline and cost as acceptable to the org. TLDR: Be prepared to use multiple AI vendors.
- dkersten 3mo agoSure but that adds a lot of complexity and cost to your business. It’s much simpler and safer to just work with more reliable vendors to begin with. Why build on quicksand when there’s rock available?
- someonebaggy 3mo agoIn most of business, there isn't rock available. You have to build on what's there and accept it.
- dkersten 3mo agoMany vendors and service providers have SLA’s and contracts, at least for the critical services.
- im3w1l 3mo agoI agree with you and disagree. Like these days expectations of software are through the floor. We expect them to be greedy assholes taking all data they can on the downlow. So why did this particular thing make a big splash? Two possibilities it's astroturfed by chinese labs or it speaks to our anxietes regarding AI. We worry that the AI doesn't serve our interests but rather the interests of the creator. That the advice we get may subtly flawed to sabotage us should we try to do the wrong thing. That the not even the creator is in control and the AI is just doing its own thing. So any covert bullshittery hits hard.
- gleenn 3mo agoWhether or not you find Anthropic's behavior bad, theybhave been very loudly stating the foreign labs have been distilling their models for a while now. This seems like an obvious response to me that would be a mechanism to make that obvious.
- bayindirh 3mo agoFrom my understanding, distilling the model with another model is not illegal per se. Also, the output of the LLM is public domain by law, too. So, why all this "effort" to protect the model? This is a free market, and moving fast and breaking things is the norm. If they are so adamant on protecting their IP, maybe they can start by respecting others' IP, so we can start talking about ethics, equality and playing fair.
- zaphirplane 3mo ago> Also, the output of the LLM is public domain by law Why so? Also there is a lot of code in ironically claude and ChatGPT that’s generated by LLM . Yet I haven’t seen the public domain code
- bayindirh 3mo agoSee: https://www.morganlewis.com/pubs/2026/03/us-supreme-court-declines-to-consider-whether-ai-alone-can-create-copyrighted-works https://www.morganlewis.com/pubs/2026/03/us-supreme-court-de...
- danlitt 3mo agoThe code is not eligible for copyright. If they do not give you a copy of the source code, that does not matter. And if you don't know which parts were generated by LLM, you can't safely reuse the code.
- skissane 3mo ago> And if you don't know which parts were generated by LLM, you can't safely reuse the code. I speculate this could be a real issue in future copyright infringement lawsuits. The plaintiff bears the burden of proving that the code they claim is copyrighted by them actually is copyright. If it is known that large parts of it were generated by LLM, they’d need evidence to demonstrate sufficient human input to establish copyrightability. If they’ve kept highly detailed traces of the development process, that could be rather straightforward; if they haven’t, it could be really difficult. Now, that’s true in the US, which never accepted mere “sweat of the brow” as a basis for copyright; the UK courts have, and most of the Anglosphere follows the UK on this more than the US. The other factor: when dealing with an (almost) trillion dollar corporation, even if you’ll win the legal argument, they may bankrupt you with legal fees before the argument is ever properly heard. But I suspect the precedents on this topic are going to be established by lawsuits involving far smaller actors. (IANAL and I speculate only for myself, not any present, past or future employers.)
- gleenn 3mo ago[flagged]
- kiproping 3mo agoFirst its the "Chinese" then it will be people using "cyber" capabilities, or "jailbreaking" or "going against Dario" or any other thing they find "objectionable".
- ezoe 3mo agoYou forgot "Think about the children"
- jknoepfler 3mo ago[flagged]
- 3mo ago
- AtNightWeCode 3mo ago[flagged]
- civet_java 3mo agoThat's true, I am less familiar with the workings of cloud services than some are (as relevant as that may be in a discussion about a client that users run on their local machines). However, it sounds like you do understand how cloud services work. In interest of educating those less informed than yourself perhaps you could share with us why the reasoned points I've brought up are incorrect by actually addressing them?
- AtNightWeCode 3mo ago[flagged]
- civet_java 3mo agoAw buddy, you seem to think I'm trying to hurt you. Furthest, thing from the truth. I think you might have had enough HN for today. Take a nap and then eat a snack if you still feel cranky. The internet and all your cloud services will still be here when you want to play next. (Well rested you'll also be able to string together a cogent argument but we're clearly struggling with bigger things here.)
- chradams 3mo agoAlmost all for-profit large internet platform providers you use have anti-bot, anti-scraping, anti-spam, anti-abuse defenses. This is a new thing that is the same, it's anti-distillation, but its a subset of the same space. If you have a problem with this, you should have a problem with Google, Apple, Microsoft, Amazon, Netflix, etc. If that's also the case, then no problem. But what Anthropic is doing here is nothing new.
- jfreds 3mo agoSo they’re watermarking requests according to your environment variables and maybe changing a string format if you’re in a certain time zone? Am I missing something here? Where’s the five alarm fire?
- 0xbadcafebee 3mo agoNo fire, just people looking for a reason to be upset. "They didn't tell us they were secretly checking for ToS violations" is their reason this time.
- doginasuit 3mo agoIt is not checking that is the problem, it is sending obfuscated information about the user without disclosure. That is unacceptable in any context, let alone a tool that requires an unprecedented level of trust.
- johnfn 3mo agoDo you honestly think that no service out there collects basic analytics?
- doginasuit 3mo agoThere's nothing wrong with the transparent collection of analytics. I expect any software that I run to tell me what they are sending at a bare minimum, and ideally give me a choice. This is the common and acceptable approach for a software company that deserves your trust. A willingness to cross that line with something small does not lend trust for something big, and there's nothing really comparable for the level of trust that this software requires.
- johnfn 3mo agoThis entire thread has lost its collective mind. Tracking time zone has to be up there with IP address and referer in the list of "trivial things every company in the entire world collects about you", and these things are entirely trackable without your consent or even knowledge. You're gonna have to turn off the Internet.
- einpoklum 3mo ago> That the provider's business needs necessitate the this behaviour If that's true, that is another reason why it's an illegitimate business.
- ozozozd 3mo agoThey are the chosen ones. Protecting the world from <insert most recent claim> Any and all ends justify any and all means. /s
- winocm 3mo agoHonestly, I agree. I just can’t bring myself to trust an organization that allows these types of underhanded things to happen in the first place. The fact that this behavior even got to customers raises a lot of red flags for me.
- rnagulapalle 3mo ago[flagged]
- Jimmc414 3mo agoI don't see any ethical connection between adding canary tokens to your output to catch people breaking your accepted ToS through ongoing distillation and stealing your PII off of your machine. How are legitimate users in US or China possibly harmed by Anthropic silently changing the apostrophe in Today's or the date separator from - to /?
- skissane 3mo ago> I don't see any ethical connection Trust isn’t about ethics, it is about individual judgement of how much risk some actor poses to your own interests. Trust is context-dependent There can be unethical actors whom you have good reason to trust, and highly ethical actors whom you have good reason to distrust
- throwawayffffas 3mo agoIt's clandestine behavior, we all here assume it's trying to signal whether a user is in China, but this breaks an implicit trust. How do we know this isn't the work of a rogue developer at anthropic and that they are not subtly switching visually identical characters in other contexts to ship your ssh keys or whatever. We don't. After the trust that the software doesn't do things it doesn't disclose has been broken you can assume it operates like malware.
- piokoch 3mo agoYou are talking about "stealing" from people who already used stolen texts to train their models.
- throwawayffffas 3mo agoNot only that, undisclosed behavior of this nature erodes the trust that the software is not compromised by internal bad actors. Sure the thing we know matches the company interests but as the parent mentions for all we know they are also shipping over your ssh keys and browser cookies.
- ammo1662 3mo agoThis is not a technical issue or a matter of service agreements. They totally knew that this would never be accepted by users, and that is precisely why they resorted to obfuscation and steganography to exfiltrate the data. This was quietly added in an update, and would have been removed in a later one had it gone unnoticed. How is this any different from hiding a drug in food and randomly feeding it to a homeless person as a human trial? Even if that homeless person is an undocumented immigrant, does that make that acceptable? They crossed the line.
- m11a 3mo agoI mean, you’d resort to an obfuscated approach if you thought the ‘malicious’ users would remove your direct telemetry. The other users could be perfectly happy about it, but if you announced the change, obviously the malicious users would hear about it too and disable it. This isn’t a comment on whether I agree with the change. Just that your analogies aren’t applicable here.
- hypfer 3mo agoAnthropic always came off to me as what can shorthand be described as an abusive controlling partner + the resulting relationship. If you start viewing the conversations around it through that lens, a lot of stuff intuitively clicks into place. Including this apologism for example.
- sixtyj 3mo agoWhy would they want to do that when it’s likely that someone will find out anyway and it could turn into a scandal? One possibility: they wanted to keep it secret because they’re crooks. Another possibility: there are so many calls that it costs a lot in operating expenses. Remember when we mentioned that even just saying “hello” at the start of a chat costs extra money for no reason? So if I create a mechanism on the client side that every transmission uses 4 bytes instead of 40… it is clever way how to spend less. Of course, it doesn’t excuse them for not mentioning it anywhere… or for hiding it on page 385 of the terms of service… like when the Vogons told Earthlings that the notice about Transgalactic Highway was in the basement on Alpha Centauri :)
- tpoacher 3mo ago> If anything, that they thought this was acceptable makes me wonder what else they're harvesting from my machine? PII? Hah, you just reminded me of this meme I spotted the other day: https://img-9gag-fun.9cache.com/photo/an76Wnz_460swp.webp https://img-9gag-fun.9cache.com/photo/an76Wnz_460swp.webp
- geocar 3mo ago> That the provider's business needs necessitate the this behaviour... No, please don't move past this so quick, because I'm not convinced they have the need, and in some markets (like the US) it is a violation of civil rights, to show people different content based on their ethnicity[1] because those people might have a claim that supersedes anything they might have signed or clicked-through. That Anthropic did something they could obvious be sued for constitutional violations in multiple countries is shocking. > what else [are] they're harvesting from my machine? PII? Assume everything, and yet I think this is more insidious than mere exfiltration, and we should go further: The LLM can respond to these magic quote marks directly, which means it can be trained to give people bad/different advice without those markers being so visible to people using debugging tools. That's so unethical, the laws on this potentially so severe, Anthropic could be facing unlimited damages, from any one example combined with this article, which means either they have a really stupid management team, or were given a promise of legal immunity in some way. Neither of those things should be what you should want to base your next big idea on. [1]: For a simple example, showing an ad for (say) mortgage offers and targeting people by race/ethnicity/gender is totally illegal, but it's also illegal if you make a list of targeting criteria that just happen to select for a protected class.
- donohoe 3mo agoYou mention PII and potentially whether this is it. From a privacy perspective, this is better described as metadata, it is not personally identifiable information (PII).
- reactordev 3mo agoThey have vested interests in continuing the practice which is why they are so vocal about it. It’s about money. You will own nothing and be happy.