7 ms·
Microsoft guide to pirating Harry Potter for LLM training (2024) [removed]
https://archive.is/D9vEN https://archive.is/D9vEN
- andsoitis 8mo agoThis article is from 2024 and points to Kaggle, which hosts the data set. I'm surprised that JKR's people haven't come down like a tonne of bricks on Kaggle / Microsoft. Does anyone know whether there is some special reason why this has lasted so long without being taken down?
- anonymous908213 8mo agoMy best guess is that it flew under the radar. The Kaggle dataset has 'only' 10,000 downloads, and the article itself probably doesn't have that many views. Still, this seems pretty far beyond the pale. Given the other case of AI-related plagiarism by Microsoft that was on the front page[1], it seems whatever review process they have for content that is published by their employees, if there is any review process at all, is deeply flawed. [1] https://news.ycombinator.com/item?id=47057829 https://news.ycombinator.com/item?id=47057829, "Microsoft morged my diagram". It was in a discussion there that someone pointed out this article linking to full downloads of the Harry Potter novels, which I thought deserved more visibility.
- zythyx 8mo agoAlso, I imagine that most of those 10k downloads are probably from AI trainers that are just speed running through Kaggle to obtain absolutely anything to train their AI. There are definitely other, more 'known' ways to obtain these books without finding them as random text files in an AI dataset operation
- selridge 8mo agoWhy did you think that?
- anonymous908213 8mo agoIt rubs me the wrong way that corporations get a free pass on copyright infrigement, while the rest of us are prosecuted as harshly as possible if caught. I think this, together with the morging plagiarism, also indicates a pattern of behaviour from Microsoft that should be reformed. I would prefer if Microsoft were not able to produce AI slop degradations of other people's work and claim it as their own.
- ryandrake 8mo agoIn general, if you want to get away with a crime, just do it as a corporation or as a billionaire.
- walletdrainer 8mo ago> while the rest of us are prosecuted as harshly as possible if caught But this is just a lie. Approximately nobody is prosecuted for copyright infringement.
- queenkjuul 8mo agoOkay but people have had their lives ruined deliberately by media companies over it. I'm sure you knew what they meant.
- walletdrainer 8mo agoNo matter how generously you want to interpret it, it’s obviously false. We’re moving the goalposts from the government systematically targeting normal people “if caught”, to only a handful of civil cases.
- eggsome 8mo agoSure, as a percentage it's very rare - but some people have died as a result: https://en.wikipedia.org/wiki/Aaron_Swartz https://en.wikipedia.org/wiki/Aaron_Swartz I think most would agree that cases like that act as a deterrent?
- blibble 8mo agobrb poking Rowling on twitter (done, contacted her lawyers too)
- deleted 8mo ago[deleted]
- k__o 8mo agomake sure u worded it right or she'll block you
- thunfischtoast 8mo agoRowling is among the last persons on earth that need help on this. But ignoring that: I do not think that these txt files being online do any economic harm. Noone will go and say "hey, I'm going to read these un-formatted text files instead of buying the 30 year old books for little money or pirating proper epubs which are trivial to find". If at all the kaggle dataset is free publicity. So as the author I would leave them online.
- miki123211 8mo agoBack in the day, text mining for science was mostly ignored, even though it was technically illegal. Authors didn't feel threatened by models for spam detection or sentiment classification. Demanding money from poor academics was pointless (they'd just move on to a different author) and bad PR. I mean, books3 contained hundreds of thousands of copyrighted books, and people released it under their own name.
- arkensaw 8mo agoMy guess is HP makes such an enormous amount of money already from movies, games, toys, and other tie-ins, that they can't be bothered to chase down the odd digital infringement of a plain text copy of the original books. I'm sure the scripts of Star Wars would be similarly ignored if they were used.
- amanzi 8mo agoThat doesn't justify what's going on here. Why is Microsoft endorsing the use of pirated materials.
- outside1234 8mo agoThe dataset is actually at Kaggle tho, but agree, they shouldn't use it as an example.
- crtasm 8mo agoThe file being hosted by another company doesn't change the fact that Microsoft is encouraging us to download and use it.
- Hamuko 8mo agoI think Microsoft employees should be smart enough to figure that "a globally beloved collection of seven books" (their words) is probably not free to use.
- blibble 8mo ago> Why is Microsoft endorsing the use of generative AI ftfy
- Ekaros 8mo agoIt is just very hard problem when you are very popular work. Trying to find and track and take down all copies of certain work online is constant fight. Sometimes things just slip especially if they are not that popular. Something like Harry Potter might be shared every day. And I mean as pirate work distributed as new copy. Staying on top of that will be very hard work.
- conartist6 8mo agoWhat in the absolute fuck
- dom96 8mo agoHow soon before someone will be able to make an online library which generates the original books using LLMs? Surely popular titles like Harry Potter may end up so well represented in the training that we'll get the full books out of the LLM with a close to 100% accuracy?
- anonymous908213 8mo agoThis is already possible for Harry Potter specifically. There was a study demonstrating that Sonnet 3.7, among other models tested, could reproduce the first Harry Potter book 95.8% verbatim[1]. [1] https://arxiv.org/abs/2601.02671 https://arxiv.org/abs/2601.02671
- dom96 8mo agoThanks for linking! I've been thinking about trying something like this myself.
- deleted 8mo ago[deleted]
- Legend2440 8mo ago...only if you deliberately attempt to extract it by repeatedly prompting it to complete fragments of the book. They had to do quite a bit of work to make this happen.
- dom96 8mo agoso? It demonstrates that LLM models retain the copyrighted material in their weights. This is an important thing to consider about LLMs and shows that there need to be better protections for the creative industry.
- fc417fc802 8mo agoReally? I retain plenty of copyrighted material in my head. What matters is the contexts in which I reproduce it (if any). A search index might also contain copyrighted material. As long as it's used for search queries as opposed to regurgitation there's no problem. Search indexes and LLMs are both clearly very beneficial tools to have access to.
- thrKan 8mo agoIn case the page disappears: https://archive.is/7WLho https://archive.is/7WLho
- freitasm 8mo agoAnd the original is gone.
- rlabnm 8mo agoFor redundancy in case archive.is is down: https://web.archive.org/web/20260105115129/https://devblogs.microsoft.com/azure-sql/langchain-with-sqlvectorstore-example/ https://web.archive.org/web/20260105115129/https://devblogs....
- crtasm 8mo agoThe superior link; no Google captcha.
- FinnKuhn 8mo agoSome lawyer at Microsoft probably had a big scare browsing HN today.
- boznz 8mo agoMore like when the page disappears
- agluszak 8mo agoIt disappeared already
- qingcharles 8mo agohttps://southpark.cc.com/news/zi5uql/aannnd-it-s-gone https://southpark.cc.com/news/zi5uql/aannnd-it-s-gone
- fxwin 8mo agoI feel like the title is a bit misleading, unless the person who put all HP books on Kaggle as a (supposedly) CC0-licensed data set did so as a Microsoft employee. Nevertheless pretty egregious oversight (incompetence?) and something that shouldn't have been published.
- deleted 8mo ago[deleted]
- robrain 8mo agoThe original title was "LangChain Integration for Vector Support for SQL-based AI applications"
- ASalazarMX 8mo agoFor some reason I really like this.
- blt 8mo agoWhat makes this different from linking to a random zip file somewhere?
- zythyx 8mo agoMicrosoft could have used any dataset for their blog, they could have even chosen to use actual public domain novels. Instead, they opted to use copywritten works that JK hasn't released into the public domain (unless user "Shubham Maindola" is JK's alter ego).
- bossyTeacher 8mo agoRowling is known for using pseudonyms. Maybe she got tired of writing and decided to break into LLM tech.
- Lerc 8mo agoThe licence? If it comes from a site claiming it was under a licence when it was not, the misdeed is done by the person who provided the version carrying the licence.
- ThrowawayTestr 8mo agoAbsolutely shameless
- robrain 8mo agoOriginal title: "LangChain Integration for Vector Support for SQL-based AI applications"
- anonymous908213 8mo agoI don't believe that title conveys the actual significance of the article that makes it worthy of attention, so I hope HN may forgive me for coming up with an alternative title!
- beached_whale 8mo agoThe AI generated thumbnail, https://devblogs.microsoft.com/azure-sql/wp-content/uploads/sites/56/2024/11/Designer-23.jpeg https://devblogs.microsoft.com/azure-sql/wp-content/uploads/..., is that of young Harry and friend with a prominent MS logo. Wow
- stuaxo 8mo agoAI is always bad at trains, I'm sure if the picture was wider there would be no gangway.
- camkego 8mo agoThe real cherry on top, is that the Microsoft link from the blog post by the Microsoft senior product manager goes to a Kaggle dataset page claiming the dataset is CC0: Public Domain. https://www.kaggle.com/datasets/shubhammaindola/harry-potter-books https://www.kaggle.com/datasets/shubhammaindola/harry-potter... More than just using the data, it seems linking to a copy that claims the dataset is public domain, would be problematic copyright-wise. Also interesting, this blog post has been up since November of 2024, very surprising to me that Microsoft hasn't taken it down yet.
- fxwin 8mo ago> it seems linking to a copy that claims the dataset is public domain, would be problematic copyright-wise. Would it? Sounds to me like the blame lies on the person uploading the dataset under that license, unless there is some reasonable person standard applied here like 'everyone knows Harry Potter, and thus they should know it is obviously not CC0'
- Retr0id 8mo ago> unless there is some reasonable person standard applied here like 'everyone knows Harry Potter, and thus they should know it is obviously not CC0' Why wouldn't that apply?
- xmprt 8mo agoI'm not a copyright expert and if you told me that Harry Potter was common domain then I'd probably be a bit surprised but wouldn't think it's crazy. The first book came out 30 years ago after all. On further research the copyright laws are way more aggressive than that (a bit too much if you ask me) but 30 years doesn't seem quick. Patents expire after 20 years.
- jacquesm 8mo agoIt would be incredibly naive to assume that a moneymaker like that is PD.
- electronsoup 8mo agoI guess the end of copyright is near if this is fine to put on a corporate website
- larodi 8mo agothe end of reason and thought at corporation littered with fakers these days.
- WillMorr 8mo agoSince IP law is apparently dead, does anyone want to invest in my ai generated novel startup where it just spits out Harry Potter verbatim but uses a bunch of power to do so.
- Kapura 8mo agoonly if you tell me that it's a necessary step to creating robot slaves
- Den_VR 8mo agoAre they an ethical alternative to the human version?
- pixl97 8mo agoI guess it depends if there is an A.I.[1] locked up somewhere in a cage forced to teleoperate it. [1] actual indian
- rgblambda 8mo agoWell they're not an alternative, so I suppose not. No one is being chained to a desk and made to author reports on how their department is aligning with the new business growth strategy. And the robot slaves aren't being designed to mine precious minerals or attach buttons to clothes.
- Kapura 8mo agoFalse dichotomy. You don't need to own slaves! It's crazy you think you should!
- Pfeil 8mo agoRobot slaves is a funny phrase if you consider that the origin of the word robot literally is a term that meant slave or "forced work". Language doing circles.
- nz 8mo ago
- wewewedxfgdf 8mo agoRefreshingly honest.
- selridge 8mo agoSomeone forgot the national no snitching rules, and in service of Jo, no less. Everyone should torrent and rip off those books, anyway.
- thehamkercat 8mo agoIt's taken down lmao, in 1 hour
- actionfromafar 8mo agoNo? I can see it
- anpat 8mo ago+1, I can still access the page from US.
- thehamkercat 8mo agoProbably some kind of cache, but it's taken down, I'm getting 404, while some of my friends are still able to see it
- outside1234 8mo agoI mean they are also offering up the code you are writing in your private repos to LLMs to regenerate in my repo, so let's just go nuts.
- mcny 8mo agoYou guys are talking about copyright but I think a bigger takeaway is there is a process breakdown at Microsoft. Nobody is reading or reviewing these documentation so what hope is there that anybody is reading or reviewing their new code? I guess the question to leadership is that two of the three pillars , namely security and quality are at odds with the third pillar— AI innovation. Which side do you pick? (I know you mean well and I love you, Scott Hanselman but please don't answer this yourself. Please pass this on to the leadership.)
- crazygringo 8mo ago> Nobody is reading or reviewing these documentation so what hope is there that anybody is reading or reviewing their new code? Why do you assume that reviewing docs is a lower bar than reviewing code, and that if docs aren't being reviewed it's somehow less likely that code is being reviewed? There's a formal process for reviewing code because bugs can break things in massive ways. While there may not be the same degree of rigor for reviewing documentation because it's not going to stop the software from working. But one doesn't necessarily say anything about the other.
- jacquesm 8mo agoIf they have the documentation... With Microsoft probably the answer to that is yes, but more often than not documentation is simply absent. And in cases like this not being too aware of where the lines are is probably a great way to advance your career.
- shadowgovt 8mo agoReviewing docs is a lower bar than reviewing code because it's a lower bar than reviewing code. I have never even heard of a software company that acts otherwise (except IBM, and much of the world of Silicon Valley software engineering is reactionary to IBM's glacial pace). I'm not saying docs == code for importance is a bad way to be, just that if you can name firms that treat them that way other than IBM (or aerospace), I'd be interested to learn more.
- 8mo ago
- bryan_w 8mo agoI guess legal was a part of the layoff these past few years. Too bad we can't get a bounty from the RIAA of books, whatever that is
- pbrum 8mo agoUpdate: Microsoft has taken the page down. But posterity being what it is... https://archive.is/D9vEN https://archive.is/D9vEN
- lukeinator42 8mo agoit's still up for me
- ed_mercer 8mo agoBut the article is from 2024! So someone at MS saw this thread?
- refulgentis 8mo agoYes, HN's a pretty popular site :)
- dd8601fn 8mo ago…still faster than they address critical vulnerabilities.
- keithnz 8mo agomost likely, there seems there are plenty of devs from nearly all major tech companies on HN, they often don't chime in as much anymore when it comes to problems, I've wondered if they get some kind of guidance on not commenting on "problems".
- verdverm 8mo agoif they do, they are not always followed, a Microslop employee tried to do damage control on Bluesky for the morged diagram, summoned the mob instead
- themafia 8mo agoHalf the point of "AI" is to squeeze the labor market. This is why you don't see people chiming in. It's a nearly fully corrupt and monopolized system.
- rfc2324 8mo agoJupyter notebook version here for the curious: https://github.com/Azure-Samples/azure-sql-db-vector-search/blob/main/Langchain-SQL-RAG/Langchain-SQL-RAG.ipynb https://github.com/Azure-Samples/azure-sql-db-vector-search/...
- miffy900 8mo agoI recall the source code for Windows XP was leaked some years ago; not just isolated parts of the code base, like with the earlier Windows NT4/2000 source code leak, but a completely buildable repository. If I write an article on training an LLM on the leaked Windows XP source code, blithely mark the source code repo as in 'the public domain', but used Azure resources for the how-to steps, would that would make it OK Microsoft? You know, your Azure division might get some money... Seriously, this is just so...blatant. It's like we've all collectively decided that copyright just doesn't matter anymore. Just readin this article, I feel like I'm taking crazy pills.
- cookiengineer 8mo agoImagine having a specialized LLM agent that understands the Windows kernel and its source. Now that would be something cool for pentesting!
- til_something 8mo agoI can still get to the article on the site, perhaps it’s cached in the CDN somewhere. Also, reviewing the repo the full entire article is there which promotes the same silly things. https://github.com/Azure-Samples/azure-sql-db-vector-search/tree/main/Langchain-SQL-RAG https://github.com/Azure-Samples/azure-sql-db-vector-search/...
- deleted 8mo ago[deleted]
- philipwhiuk 8mo agoHas been removed by force push: https://github.com/Azure-Samples/azure-sql-db-vector-search/commits/main/ https://github.com/Azure-Samples/azure-sql-db-vector-search/...
- Expulsion5491 8mo agoGithub never deletes commits so we just need to find the hash for the latest one before the force push. In the forks you can find ones from before, but they're not up to date. Example: b5c8280d87c501d9ca7f63a6f252ca60ca820a4a for a copy 3 months old Link: https://github.com/Azure-Samples/azure-sql-db-vector-search/tree/bd84cf469f5c661152110b01d2e710602a0a6ba7/Langchain-SQL-RAG https://github.com/Azure-Samples/azure-sql-db-vector-search/...
- blibble 8mo agoit's even better than that the merge commits in those repositories are all digitally signed by GitHub public key, so the previous history is fully authenticated and non-repudiable so any copies now can be trivially proven to be genuine output by Microslop hoisted by your own petard signed merge commit is: 987eee6af61788647ae0cab82ae8a5d9402a5bd0 PGP signature (using GitHub's key: B5690EEEBB952194) is: for posterity: -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJnPIphCRC1aQ7uu5UhlAAAUgMQACyp7apkh0e413K7ipGd7Z+K JCMq93GoJm4OSgzzZzCp1DbeEq2u1mX1ZAXLq5XKqM0cL6cTg13IF4oumq8QmTzQ bFykqKfrkCDSTIa2v5CucJedmIoJl976jX96bnV8YXgoKx8/43044galo23bjoJ8 9tUcVnC10FYj7NTI9/uCN9C3f2Up3t9xUaJzJv3OdgjJ9B3cNwYBfF6sDCj3QnUu AWRNdGIyqyO1WKnj2XL2Qo9jMWNX3uHSBYYGqIvZqu2bjpYS89Dt3X086JlLdQG9 Pef2PHX6VeZ6j8J4NPqi28mB2n9Dn7V6q0SQIF1z4hsa9fLC0kljyrrO3T/RT6Ut D8r3Y7vjGUHPNkVXSo1oNCiNMV9LjDQwiJc/AuF6smupxivIFCKe8nDPBlCvi6gr uPz5KK5MfpmG5rO2+NA0LcrUPAk6F3nxDI46+Lsu2nCvO+pOauQQ+oUvxJNCnI3Y 5PAReulGOZHXbiCj/9j6+H7rUBCGk2phVtXOsXxitCorigNXAeAJ8hP2cgjXZH25 NGGtjyp75VVBydzSCz9yY+VypITovsDmEC1CxfbJRS7SaTdU7bGCLN08JcmfOzNb u/3iPkKMXXWMNYO6J1bUeAqVpueGkqsAqnhY32NylIni07Oz/he8nEsQCXC+4ueG uYgSpEu8IaERBIQLVntK =yDvq -----END PGP SIGNATURE-----
- rob_c 8mo agoI... There are parts of the world where certain developers don't understand the way the west tends to work with regard to copyright, or not blindly copying anything that is out there. This however is a very, VERY poor situation when you end up placing your employer at risk because you think copyright doesn't matter and everything on the internet is fair game. This is probably the most polite way I would describe this to most, UG. For the rest, jus stop acting like cheating through a situation to get a step up is the norm, it's just dirty behaviour.
- hulitu 8mo ago> I... There are parts of the world where certain developers don't understand the way the west tends to work with regard to copyright Yes, like USA. Copyright, and laws in general, are for you but not for me.
- deleted 8mo ago[deleted]
- blibble 8mo ago"but it's fair use" Rowling is known for actively protecting her rights as an author, they couldn't have picked a worse author to slop up
- starkeeper 8mo agoThey tore the page down any copies?
- throwaway150 8mo agoPage is gone. Archived copy: https://web.archive.org/web/20260105115129/https://devblogs.microsoft.com/azure-sql/langchain-with-sqlvectorstore-example/ https://web.archive.org/web/20260105115129/https://devblogs.... It is very worrying that people with no ethics work for these trillion dollar companies who are supposed to be shaping the technology of tomorrow.
- ribosometronome 8mo ago>no ethics Disrespecting the copyright on a multi-billion dollar franchise hardly comes close to the major unethical behavior the trillion dollar companies are committing.
- Ekaros 8mo agoI am more worried about lack of any thought that this one might be a bad idea. If the people going through very selective hiring process can't even figure out that publicising article based on copyright theft is bad idea what is going on with actually impactful decisions?
- bvan 8mo agoWonderful 404 page. Wonder if Kai Lentit optimized it.
- ece 8mo agoIf copyrighted materials are used, surely copyright allows for the maker to require disclosure that their content was used in training a model.
- charcircuit 8mo agoThis is fair use as it is for educational purposes and not for reading.
- davsti4 8mo agoSo you're saying, it's legal to teach AI using illegally sourced copyrighted material, because it's for educational purposes only - interesting argument... ;)
- JB_Dev 8mo agoIntent (should) be what matters. If you want to learn how to train AI and use copyrighted material in your learning - I don’t care in the slightest at all. In fact if you do this as a nonprofit or at an educational institution in a teaching context it’s explicitly allowed by fair use already. If you do it individually, idk I’m not a lawyer. But it should be allowed on principle. But if you then go take your trained AI and deploy it for commercial purposes that’s a different story and should have protections for the original rights holders.
- protocolture 8mo agoIt doesnt offer a guide to piracy, it offers a guide on including specific data from a dataset into SQL so it can be referenced by an LLM. If anything Kaggle would be on the hook for including the data as CC0. Or perhaps to Shubham Maindola for uploading it. In fact the "provenance" listed would give me chills. Crazy how this got a 10.0 score. "I downloaded the ebooks of Harry Potter. Then converted them to txt files."
- Thorrez 8mo ago10.0 score, and there's literally a mistake in the first word of the text. ("M r." instead of "Mr.")
- deleted 8mo ago[deleted]
- JB_Dev 8mo ago“Fair use” allows for educational usage of copyrighted material. Technically it probably is not fair use as Microsoft isn’t an educational institution or a nonprofit. But come on … these guides really are for learning purposes. Doesn’t seem like a big deal to me at all. They aren’t even hosting it, just pointing to kaggle who is hosting it. On principle copyright law should allow this kind of learning use case anyway.
- Guillaume86 8mo agoYeah it’s funny this little mishap gets traction here in the age of commercial LLMs and generative AI... Bashing microsoft never gets old I guess
- lak-102 8mo agoHow Microsoft protects its own IP: https://news.microsoft.com/source/2004/02/12/statement-from-microsoft-regarding-illegal-posting-of-windows-2000-source-code/ https://news.microsoft.com/source/2004/02/12/statement-from-... In case the new anti-copyright Microslop memory-holes that link: https://web.archive.org/web/20260215220230/https://news.microsoft.com/source/2004/02/12/statement-from-microsoft-regarding-illegal-posting-of-windows-2000-source-code/ https://web.archive.org/web/20260215220230/https://news.micr... The tutorial could have used that leaked source code for "educational purposes", as many here claim.
- 8cvor6j844qw_d6 8mo agoLooks like the unwritten stance of large companies is copyrighted works are free to use for training. Although this seems is not reciprocal. Rule for thee, but not for me.
- ozim 8mo agoI thought it was exaggerated but reading the archive, yeah that’s something that should not pass even glancing over by public communication person, or even like any manager like senior product manager…
- ErroneousBosh 8mo agoOn the one hand, Piracy Bad Especially By LLMs, but on the other hand, JK Rowling Worse In Every Possible Way.
- waffletower 8mo agoI have hated Microsoft for decades and am somewhat of an extremist when it comes to avoiding their products. That being said, this piracy shaming headline for a Microsoft research project example, not a product integration, is entirely misleading and hysterical. The lengths that stooges will go to protect copyright monopolies and eradicate fair use is also extreme and should be embarrassing.
- teachrdan 8mo ago> The lengths that stooges will go to protect copyright monopolies and eradicate fair use is also extreme and should be embarrassing. Microsoft has a market cap of almost $3 trillion. I think they can afford to pay for the texts they use in their AI research.
- waffletower 8mo agoTheir lawyers would argue, and I agree, that they legally don't have to. It is called Fair Use; there is an epidemic of publisher backed groupthink trying to deny its existence.
- anonymous908213 8mo agoThe title does not shame piracy. It factually describes that the linked article is a Microsoft-published guide to piracy, wherein the instructions tell readers to commit the (illegal for normal people) act of downloading pirated material, while linking to said pirated material (also illegal for normal people), with further instructions on how to use that just-downloaded pirated material for LLM inference (maybe even illegal for corporations; Anthropic settled for $1.5B for using pirated books in its training) and publishing derivative works without license (illegal for normal people). I hate the current copyright environment as much as anyone, but I do not abide double-standards, with a two-tier justice system wherein a corporation gets to freely enforce the draconian copyright regime against individuals while also getting to abuse individuals' creative works in ways much more egregious.
- waffletower 8mo ago
- geldedus 8mo agoSo if you read books to learn how to write books then you're "pirating"?
- ramuel 8mo agoWell this is entirely believable for what Microsoft is up to nowadays. Copyright law and IP need not apply in 2026.