5 ms·
Anna's Archive – LLM Training Data from Shadow Libraries
- TotalCrackpot 3y agoI posted this earlier! https://news.ycombinator.com/item?id=37928320 https://news.ycombinator.com/item?id=37928320
- nanna 3y agoNo way!?
- deleted 3y ago[deleted]
- pitched 3y agoI’ve tried to dig a bit through the libgen fiction archive. Surprising to me, I found that the vast majority was Romance novels. I’ve since heard this from others too that most published books are Romance. If this collection is trying to solve the problem of having large tagged and sorted data for training, even if it can’t be commercialized, it might also be introducing another where the data still needs to be weighted and filtered. Not to knock on Romance specifically but even training on all scientific papers ever written, whether peer-reviewed, highly cited, retracted, debunked, etc. might explain some hallucinations it ends up making.
- kelsey9876543 3y agohumans daydream about romance, why can't computers?
- PrayagBhakar 3y agoI don’t think it’s as much about day dreaming but more so about garbage in, garbage out. We know LLMs perform way better when you use high quality input data [0]. Having your input data be skewed towards one genre isn’t great. And thats not even getting into the quality of these books. [0] https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644
- GTP 3y agoHumans don't daydream about dating a computer, why should a computer daydream about dating a human? ;)
- deleted 3y ago[deleted]
- candiodari 3y agoBecause that will sell.
- jhbadger 3y agoAnd not just romance in the mainstream Harlequin mode -- there's huge amounts of basically text porn there which shows up often in searches for something else. I didn't know this was still being written -- I thought film and Internet video had killed off the "dirty book" decades ago.
- dacryn 3y agoif anything internet made textual dirty stories explode as well. It's just a different target audience. Especially women don't like the highly visual stuff. They tend to favor a relaxing text.
- schleck8 3y agoThese are called smut now and most of the readerbase is female. I believe it's often regarded a more "socially-aligned"/less hardcore porn substitute. https://www.urbandictionary.com/define.php?term=Smut https://www.urbandictionary.com/define.php?term=Smut
- globular-toast 3y agoHow did you miss Fifty Shades of Grey?
- artninja1988 3y agoGod bless you Anna. Also, how do we make sure this/ mirrors of it can't be taken down?
- all2 3y agoThey actually have a "premium" membership you can subscribe to.
- politelemon 3y agoFirst time I've encountered this Anna's Archive. Is this just one person doing all the work or a group? It seems similar to libgen, actually it mentions mirroring libgen. Doesn't it get targeted by publishing company lawyers and such?
- seanthemon 3y agoI highly recommend giving his/her blog post [0] a read. [0] https://annas-blog.org/how-to-run-a-shadow-library.html https://annas-blog.org/how-to-run-a-shadow-library.html
- politelemon 3y agoFantastic read thanks for sharing that
- spokeonawheel 3y ago[dead]
- pilimi_anna 3y agoPlease seed our torrents! If someone could contribute an automated torrent tracker that shows which torrents are most in need of seeding, we'd incredibly appreciate that (all our code is open source).
- 0x49d1 3y agoJust went and donated them. Nice to have such products: science research papers are always under paywalls, but sometimes just "ordinary" people want to read that to wide their scope and knowledge. Knowledge should not be paywalled like that. Also there are some situations sometimes that some books are just not available in some countries. In that cases such products can be used I think. In other cases - sure we need to support authors(sometimes publishers/resellers) directly if we can.
- pilimi_anna 3y agoThank you!! Really appreciate it.