5 ms·
I work in a very interdisciplinary, and somewhat niche, tech/engineering field. For the past 15 years, I've been saving every relevant PDF that I can find -- m
by A_D_E_P_T 3y ago
I work in a very interdisciplinary, and somewhat niche, tech/engineering field. For the past 15 years, I've been saving every relevant PDF that I can find -- mostly studies of the sort published by Elsevier and Springer, but also books and presentations. I now have around 10k, which probably makes it the largest private library focused on this particular domain of expertise.
It has been extremely useful, especially because it's text-searchable and the really important papers are properly categorized.
A local LLM will make it 100x more useful. Also, it might not even need be "local." If I make it available via the web, I can probably sell access to other scientists and engineers in my field.
Recent advances really benefit data hoarders out there.
I'd add that these days it totally makes sense to download libgen's entire archive, because (1) storage has never been cheaper, and (2) you can use it to train local LLMs.
- hervature 3y ago> If I make it available via the web, I can probably sell access to other scientists and engineers in my field. Out of curiosity. Does this statement come from complete ignorance of or complete disregard to copyright of the author?
- throwaway11460 3y agoIf it's research it very probably is at least partially publicly funded. Regardless of whatever the law says, I don't think it's immoral to take it and offer better services around it that will be useful enough that someone decides to pay.
- hervature 3y agoDo you not see the hypocrisy in stating that someone should be able to take something partially publicly funded and profit from it while the creator of said work should not retain some rights over said profit? By extension of the transitive property of the nebulous "partially", the LLM wrapper should be provided for free with complete disregard to the wrapper's creator since it is a derivative of partially publicly funded work.
- throwaway11460 3y agoYeah, the LLM maybe - though nobody paid for the training costs in that case and that feels weird. If the public paid for a work, it should be able to use it and not be required to give away their own derivative work for free since they already paid for it through taxes. While I rather like the idea of having to provide access to derivates of publicly funded works, I fear that people would rather not use it than invest money into innovative approaches of using it. Of course if the public pays for the training and development costs, then by all means it should be available. And the library itself and the computing resources to operate it cost money that someone needs to pay. Publishers didn't pay for the research and yet they can profit from it - why this guy shouldn't?
- jddj 3y agoRightly or wrongly, Elsevier has a market capitalisation of £62B for essentially doing what GP is proposing.
- p_l 3y agoThe hoarders of IP in this case usually have done 0 work to produce it (Elsevier et al).
- walterbell 3y ago> complete ignorance of or complete disregard to copyright A question that could be appended to many LLM discussions!
- luqtas 3y agobut the LLM is no different than a human doing that & outputs COMPLETELY unique strings! - said the profiting robot parked at the gray zone law area waiting for lobyists turn it white
- znpy 3y agoOpenAI et similia seems to be doing just fine though
- thomastjeffery 3y agoI see you have encountered the first reality of "personal library". Of course, the idea of selling access to your private digital collection (or a derivative model) is relatively more absurd than the idea of monopolizing the original published work... Even so, this is as good a time as any to reconsider the practicality of copyright.
- squigz 3y agoThe data hoarder community would encourage you to release that collection for free, not try to profit from it.
- matthewmorgan 3y agoHow would you go about training an llm on 10k large pdfs?
- RetroTechie 3y ago> I'd add that these days it totally makes sense to download libgen's entire archive, because (1) storage has never been cheaper, and (2) you can use it to train local LLMs. Hardly. Data hoarding comes with most downsides of hoarding physical objects. It's just smaller, cheaper & easier to process. There's people that get rid of any physical object they haven't used in the past year (or 2, or 5, whatever). This makes sense. Imho, every object you own falls in 3 categories: 1) Things you use on a (semi?) regular basis. They make your life easier/nicer. 2) Things that are valuable. For flexible metrics of what constitutes value (sentimental, nostalgia, monetary, insurance against 'disaster', ...) Not having used something in a long time is a good hint it's not valuable. 3) Luggage. Whose value is negative. It doesn't provide anything, just takes up space, drains mental energy (and possibly other resources), and in doing so gets in the way of other pursuits. Data is no different. For any single piece of it, you either use it from time to time, somehow derive value from it, or it is useless luggage that you drag around at a cost. Apply good judgement in what to hoard.
- throwaway14356 3y agoi just keep everything ive bothered to keep, (data and stuff) i put it in a sensible place and it only consumes time when looking for it. the new external drive is so much larger than the oldest ones i put the new stuff on it and use the rest of the space to backup old drives. friends are constantly purging stuff but they seem unaware how much time and effort it takes. the significant other wanted to clean up her old photos rather than upgrade the icloud. its an insane amount of work?
- squigz 3y ago> Data hoarding comes with most downsides of hoarding physical objects. It's just smaller, cheaper & easier to process. So the same downsides, except ... much better? > There's people that get rid of any physical object they haven't used in the past year (or 2, or 5, whatever) > Data is no different. ... you either use it from time to time I suppose the real question then is what timeframe do you consider "using it from time to time"? It seems to me that this depends very much on the person, and likely on the objects themselves - probably a large appliance you haven't used in a year is likelier to be thrown away than a small item you haven't used in several. Considering data storage is smaller, cheaper, and easier to manage, I suppose a reasonable timeframe for keeping it, just in case, would be a lot longer than physical items. This of course says nothing of the societal and cultural value such archivists safeguard.
- netdevnet 3y ago> A local LLM will make it 100x more useful. Also, it might not even need be "local." If I make it available via the web, I can probably sell access to other scientists and engineers in my field. That's not legal. Just because you own some files does not mean that you own the IP of the content within.
- __MatrixMan__ 3y agoIt is, however, the right thing to do.
- squigz 3y agoProfiting off it is not the right thing to do.
- __MatrixMan__ 3y agoCorrect. Once you go communist you can't go back. Otherwise you're just a thief.
- financetechbro 3y agoThat’s OpenAI’s business model tho