5 ms·
The real moat is an abundance of high quality data.
by theLiminator 3y ago
The real moat is an abundance of high quality data.
- classified 3y ago... stolen without regard for copyright and licensing.
- pas 3y agoFair use!? /s
- quickthrower2 3y agoIMO the real moat right now is expertise / smart teams and cash.
- hospitalJail 3y agoThe infrastructure/training libraries already exists. I'm sure you can get people who worked at scale that can figure out how to glue things together. Reddit, twitter, etc.. raising prices is going to make it more expensive.
- quickthrower2 3y agoIf you are right then it just becomes who wants to throw the most cash in like a giant game of poker but where you don’t know the pot odds.
- JimmyRuska 3y agoWell open AI raised eye brows by crawling the internet and using everyone's data to make a commercial product One day some new startup will train on all of libgen and torrent networks, but it will be very hard to prove. You'll keep getting these gaps up in questionable morality and legality, and even openai will complain about playing fair
- why_only_15 3y agoMany people train on libgen/torrent in the form of books3 (e.g. LLaMa does this).
- fragmede 3y agoGoogle Classroom, teenager's essays, written by humans, for learning what it means to be human, and graded by humans, is a richer dataset than anything else I can think of that anyone else couldn't get their hands on.
- londons_explore 3y agoAn awful lot of teachers can grade a 10 page essay in about 90 seconds... Skim read it, mark out some grammar errors, assign it a grade based on the quality of the opening and closing paragraphs.
- fragmede 3y agoYup, and they're doing it the whole country over, and putting that data in to Google Classrooms for Bard to know "this is C-grade work" and "this is A-grade work". Knowing what's deemed good and bad writing is where I'm thinking this dataset shines for training LLMs.
- pas 3y agoThePile already contains some content from a torrent, and there's as lawsuit alleging that Meta has committed copyright infringement by using it. https://www.theverge.com/2023/7/9/23788741/sarah-silverman-openai-meta-chatgpt-llama-copyright-infringement-chatbots-artificial-intelligence-ai https://www.theverge.com/2023/7/9/23788741/sarah-silverman-o...
- baq 3y agoYeah they have the internet from before LLMs were used for anything, so the data is not poisoned. Not unlike carbon dating becoming useless for estimating age of anything made after nuclear atmospheric tests, or low-background steel.
- joiqj 3y agoYou talk as if humans weren't perfectly capable of coming up with nonsense. Blogs upon blogs full of worthless pap that is there for SEO reasons have existed for like a decade already.
- pixl97 3y agoAnd those blogs took a decade+ to make, and now in another year we'll make that much information again. Then it will be that much information in a month. Then that much pap in a day. And in the past it was still a million people making that much crap. Now it's a single "entity" making that much crap with it's own style and mistakes.