7 ms·
(1) Twitter's data is accurately timestamped, (2) there's new data constantly flowing in talking about recent events. There's no other source like that in Engli
by adtac 3y ago
(1) Twitter's data is accurately timestamped, (2) there's new data constantly flowing in talking about recent events. There's no other source like that in English other than Reddit.
AFAIU neither of those are relevant to GPT-like architectures but it's not inconceivable to think there might be a model architecture in the future that takes advantage of those. Purely from a information theoretic POV, there's non-zero bits of information in the timestamp and relative ordering of tweets.
- berkes 3y ago> there's new data constantly flowing in talking about recent events. In which the distinction between "data" and "information" is crucial. Especially now that the "floodgates" have been re-opened regarding misinformation, bots, impersonators and the likes. Data is crucial when in need of training body. But information is crucial when the training must be tuned, limited or just verified.
- mejutoco 3y agoWire news services come to mind as an alternative.
- Wowfunhappy 3y agoPerhaps, but the quantity of data is comparatively miniscule.
- mejutoco 3y agoThe quantity of information is probably higher though :)
- hef19898 3y agoAgreed, not that anyone really seems to care so.
- Wowfunhappy 3y agoOf course, but for training data current LLMs seem to need quantity above all else.
- bee_rider 3y agoI’m sure it depends on the type of prediction task, right? Current LLMs are trying to predict typical human prose from samples pulled from the internet. So it isn’t as if they are sacrificing quality for quantity. A bunch of text from the internet is a very good representation of typical human prose. Whether it is well written or the descriptions contained in the prose accurately represent, like, actual physical reality is another issue. Maybe they want to predict something with, like, less dimensionality but more utility than a paragraph of fiction.
- holler 3y agoWhat about HN?
- 0xfae 3y agoHN is a very specific demographic and probably orders of magnitude smaller in scale. Similar, but also not really comparable.
- capableweb 3y agoSmaller in scale, yeah, but probably high bias towards technical content and written by people who (mostly) care about how to write properly. There is at least 37,354,035 items that could be indexed with lots of it high quality, percentage wise probably higher quality/post than Twitter, Facebook and other sources.
- nextos 3y agoProbably HN is already part of The Pile [1]? I guess X is harder to scrap without permission. [1] https://pile.eleuther.ai https://pile.eleuther.ai
- capableweb 3y agoIt is indeed: We introduce new datasets derived from the fol- lowing sources: PubMed Central, ArXiv, GitHub, the FreeLaw Project, Stack Exchange, the US Patent and Trademark Office, PubMed, Ubuntu IRC, HackerNews, YouTube, PhilPapers, and NIH ExPorter. We also introduce OpenWebText2 and BookCorpus2, which are extensions of the original OpenWebText (Gokaslan and Cohen, 2019) and BookCorpus (Zhu et al., 2015; Kobayashi, 2018) datasets, respectively. From https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 (The Pile: An 800GB Dataset of Diverse Text for Language Modeling)
- reaperducer 3y ago...HackerNews... And there's my incentive to stop posting on HN. It's been a blast, guys. I'm going back to lurker mode.
- matchagaucho 3y agoIt could also be argued that the Twitter firehose requires substantial RLHF, de-biasing and moderation controls because of its colloquial nature.
- politician 3y agoSafety/Alignment researchers have been too fixated on making the one perfect LLM that has zero bias (or, arguably, one preferred bias) in my opinion. I don't think Musk is the type of person to make the same mistake, so we'll either end up with a Twitter LLM that accurately represents the sum total of the Twitter firehose, and/or many derivative LLMs each having a set of, possibly orthogonal, biases. Honestly, I think the later is preferable and would represent the diversity of opinions in reality more accurately. Given the data source, I think it will be important to be able to switch between LLM personalities in the future to get the "crowd truth".
- rapind 3y ago> Twitter LLM that accurately represents the sum total of the Twitter firehose, We need an xkcd showing a conversation between twitter, reddit, and hacker news based LLMs. Political rage meets memes meets pedantry.
- temporalparts 3y ago> There's no other source like that in English other than Reddit 1) Facebook Posts/Comments, 2) Instagram Posts/Comments, 3) Youtube Comments, 4) Gmail content, 5) LinkedIn Comments, 6) TikTok contents / comments X and Reddit are definitely valuable, but they're definitely not unique. I think Meta and Google have inherent advantages because their data is not accessible to LLM competitors and they have the actual capabilities to build great LLMs. Unless X decides to tap AI talent in China, they're going to have a REALLY hard time spinning up a competitive LLM team compared to OpenAI, Google, and Meta, which I think are the top three LLM companies in that order.
- emodendroket 3y agoHiring a lot of experts from China sounds extremely politically challenging.
- fjfuvucucuc 3y ago[flagged]
- emodendroket 3y agoOh give it a rest, will you? Hiring AI "also-rans" or whatever term satisfies your sense of national chauvinism then. For kicks I tried searching for news about "Chinese AI experts" and found the CCP has apparently infiltrated august institutions such as the Financial Times and Harvard Business Review (here, for instance: https://hbr.org/2021/02/is-china-emerging-as-the-global-leader-in-ai https://hbr.org/2021/02/is-china-emerging-as-the-global-lead...). Maybe you can go pester them.
- thefurdrake 3y agoI am 100% certain that there is at least a single AI expert in China. This is bait, right?
- fjfuvucucuc 3y ago
- peter422 3y agoJust take an aggregate of every news article written in the top 100 newspapers. That's high quality content, timestamped and about current events. There is very little content on Twitter that compared in quality to one will written news article.
- daem 3y agoThere is information posted to X which the top 100 newspapers are not willing to / do not care to publish.
- reaperducer 3y agoAs we've learned in recent months, it goes both ways.
- throwaway4aday 3y agoI would assume this is simply not enough data. You also have access to all public domain books and they usually make up a small fraction of the data used to train a model from scratch. For fine tuning a model, having a unique source of high quality data is probably valuable even if small.
- deleted 3y ago[deleted]