4 ms·
LLMs are comprised of just three elements Data Compute Algorithms All three are just scratching the surface of what is possible. Data: What has been scrape
by imranq 3y ago
LLMs are comprised of just three elements
Data
Compute
Algorithms
All three are just scratching the surface of what is possible.
Data: What has been scraped off the internet is just <0.001% of human knowledge as most platforms cannot be scraped so easily, are in formats that are not in text like video, audio, or just plain old pieces of paper undigitized. Finally there are probably techniques to increase data through synthetic means, which is purportedly OpenAI's secret sauce to GPT-4's quality.
Compute: While 3nm processes are approaching an atomic limit (0.21nm for Si), there is still room to explore more densely packed transistors or other materials like Gallium Nitride or optical computing. Not only that but there is a lot of room in hardware architecture to allow more parallelism and 3-D stacked transistors.
Algorithms: The transformer and other attention mechanisms have several sub-optimal components to them like how arbitrary the Transformer is in terms of design decisions, and quadratic time complexity for attention. There also seems to be a large space of LLM augmentations like RLHF for instruction following and improvements in factuality and other mechanisms.
And these ideas are just from my own limited experience. So I think its fair to say that LLMs have plenty of room to improve.
- imtringued 3y agoI'm sorry but you can have process nodes smaller than an atom. The size of atoms is irrelevant here. The process node refers to what dimensions a theoretical planar transistor would have to be equivalent to the current 3D transistors. If you stack multiple transistors on top of one another, the process node gets smaller regardless of what you think.
- discreteevent 3y ago> LLMs are comprised of just three elements > Data > Compute > Algorithms Not to be facetious but so is all other software. LLMs appear to scale in correlation to the first two but it's not clear what the correlation is and that's the basis of the question being asked.
- deleted 3y ago[deleted]
- joecool1029 3y agoFor data though, as LLM's generate more output, over time wouldn't they be expected to mess themselves with their own generated data? Wouldn't that be the wall we'll hit? Think of how shitted up Google Search is with generated garbage, I'm imagining we're already in the 'golden age' where we were able to train on good datasets before it gets 'polluted' with LLM generated data that may not be accurate, and it just continues to become less accurate over time.
- edmundsauto 3y agoThere must be signals in the data about generated garbage, otherwise humans wouldn't be able to tell. Something like PageRank would be a game changer and potentially solve this issue.
- throwaway4aday 3y agoCleaning and preparing the dataset is a huge part of training. Like the OP mentioned, OpenAI likely have some high quality automation for doing this and that's what's given them a leg up above all other competitors. You can apply the same automation to clear out low quality AI content the same way you remove low quality human content. It's not about the source, just the quality matters.
- quickthrower2 3y agoWe need models that need less language data to train. Babies learn to talk on way less data than the entire internet. We need something closer to human experience. Kids have a feel for what is bullshit before they have consumed the entire internet :-). I think feeding the internet into a LLM will be seen as the mainframe days of AI.
- Volundr 3y ago> Babies learn to talk on way less data than the entire internet. Is this actually true? My gut check says yes, but I'm also unaware of any meaningful way to actually quantify the volume of sensor data processed by a baby (or anyone else for that matter), and it wouldn't shock me to discover if we could we'd find it to be a huge volume.