3 ms·
I haven‘t run the code, but I‘m impressed by the small size. The first ELISA programs have been larger. And within the past 4 years we can fit this into bytes.
by hilti 2y ago
I haven‘t run the code, but I‘m impressed by the small size. The first ELISA programs have been larger. And within the past 4 years we can fit this into bytes.
If someone has a hint where the magic lies in, please explain it to me.
Is it the GELU function or the model that‘s
downloaded through the bash script?
- 01HNNWZ0MV43FF 2y agoMost of the magic is in the 475 MB model file downloaded through the Bash script.
- dogboat 2y agoAnd probably like 100Gb of training data or whatever :). HN had a heady debate at least once about if that training data is source or not.
- Sharlin 2y agoI guess the training data should be called assets. Analogously to how you can download the Quake source, and build it and get a working executable, but without the (non-open) assets you cannot replicate what you get by buying the game.
- fluoridation 2y agoBy that analogy, the executable code would be the game engine, the model that inference is done on would be the assets, and the training data would be... I guess the PSDs and high-poly models that get compressed and simplified to turn them into game assets.
- Sharlin 2y agoIt depends on whether the analogue of playing a game is training or inference. If the former, then the training data are the assets, the training software is the game, and the resulting model is the game experience (or if you want something more concrete, savegames and recorded play I guess). But yeah, if gameplay equals the use a trained model, then the model is an asset bundle (the pak0.pak if you will) and training data is the original models, textures etc, and the training software is all the programs that are used in the asset production pipeline.
- fluoridation 2y agoWell, consider that training is a batch processing step, while inference is the that's interactive. So training is more analogous to compilation than to gameplay.
- hilti 2y agoNot really impressed after running it. AI: How can I help you? Human: Who are you? AI: I am Alice. Human: Tell me something about a computer. AI: I am a computer model trained by OpenAI. How can I help you? Human: What is a computer? AI: I am a computer model trained by OpenAI. How can I help you? Human: Explain mathematical addition. AI: Explain mathematical multiplication. Human: 2+2 AI: 2+2 Human: Sum 2+2 AI: Sum 2+2 Human: What are your capabilities? AI: I am a computer model trained by OpenAI. How can I help you.
- Sharlin 2y agoI'm impressed that it's even that conversational, given that GPT-2 was never trained for conversations.