3 ms·
where the line is between a "behavioral foundation model" and an LLM with behavioral pretraining. Is it the data, architecture, or something else?
by Qhx42 1y ago
where the line is between a "behavioral foundation model" and an LLM with behavioral pretraining. Is it the data, architecture, or something else?
- rgabrielsson 1y agoGreat question! It's three things: 1) Data. Every next event can be one out of billions of potential actions, so our vocabulary are in billions instead of in 100K as for LLMs 2) Architecture. Every action is multimodal, it consist of images, text, and numbers, so we need an architecture that is optimized for that. 3) Loss function. Learning efficiently on behavior where the data, downstream tasks, and bottlenecks are different requires different loss functions for it to work well
- Qhx42 1y agoIf youre using a transformer model, why reinvent the stack? LLMs adapted to code, speech, proteins, all with custom vocab, modalities, and objectives. Why is behavior a domain that requires a whole new foundation, instead of fine-tuning? What breaks if you build on existing LLM infra?