3 ms·
I personally find approaches like this the correct way forward. An input analyzer that finds out what kinds of tokens the query contains. A bunch of specialize
by nine_k 3y ago
I personally find approaches like this the correct way forward.
An input analyzer that finds out what kinds of tokens the query contains. A bunch of specialized models which handle each type well: image analysis, OCR, math and formal logic, data lookup,sentiment analysis, etc. Then some synthesis steps that produce a coherent answer in the right format.
- michaelt 3y agoThen you might enjoy looking up the "Mixture of Experts" model design.
- numeri 3y agoThat has nothing to do with the idea of ensembling multiple specialized/single-purpose models. Mixture of Experts is an method of splitting the feed-forwards in a model such that only a (hopefully) relevant subset of parameters is run for each token. The model learns how to split them on its own, and usually splits based not on topic or domain, but on grammatical function or category of symbol (e.g., punctuation, counting words, conjunctions, proper nouns, etc.).
- michaelt 3y agoAn ensemble of specialists is different to a mixture of experts? I thought half the point of MoE was to make the training tractable by allowing the different experts to be trained independently?
- CuriouslyC 3y agoYeah. Have a multimodal parser model that can decompose prompts into pieces, generate embeddings for each of them and route those embeddings to the correct model based on the location of the embedding in latent space. Then have a "combiner/resolver" model that is trained to take answer embeddings from multiple models and render it in one of a variety of human readable formats. Eventually there is going to be a model catalog that describes model inputs/outputs in a machine parseable format, all models will use a unified interface (embedding in -> embedding out, with adapters for different latent spaces), and we will have "agent" models designed to be rapidly fine tuned in an online manner that act as glue between all these different models.
- hackerlight 3y agoDoesn't the human brain work like this? Yeah it's all connected together and plastic and so on, but functions tend to be localized, e.g vision is in occipital area. These base areas are responsible for the basic latent representations (edge detectors) which get fed forward to the AGI module (prefrontal cortex) that coordinates the whole thing based on the high quality representations it sees from these base modules. This strikes me as the most compute efficient approach.