4 ms·
As someone who doesn’t know much about how these models work or are created I’d love to see some kind of breakdown that shows what % of the power of GPT4 is due
by Xenoamorphous 3y ago
As someone who doesn’t know much about how these models work or are created I’d love to see some kind of breakdown that shows what % of the power of GPT4 is due to how it’s modelled (layers or whatever) vs training data and the computing resources associated with it.
- MacsHeadroom 3y agoBy % of cost it's 99.9% compute cost and 0.01% data costs. In terms of "secret sauce" it's 95% data quality and 5% architectural choices.
- mensetmanusman 3y agoThis isn't precisely knowable now, but it might be something academics figure out years from now. Of course, first principles of 'garbage in garbage out' would put data integrity very high, the LLM code itself is supposedly not even 100k lines of code, and the HW is crazy advanced. so the ordering is probably data, HW, LLM model This also fits the general ordering of data = all human knowledge HW = integrated complexity of most technologists LLM = small team Still requires the small team to figure out what to do with the first two, but it only happened now because the HW is good enough. LLMs would have been invented by Turing and Shannon et al. almost certainly nearly 100 years ago if they had access to the first two.