3 ms·
True, yet they have a Moore's Law like growth going for properties like their context windows.. I think the larger problem with letting them be verbose is Occam
by panflute 5mo ago
True, yet they have a Moore's Law like growth going for properties like their context windows.. I think the larger problem with letting them be verbose is Occam's razor. The more verbose they are the more variant behavior they will have where any variation that is not strictly necessary is likely to include incorrect behavior.
- deleted 5mo ago[deleted]
- pona-a 5mo agoAttention is an O(n^2) algorithm. Combined with Moore's doubling, it will at best produce linear growth (assuming Moore's law is still remotely close to alive)
- panflute 5mo agoI don't think many developers like software bloat or what it has meant for our professional reputation but we would be dishonest if we predicted a future without outcomes where ugly brute force wins given all the constraints. It is not Moore's law that dictates context window that is only an analogy and so far it has been exponential growth that went from well bellow humans to more than a human can deal with in terms of short term tasks.