Sparse networks are the future, there's definitely a few major algorithmic hurdles we'll have to cross before they end up a real option but long term they will

Sparse networks are the future, there's definitely a few major algorithmic hurdles we'll have to cross before they end up a real option but long term they will dominate (after all they already do in the living world). All our current approaches rely on dense matrix multiplications. These approaches necessitate a tremendous amount of communication bandwidth (and low latency collectives). This is extremely challenging and expensive to scale O(n^2.3). The constraints of physics and finance make significantly larger models out of reach for now.