I suspect that there are at least 1 or 2 more significant discoveries in terms of architecture and general way of models working, before these things become act

I suspect that there are at least 1 or 2 more significant discoveries in terms of architecture and general way of models working, before these things become actual experts. Maybe they will never get there and we will discover how to better incorporate facts and reasoning, rather than just ingesting billions of training data points.