Also, Google released the Reformer just 2 days ago, and they claim it can ingest orders of magnitude more data than the Transformer. https://ai.googleblog.com/

Also, Google released the Reformer just 2 days ago, and they claim it can ingest orders of magnitude more data than the Transformer. https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html?m=1 https://ai.googleblog.com/2020/01/reformer-efficient-transfo... TPUv3 is estimated to be 12 or 16nm process node, so the performance of TPU's next versions could still double over the next years (if needed). At that pace of model research and hardware improvement, I would say we are still in the AI spring.