They're dynamically scaled and something must be broken. If you inspect source you can find the raw images, here's a few: https://www.cerebras.net/wp-content/u

They're dynamically scaled and something must be broken. If you inspect source you can find the raw images, here's a few: https://www.cerebras.net/wp-content/uploads/2023/03/Downstream-tasks-figure.png https://www.cerebras.net/wp-content/uploads/2023/03/Downstre... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-laws-blog-comparison.png https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-Laws-blog-fig-2.png https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... EDIT: Looks like it scores better with less training - up until it matches GPT-J/Pythia/OPT and doesn't appear to have much benefit. It maybe scores slightly better then GPT-J which is pretty "eh", I'm not sure if GPT-J level performance is really useful for anything? NeoX 20B outperforms it in everything if you don't care about the amount of training needed. Does the better performance for less training matter if that benefit only applies when it's only performing a lot worse then GPT-J? It appears to lose it's scaling benefits before the performance is interesting enough to matter?