I agree with you totally. For me AGI is achieved when AutoGPT is at a point when it's able to improve its own algorithm (specifically improve on the GPT archit

I agree with you totally. For me AGI is achieved when AutoGPT is at a point when it's able to improve its own algorithm (specifically improve on the GPT architecture). Flash attention came out a bit less than a year ago (27 May 2022), which was a great scaling improvement for getting rid of O(n^2) memory requirement in the length of attention. I guess the next one we need will be one of the solutions that change the FLOPS for computing attention from n^2 to n*log(n) (there was already a paper that achieved it using resizable gauss filters where the size and the base convolution filter is learned separately). A few of these kind of ,,breakthroughs''/algorithmic improvements are the only ones needed to get to AGI (self improving machine) in my opinion.