Well I also don't think JEPA is a bad idea or anything. I don't think most of the alternative architectures or tweaks i've seen are bad ideas. On the contrary,

Well I also don't think JEPA is a bad idea or anything. I don't think most of the alternative architectures or tweaks i've seen are bad ideas. On the contrary, some of them seem very cool and i'm all for interesting new ideas. I guess my opinion is more on the supposed necessity of it all. I think the transformer is a powerful general learner. I think it's enough, and on the matter of intelligence, sometimes i think we make the mistake of zoning in too much on potentially spurious details. We still don't know much about Intelligence at the end of the day, what is and isn't really important, what is and isn't simply a detail of the environment and what kind of seemingly bizarre but surprisingly equivalent mechanisms can arise in the face of vastly different environments.