3 ms·
In 2017 I worked tirelessly with my colleagues to implement and replicate the first transformer paper. Yesterday I left Opus 4.8 to go do some architecture res
by angusturner 4mo ago
In 2017 I worked tirelessly with my colleagues to implement and replicate the first transformer paper.
Yesterday I left Opus 4.8 to go do some architecture research, with GPU access.
It replicated and trained a credible baseline. It implemented some ideas I'd been thinking about, and wrote custom CUDA kernels for them. It read and summarised dozens of related papers.
It has since run dozens of experiments, with minimal supervision. When a model is unstable it kills it, documents why, fires off a new configuration.
The realisation that frontier labs are doing this at scale with unlimited GPU and token budgets.
It actually scares me a bit. The realisation that the next big breakthroughs will only have light human involvement.
The prospect of recursive self improvement feels more to real to me all of sudden
- Flere-Imsaho 4mo agoI'm assuming you saw this from yesterday: https://www.anthropic.com/institute/recursive-self-improvement https://www.anthropic.com/institute/recursive-self-improveme... We are at the foot of a very sharp upward trajectory.
- _zoltan_ 4mo agoI've been doing the same. take papers, define a high level goal, then let it iterate. I have access to DGX boxes and watching the model rewrite stuff to take NVLink into account after it discovered it was great :-)
- IanCal 4mo agoThis is an interesting read: https://ai-2027.com/ https://ai-2027.com/ I'm not going to say it's a perfect prediction, but I do find the trajectory of "can write something reasonable" to "oh can write snippets of code" towards larger and larger systems feels like it's played out - the common thing I see more now is that people talk of "taste" that the humans are contributing more than the raw coding part. I get what you mean with this rather automated research, I've done it on a smaller scale with performance work because it can run/test/measure/propose changes/debug and loop. I can throw a vague idea at it, guide it or discuss with it and go and make a coffee.
- lowbloodsugar 4mo agoThat was a “fun” read. Like Nick Bostrom’s Superintelligence [1]. [1] https://www.goodreads.com/book/show/20527133 https://www.goodreads.com/book/show/20527133
- adastra22 4mo agoA book that has been thoroughly discredited by actual events.
- Lplololopo 4mo agoYeah i find it very ignorant, at the current state to assume exponential growth etc. is all fantasy and everything is just hype. I think its more like driving very fast, keeping an eye very close to the road and not knowing if there is a speed limit ahead very soon or not.
- maCDzP 4mo agoThis. I managed to run Gemma 4 31B on AMD MI250X for CPT and SFT with Claude. And I have ZERO experience and knowledge of how to train and work with GPU:s. The training didn’t go where I wanted. But I manage to direct the AI to build it. It’s crazy. I am excited and scared.
- mathisfun123 4mo ago> It replicated and trained a credible baseline ... > The prospect of recursive self improvement feels more to real to me all of sudden you really don't understand why these are two completely different tasks?
- alfiedotwtf 4mo ago> The prospect of recursive self improvement feels more to real to me all of sudden The Skynet Funding Bill is passed. The system goes on-line August 4th, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug. Skynet fights back.