34 ms·
It would be interesting to see how hard it would be to walk these models towards general relativity and quantum mechanics. Einstein’s paper “On the Electrodyna
by anotherpaulg 11mo ago
It would be interesting to see how hard it would be to walk these models towards general relativity and quantum mechanics.
Einstein’s paper “On the Electrodynamics of Moving Bodies” with special relativity was published in 1905. His work on general relativity was published 10 years later in 1915. The earliest knowledge cuttoff of these models is 1913, in between the relativity papers.
The knowledge cutoffs are also right in the middle of the early days of quantum mechanics, as various idiosyncratic experimental results were being rolled up into a coherent theory.
- deleted 11mo ago[deleted]
- ghurtado 11mo ago> It would be interesting to see how hard it would be to walk these models towards general relativity and quantum mechanics. Definitely. Even more interesting could be seeing them fall into the same trappings of quackery, and come up with things like over the counter lobotomies and colloidal silver. On a totally different note, this could be very valuable for writing period accurate books and screenplays, games, etc ...
- danielbln 11mo agoAccurate-ish, let's not forget their tendency to hallucinate.
- mlinksva 11mo agoDifferent cutoff but similar question thrown out in https://www.dwarkesh.com/p/thoughts-on-sutton#:~:text=If%20you%20trained%20an%20LLM%20on%20the%20data%20from%201900%2C%20it%20wouldn%E2%80%99t%20be%20able%20to%20come%20up%20with%20relativity%20from%20scratch https://www.dwarkesh.com/p/thoughts-on-sutton#:~:text=If%20y... inspiring https://manifold.markets/MikeLinksvayer/llm-trained-on-data-from-1900-comes https://manifold.markets/MikeLinksvayer/llm-trained-on-data-...
- machinationu 11mo agothe issue is there is very little text before the internet, so not enough historical tokens to train a really big model
- tgv 11mo agoI think not everyone in this thread understands that. Someone wrote "It's a time machine", followed up by "Imagine having a conversation with Aristotle."
- concinds 11mo agoAnd it's a 4B model. I worry that nontechnical users will dramatically overestimate its accuracy and underestimate hallucinations, which makes me wonder how it could really be useful for academic research.
- DGoettlich 11mo agovalid point. its more of a stepping stone towards larger models. we're figuring out what the best way to do this is before scaling up.
- spicyusername 11mo agoIf there's very little text before the internet, what would scaling up look like?
- lm28469 11mo ago> the issue is there is very little text before the internet, Hm there is a lot of text from before the internet, but most of it is not on internet. There is a weird gap in some circles because of that, people are rediscovering work from pre 1980s researchers that only exist in books that have never been re-edited and that virtually no one knows about.
- throwup238 11mo agoThere is no doubt trillions of tokens of general communication in all kinds of languages tucked away in national archives and private collections. The National Archives of Spain alone have 350 million pages of documents going back to the 15th century, ranging from correspondence to testimony to charts and maps, but only 10% of it is digitized and a much smaller fraction is transcribed. Hopefully with how good LLMs are getting they can accelerate the transcription process and open up all of our historical documents as a huge historical LLM dataset.