5 ms·

Is it possible that GPT-4 already had this new optimization somewhere in its training set? The optimizing patch DeepMind published on has been floating around f

Is it possible that GPT-4 already had this new optimization somewhere in its training set? The optimizing patch DeepMind published on has been floating around for several months at this point… Edit: I found the merges for sorting (Jan 2022) [1] and hashing (Jan 2023) [2]. Both of these are very plausibly in the training set for GPT-4, which was frozen sometime in March 2023. [1] https://reviews.llvm.org/D118029 https://reviews.llvm.org/D118029 [2] https://github.com/abseil/abseil-cpp/commit/74eee2aff683cc7dcd2dbaa69b2c654596d8024e https://github.com/abseil/abseil-cpp/commit/74eee2aff683cc7d...

deleted 3y ago
[deleted]
drew-y 3y ago
Probably not. According to OpenAI, most of its data comes from before 2021: > GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its data cuts off (September 2021) https://openai.com/research/gpt-4 https://openai.com/research/gpt-4
tejohnso 3y ago
> Is it possible that GPT-4 already had this new optimization somewhere in its training set? Isn't it necessary for GPT-4 to have had the optimization somewhere in its training set given how it works?
- wilg 3y ago
  No, it's not necessary for it to have had the optimization somewhere in its training set given how it works. LLMs can solve problems they were not trained on, with varying degrees of success.