5 ms·

> The performance is abysmal after ~32k. Huh. We've not seen this in real-world use. 2.5 pro has been the only model where you can throw a bunch of docs into i

> The performance is abysmal after ~32k. Huh. We've not seen this in real-world use. 2.5 pro has been the only model where you can throw a bunch of docs into it, give it a "template" document (report, proposal, etc), even some other-project-example stuff, and tell it to gather all relevant context from each file and produce "template", and it does surprisingly well. Couldn't reproduce this with any other top tier model, at this level of quality.

spmurrayzzz 1y ago
We're a G-suite shop so I set aside a ton of time trying to get 2.5 pro to work for us. I'm not entirely unhappy with it, its a highly capable model, but the long context implosion significantly limits it for the majority of task domains. We have long context evals using internal data that are leveraged for this (modeled after longproc specifically) and the performance across the board is pretty bad. Task-wise for us, it's about as real world as it gets, using production data. Summarization, Q&A, coding, reasoning, etc. But I think this is where the in-distribution vs out-of-distribution distinction really carries weight. If the model has seen more instances of your token sequences in training and thus has more stable semantic representations of them in latent space, it would make sense that it would perform better on average. In my case, the public evals align very closely with performance on internal enterprise data. They both tank pretty hard. Notably, this is true for all models after a certain context cliff. The flagship frontier models predictably do the best.
ttul 1y ago
Are you by any chance a lawyer? I’m asking because I’m genuinely curious whether lawyers are starting to use the SOTA LLMs in day-to-day drafting and review work. I use the LLMs as a CEO as a poor substitute for my in-house counsel when I just need _an_ answer quickly (i.e. when counsel is literally asleep); however, for anything serious, I always defer to them because I know LLMs make mistakes and obviously cannot offer professional liability cover.

spmurrayzzz 1y ago

We're a G-suite shop so I set aside a ton of time trying to get 2.5 pro to work for us. I'm not entirely unhappy with it, its a highly capable model, but the long context implosion significantly limits it for the majority of task domains. We have long context evals using internal data that are leveraged for this (modeled after longproc specifically) and the performance across the board is pretty bad. Task-wise for us, it's about as real world as it gets, using production data. Summarization, Q&A, coding, reasoning, etc. But I think this is where the in-distribution vs out-of-distribution distinction really carries weight. If the model has seen more instances of your token sequences in training and thus has more stable semantic representations of them in latent space, it would make sense that it would perform better on average. In my case, the public evals align very closely with performance on internal enterprise data. They both tank pretty hard. Notably, this is true for all models after a certain context cliff. The flagship frontier models predictably do the best.

ttul 1y ago

Are you by any chance a lawyer? I’m asking because I’m genuinely curious whether lawyers are starting to use the SOTA LLMs in day-to-day drafting and review work. I use the LLMs as a CEO as a poor substitute for my in-house counsel when I just need _an_ answer quickly (i.e. when counsel is literally asleep); however, for anything serious, I always defer to them because I know LLMs make mistakes and obviously cannot offer professional liability cover.