4 ms·
It’s possible that they could be using fallback models during peak load times (west coast mid day). I assume your traffic would be routed to an east coast data
by janalsncm 10mo ago
It’s possible that they could be using fallback models during peak load times (west coast mid day). I assume your traffic would be routed to an east coast data center though. But secretly routing traffic to a worse model is a bit shady so I’d want some concrete numbers to quantify worse performance.
- dcre 10mo agoTo be clear, the company has very directly denied doing this.
- denysvitali 10mo agoThey did yes, but should we trust them? I remember clearly this problem happening in the past, despite their claims. I initially thought it was an elaborate hoax, but it turned out to be factually true in my case.
- dcre 10mo agoI tend to think it would be very hard and very risky for large, successful companies to systematically lie about these things without getting caught, and the people who would be doing the lying in this case are not professional liars, they’re engineers who generally seem trustworthy. So yes, if there is a degradation, I think bugs are much more likely than systematic lying.
- cma 10mo agoThe TPU implementation used approximate top-k instead of the exact used on nvidia. While that wouldn't matter too much and there was a bug with it, it still was a cost savings thing not to use exact from the beginning because it wasn't efficient on TPUs which they were routing to under load. So it was a bit of a model difference under load, even aside from the bug.
- dcre 10mo agoTo the extent this is an accurate characterization (somewhat, I think), they considered the quality difference a bug and fixed it!