7 ms·

I don't work at OpenAI but as a Systems Engineer it's pretty easy to conceptualize how they'd do it. - Instead of serving up one 175B parameter model, ChatGPT

I don't work at OpenAI but as a Systems Engineer it's pretty easy to conceptualize how they'd do it. - Instead of serving up one 175B parameter model, ChatGPT is probably a collection of smaller models that classify the input and then send it off to an optimized model to reduce the memory footprint for that task, such as classification, summarization, coding, and so on. - Faking the memory from the chat. Context size = compute time. If they can "compress" a conversation by summarizing previous context, it will likely reduce the tokens they need to generate for each response. It definitely feels like they are doing something like this, as the chat gets longer and the model is less able to recall context from the chat. - Model quantization: This is being heavily used in the OSS world and I'd be shocked if OpenAI weren't trying something similar. You can use quantized models without losing much accuracy and further reduce the memory footprint. - Heavy use of caching. If I were them I'd be caching like crazy. There are a lot of clever tricks with caching variants, so users don't realize they're looking at a cached response. You could take the cached response from the big LLM, then using a simple LLM to reword it so users think they're still talking to the big, smart model. - All the standard stuff in a BigCo stack (Containers, some kind of IaC, CI/CD stacks, testing environments, various types of databases or caching software, service discovery, observability tooling, etc) And probably a shitload of glue code that makes the magic happen behind the scenes.

dbish 4y ago
Very much doubt #1, the whole reason this works is model size. Smaller models and some intent model on top of it are very unlikely to produce the results they've been able to (and goes against everything they've shared on how this works). For #2, they've talked about this openly (and if you use any of the underlying models directly you can see it), there is a limited context window (~4k tokens for davinci-003), so time aside, they have to compress to fit that context window.

dbish 4y ago

Very much doubt #1, the whole reason this works is model size. Smaller models and some intent model on top of it are very unlikely to produce the results they've been able to (and goes against everything they've shared on how this works). For #2, they've talked about this openly (and if you use any of the underlying models directly you can see it), there is a limited context window (~4k tokens for davinci-003), so time aside, they have to compress to fit that context window.