4 ms·
According to the press release, "we achieved an impressive Time-to-First-Token of approximately 19 seconds for a gemma3:4b model" Imagine, you have a very smal
by rvnx 1y ago
According to the press release, "we achieved an impressive Time-to-First-Token of approximately 19 seconds for a gemma3:4b model"
Imagine, you have a very small weak model, and you have to wait 20 seconds for your request.
- infecto 1y agoImagine running a production client facing api and not overprovisioning it.
- happyopossum 1y ago> Imagine, you have a very small weak model, and you have to wait 20 seconds for your request. For your first request, after having scaled to 0 while it wasn’t in use. For a lot of use cases, that sounds great.
- steren 1y agoAlso, a GPU instance needs 5s to start. The test depends on how large the model is. So a "very small weak model" can lead much faster than 20s