5 ms·
I have a ML infra that does between 1 million and 2 million inferences per day (granted it is not a chat bot or code generator, but it is legit ML inference in
by mhuffman 4y ago
I have a ML infra that does between 1 million and 2 million inferences per day (granted it is not a chat bot or code generator, but it is legit ML inference in a similar mode) and it only cost $120 per month for the server (also runs multiple containers and is a web server), which mostly sits idle.
- swyx 4y agoright, but what size of model? i assume these large language models are... well, large