4 ms·

You're getting downvoted but what you said is true. The cost of self-hosting (and achieving +70 tok/sec consistently across the entire context window) has never

You're getting downvoted but what you said is true. The cost of self-hosting (and achieving +70 tok/sec consistently across the entire context window) has never been low enough to justify open source as a viable competitor to proprietary models of OpenAI, Google, and Anthropic.

grepfru_it 1y ago
I am curious the need for 70 t/sec?
- Aeolun 1y ago
  Waiting minutes for your call to succeed is too frustrating?
  - ekianjo 1y ago
    Depends entirely on the use case. Not every LLM workflow is a chatbot
    - jbellis 1y ago
      no, but if you're not latency sensitive you should probably be using DeepSeek v3 (cheaper than flash, significantly smarter)
      - lostmsu 1y ago
        What makes you believe DeepSeek is smarter than Flash 2.5? It is lower on all leaderboards.
        jbellis 1y ago
        you're right, I should clarify that I'm talking about no thinking mode, otherwise flash goes from "a bit more expensive than dsv3" to "10x more expensive"
- cootsnuck 1y ago
  High concurrency voice AI systems.
  - grepfru_it 1y ago
    Why are you self hosting that?