3 ms·

It would be understandable that they are focused currently on inference speed, but features like structured output and prompt caching make it possible to build

by brylie 2y ago

It would be understandable that they are focused currently on inference speed, but features like structured output and prompt caching make it possible to build more capable LLM applications. Does Cerebras support reliable structured output like the recent OpenAI 4o?

bigcat12345678 2y ago
This question is a bit out of context. Cerebras is a startup producing innovative AI chips. Their chips are super cool, and I personally believe Cerebras is ahead of the industry and is on the right technical path. As a matter of fact, Cerebras started with HPC chips. Then pivoted to AI like everyone else. They are still deep in the trench for survival. Given that, they have very little software prowess compared to AMD (which has *terrible* software stack for AI GPUs look at https://github.com/ROCm/rdc https://github.com/ROCm/rdc, an equivalent to NVIDIA DCGM, which virtually has no maintainer, and no one is using it), NVIDIA (the golden standard of software stack for AI GPUs); and you are referring to structured output and prompt caching which are prominently developed by LLM research institutions (OpenAI Anthropic, each of which have way more funding than Cerebras) In the end, educate yourself, and do not put unrealistic expectation on startups.
- sanxiyn 2y ago
  As OpenAI themselves admits, structured output feature in question was developed in open source world first with zero funding.
  - bigcat12345678 2y ago
    This point is moot. The point remain that Cerebras is not in a position to focus on structured output or prompt caching.
noident 2y ago
They are running stock Llama 3.x. If the underlying models support structured output, so will they. For example, I know the latest batch of Mistral models all have json output support.
- throwawaymaths 2y ago
  Structured output is a token picker feature, not (just) a model feature.

by brylie 2y ago

bigcat12345678 2y ago

This question is a bit out of context. Cerebras is a startup producing innovative AI chips. Their chips are super cool, and I personally believe Cerebras is ahead of the industry and is on the right technical path. As a matter of fact, Cerebras started with HPC chips. Then pivoted to AI like everyone else. They are still deep in the trench for survival. Given that, they have very little software prowess compared to AMD (which has *terrible* software stack for AI GPUs look at https://github.com/ROCm/rdc https://github.com/ROCm/rdc, an equivalent to NVIDIA DCGM, which virtually has no maintainer, and no one is using it), NVIDIA (the golden standard of software stack for AI GPUs); and you are referring to structured output and prompt caching which are prominently developed by LLM research institutions (OpenAI Anthropic, each of which have way more funding than Cerebras) In the end, educate yourself, and do not put unrealistic expectation on startups.

sanxiyn 2y ago
As OpenAI themselves admits, structured output feature in question was developed in open source world first with zero funding.
- bigcat12345678 2y ago
  This point is moot. The point remain that Cerebras is not in a position to focus on structured output or prompt caching.

noident 2y ago

They are running stock Llama 3.x. If the underlying models support structured output, so will they. For example, I know the latest batch of Mistral models all have json output support.

throwawaymaths 2y ago
Structured output is a token picker feature, not (just) a model feature.