3 ms·
Another interesting potential market here will be 'LLM in a box'. All the hardware and other tooling in a prebuilt, but modular, package ready to go. Pay one up
by somenameforme 2mo ago
Another interesting potential market here will be 'LLM in a box'. All the hardware and other tooling in a prebuilt, but modular, package ready to go. Pay one up-front cost, get a system running [whatever open LLM] with a token rate of [x], optionally configured to be immediately ready for distributed usage. Basically the opposite of cloud stuff: no rent, no dependency, 100% guaranteed uptime, guaranteed security/privacy (at least subject to your own actions), and so on.
- jurgenburgen 2mo ago> 100% guaranteed uptime Disagree there but I think this is an interesting idea. We would need to find some more cost-efficient hardware to run it on than Nvidia GPUs.
- pulse7 2mo agoIt will come... all big hardware players (Intel, AMD, Broadcom) and dozens of startups (Tenstorrent, etc.) are working on it...
- Godsend69 2mo ago[dead]
- bevekspldnw 2mo ago“100% guaranteed downtime when you least can afford it and the support tickets are your problem.” We’ve a hybrid shop, including hosting our own ML infra, and we save a ton from cloud spend with local ML. Easily one million USD over past three years. But it’s not “free”, you are shifting a lot of labor into your plate.
- hypfer 2mo agoAnd with that also gain institutional knowledge, skill up your workers and attract talent that wants to work on this stuff. All boils down to short-term/long-term thinking.
- gruturo 2mo agoThis. People WANT to work on this stuff. And having skilled workers is a precious advantage.
- bevekspldnw 2mo agoStill has to break even on the balance sheet, especially at a bootstrapped startup. We actually made most of the financial windfall in translation API fees oddly enough. For our own model training we needed to do some large scale translation tasks of a large dataset (1M or so documents, 10 or so target languages), running full-size NLLB on-prem saved us an absurd amount of money vs Google Translate API. (For reference doing 1M target docs into a single language in Google Translate API is roughly $120k list price. You can run full size NLLB on an 48GB NVIDIA A600 and the major difference for us was speed, but for this task time to completion wasn’t an issue.)
- adrian_b 2mo agoPalantir already offers a "turnkey AI datacenter", i.e. a rack with "NVIDIA Blackwell Ultra systems with eight NVIDIA Blackwell Ultra GPUs and NVIDIA Spectrum-X™ Ethernet networking for AI training and inference". It is said that it comes with all hardware and software required to run inference or training with an open weights LLM. The existence of this product, which competes with cloud-based offerings like those of OpenAI and Anthropic, is presumably the reason why the Palantir CEO criticized very harshly some time ago the business model of OpenAI/Anthropic. While I doubt that the ethics of Palantir is any better than of OpenAI/Anthropic, in this particular case I have to agree with Alex Karp about "Sovereign AI", i.e. that only losers will make their business completely dependent on an external entity like OpenAI or Anthropic, who are certainly not trustworthy.
- vrganj 2mo agoI'm not sure a data center run by ... Palantir of all organizations is what people have in mind when they worry about data sovereignty.
- adrian_b 2mo agoThey are selling it, not running it. It is just a dedicated computer system, which should be managed by its owner, like any other on-prem servers. I doubt that it has a good price/performance ratio, but it is a solution for those who feel that they do not want to search, buy, assemble, install and configure every HW/SW component.
- numpad0 2mo agoWhat makes that kinda complicated is that multi-user throughput of LLMs scale well but single-user performance often stays constant at low ends. If you could saturate e.g. 16 concurrent session-month of demand, you can just go buy 16 of 32GB GPUs and start charging monthly for inference. That could work if you had e.g. over thousand total employees with hundreds of devs eager to trying it out, but only if the company is also interested in a private inference experiment.
- zozbot234 2mo agoYou're talking about multi-session vs. single-session throughput. A single user can easily leverage multiple sessions via e.g. subagent swarms, especially on a lower-end setup where any single session is going to be quite slow. Saturating utilization during off-hours is harder but potentially quite feasible by assigning lower priority, unattended tasks/inference loops.
- jnwatson 2mo agoI see ads for this all the time.
- kelvinjps10 2mo agoso something like this? https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox
- codethief 2mo agoExactly! As I've argued here on HN before, such an "LLM in a box" might end up being serviced/upgraded once or twice a year by a company very similar to the one servicing the coffee machine at the office. In contrast to databases, storage, etc. it doesn't matter much if the box breaks at some point – they'll just come by and replace it with a new one – and there's barely any software on the box to speak of, at least none that requires continuous development and feature upgrades, beyond rolling out security patches. This makes the business case drastically different from cloud and SaaS offerings, where most of the moat is in the software and the state maintenance (and the vendor lock-in of course). The LLM in a box is destined to become a commodity.