3 ms·
And hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for
by 999900000999 3mo ago
And hire 2 or 3 dev ops to keep it running ?
That another 400 to 700k.
It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
- clint 3mo agoJust let it manage itself, what could go wrong! :)
- russell_h 3mo ago> However, I don’t trust hosted LLMs for anything that needs to be private. Why not? Do you trust AWS with things that need to be private?
- Kevcmk 3mo agoMore than I trust frontier labs. AWS doesn't need to recoup 9 digits USD of capex
- senderista 3mo agoSo you can just use Bedrock?
- retinaros 3mo agoaws its their business to make ur data safe. frontier lab business is to use your data and mine to train models
- wongarsu 3mo agoWhere do I sign up to get 200k/yr to keep one rack running? Sounds like an incredibly chill job
- arjie 3mo agoApparently it’s going to take the 3 of us to do this, mate. Going to get so much reading done.
- LeonM 3mo agoWhat you get is not what you cost. 40% overhead is quite typical, so you'd be looking at $120k/year. In the USA I'd consider that a competitive salary for an admin capable or keeping a $6M rack of specialized hardware running 24/7.
- margalabargala 3mo agoYeah but you don't need two such people, or even one, dedicated to this single rack. A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.
- 999900000999 3mo agoWho is going to upgrade the models ? Who is going to fix it when the api does something weird ? Who is going to proactively make sure it’s not overheating? Chat GPT has enterprise contracts for a reason.
- deleted 3mo ago[deleted]
- stymaar 3mo agoAll of the answers to your questions above are in gp's comment already: > A company of the size that this is worthwhile for, probably has dedicated devops on staff already
- margalabargala 3mo agoThe same person managing the company's email accounts and whatnot. I didn't say it's fire-and-forget. I'm saying all that is maybe a day of work every 3 months.
- DiskoHexyl 3mo agoIt doesn’t really work like that. The companies which have the will and the budget to host their own LLMs typically require a ton of other, much smaller, models as well. They have internal security requirements, guardrails, audit, critical workflows start depending on your onprem setup, downtime is now something that’s not even allowed. There’s going to be a zoo of tooling, lots of bespoke work with internal clients who have no clue about docker, but now want their vibe-coded app to access a model they downloaded yesterday, and this model better be served and monitored 24/7 because now C-levels use it. No one is going to budget several millions to then look at an email admin who hears ‘cuda’ for the first time in their life and ask them to just support the entire thing somehow. And god forbid it’s an AMD setup. This obviously isn’t relevant for a 10-person startup and their second-hand xeon with a single H100. From my experience only the companies which are REALLY interested in privacy and data security bother with hosting their own models
- GodelNumbering 3mo ago> And hire 2 or 3 dev ops to keep it running Not a devops but I'd say one full time is already too many.
- dboreham 3mo agoYes but zero is not enough and where do you get a fraction of a competent dev op from?
- layer8 3mo agoFrom the other dev-op work you’re doing.
- ayewo 3mo agoUnderstood but sharing your existing devops resources with this will soon become a bottleneck especially when any major downtime will keep several engineers (and long-running agents) blocked from any meaningful work until availability improves.
- layer8 3mo agoNot my experience, from an SMB that maintains its own hardware and services. You have a certain contingent of competent engineers who distribute their work across projects, and it generally works out fine. Or course you plan with some redundancy and fall-back plans in your systems.
- ayewo 3mo agoThe top poster mentioned LLM spend of millions/month to justify the estimated capex of $6m to self-host Kimi on own infra. Add to this number another $1.5m/yr in opex, so not sure I’d call such an enterprise wealthy enough to spend those kinds of sums on LLMs an “SMB”.
- JacobAsmuth 3mo agoNow you're thinking like management!
- lumost 3mo agoThere will be cloud/SaaS vendors who have lower cost of labor/capital due to automation and financing terms. Having these models in the open caps the inference margin.
- slicktux 3mo agoJust like that new jobs created by AI! Localized model maintainer/technician.
- toomuchtodo 3mo agoYou’ll slap some training on existing technologists/infra/sysadmin folks and perhaps have a support contract for the edge cases (hardware troubleshooting and advanced replacement). (managed an entire data center building with thousands of servers a lifetime ago with ~2-3 other people, it’s only gotten easier over the last two decades imho)
- shrubble 3mo agoAs mentioned, it is a "large enough company" already; they have full time sysadmins running things. Adding another rack beside the VMWare cluster, managing any storage/networking issues, etc. will be incremental costs; they already have a pager (probably not a pager not anymore just an app on their phone) like rotation schedule etc.
- w0m 3mo ago> I don’t trust hosted LLMs for anything that needs to be private I'd update this to 'I don’t LLMs for anything that needs to be private' What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.
- mdp2021 3mo ago> What's to prevent the LLM from A NN per se is a file... The executable that runs it can "act"...
- JacobAsmuth 3mo agoWhy do you feel that creating the LLM from scratch is sufficient to trust it? Are you suggesting that you personally would read all 15 trillion tokens (plus every single agentic trade used in RL, along with its relative advantage in the batch) and personally guarantee that gradient descent would train a model which would not exfiltrate your corporate data? Or that perhaps you have a perfect alignment algorithm which you are unwilling to share with the broader research community (evil)?
- w0m 3mo ago> Why do you feel that creating the LLM from scratch is sufficient to trust it If you're doing the training yourself, you at least have a verifiable supply chain and an audit trail. Today, we have no idea if a black-box model handed to us is coded to recognize specific domains or patterns and back door an application in a sneakily targeted way. Black box is a black box. Many open-weight models clearly haven't been trained or created in the manner their creators claim---which raises the obvious question: if they lied about the recipe, what else did they lie about? Putting those models in a production capacity scares the living crap out of me. That said, building from scratch isn't about achieving mathematical perfection or manually auditing 15 trillion tokens---that's impossible. It's about eliminating third-party supply chain risk and having actual governance over the pipeline. Of course, that doesn't mean we can magically guarantee gradient descent won't produce weird emergent behaviors, or that we can blindly trust OpenAI not to backdoor things. But at least with the latter, you're making a calculated operational decision rather than blindly trusting an opaque black box of entirely unknown provenance.
- JimmaDaRustla 3mo agoWhy would a singe system require 3 full-time dev ops?