justrunitlocal

3 ms·

1. 1.▲
   by justrunitlocal 5mo ago

   It's fine! There's no world where individuals can buy this kind of stuff. Our company is too small to do it, but I'd love for there to be a public utility of sorts for being able to use LLMs. It is absurd that only these >
2. 2.▲
   by justrunitlocal 5mo ago

   It was pretty hard to justify the purchase to the board but we got a decent deal from a nearby data-center (~15% discount). Thankfully, it's fixed cost, its an asset we can use for our taxes, and it will survive for years to come. The
3. 3.▲
   by justrunitlocal 5mo ago

   We've been running our 10 dev org on 8 H100s on open models (with some tweaks). Sure they aren't as good as the big providers but they 1. don't go down 2. have pretty damn high tok/s. It pays for itself. Posting with a f