5 ms·
I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems
by lumost 5mo ago
I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription.
There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than sufficient.
- bigmadshoe 5mo agoCan you give an example of such a problem?
- jjcm 5mo agoDecompiling a binary and recreating the source, doing a full line-by-line security audit, always-on agents monitoring state minute-by-minute, etc. I would very easily find ways to hit that level of token usage if it was cheaper/faster.
- lumost 5mo ago"Design me a 3d printable rocket engine for a hobby rocket project. Verify it's design in a full simulation. Iterate until it works reliably in simulation based on a verified printable design on a consumer laser sintering device (or substitute contract manufacture for under 1000 dollars)." This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics studies, material analysis, operating systems etc.
- SiempreViernes 5mo agoFrom the prompt it seems evident the envisioned user doesn't have an interest in designing the motor themselves, so why not simply buy a stock motor?
- lumost 5mo agoI can't put a 10 page narrative on how my specific motor should work into a hacker news post ;) you can also imagine the above where the goal is to have the ai exceed the performance of stock motors.
- lopis 5mo agoI'm excited to have my agent read and summarize your article into 5 bullet points.
- tomcam 5mo agoStop it, you tease. I'm getting a little tingly
- bobim 5mo agoAre there already skills around modelling, simulation and post-processing? Any pointers?
- bigmadshoe 5mo agoI’m not convinced at all that the model won’t just get stuck in a loop where it doesn’t understand how to fix the broken rocket. I see similar failure modes in far simpler projects strictly confined to coding. This feels closer to “make me a profitable business, make no mistakes” than to a simple coding project.
- schreiaj 5mo agoYou almost certainly do not want an LLM to do that. Leap71 actually has computational models generating functional rocket engines that way. You could absolutely wrap a tool like that in a shell and handle control with an LLM and not need anywhere near the tokens. Thats the thing - these models see and predict tokens. For any real engineering you get more bang for your buck using math.
- pjc50 5mo agoVerifying how the model works against the real world is the difficult, dangerous bit. There might be some interesting side effects from making simulation software, which is currently either an expensive niche or quirky university project (SPICE always has that feel).
- nodja 5mo agoNot OP but if I had a couple RTX 6000 I'd throw them at decompiling bloodborne to play on PC without emulation.
- neals 5mo agoI'm curious how hardware and power cost would stack up to subscription cost
- jjcm 5mo agoI did an estimate of that if you're interested: https://x.com/pwnies/status/2028831699736637912 https://x.com/pwnies/status/2028831699736637912 The TL;DR though is that a 10-15b param model baked into an ASIC with the latest fab tech would take around 62W of power draw when active. At ~10k+ t/s though it likely would only be active for short bursts of time. It'd fit perfectly fine within the thermal envelope of a laptop. The approach makes a lot of sense. Once you get to those speeds, latency of the network becomes one of the bigger bottlenecks, so local has a real advantage over a subscription.
- IanCal 5mo agoIs latency of the network that noticeable? Aren’t we talking low hundreds of ms at worst here? Much lower for something close regionally.
- wmf 5mo agoYou're not counting the capex which could be the same cost as 5-10 years of Claude.
- giantrobot 5mo agoThis assume Claude's price doesn't change. Which isn't a great assumption considering inference providers are moving to usage based billing. Also the VC money isn't going to last indefinitely. Current inference providers are being subsidized with VC money at this point.
- nl 5mo ago> Current inference providers are being subsidized with VC money at this point. This isn't true. Anthropic is making an operating profit including the loss making subsidised subscriptions (but excluding training). Your normal inference provider is doing great. Do the math on a H100 rental and you can see the margins.
- barnas2 5mo agoA company called Taalas is working on something like that. Not Opus4.6 quality, but I'm sure they're targeting larger models. Currently they're using a LLama 8B model. It runs at ~17k tokens per second, and you can test it at https://chatjimmy.ai/ https://chatjimmy.ai/.
- mirekrusin 5mo agoIt starts to be interesting when latency is better than average website.
- all2 5mo agoThe immediate load (less than 200ms on my machine through a slow connection) is quite pleasant, tbh.
- vineyardmike 5mo agoI’m not sure if this is what you meant, but at 17k t/s, you start to compete with the speed of network calls. You could approach the point of generating an HTML/js/css page faster than some websites can be returned over the network.
- normalaccess 5mo agoWhen that happens, they will be able to rewrite reality in real time.
- Brisk4t 5mo agoI'm rooting for them HARD but they've been quiet since their last (and only) blog. X and LinkedIn are empty too. I really hope it wasn't a pipe dream.
- tomaytotomato 5mo agoThat's cool, I just tested it out and it is fast but unfortunately its accuracy is not great.
- goofy_lemur 5mo agoOk heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cutting edge will always be cloud
- c-hendricks 5mo ago> cutting edge will always be cloud Don't think anyone was refuting that? And of course when you pool resources you have access to more resources.
- zamadatix 5mo agoThey just mean this part: "where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription." Upgrading local hardware will remain the more expensive alternative to the subscription regardless what the relative cost of running the models themselves are. If the local hardware to do so becomes affordable then the subscription will be even more affordable, not expensive. At least for these kinds of mega tasks. For more micro task we will always end up with unutilized local compute we already purchased which will be "free" since we already paid for non-AI reasons (e.g. a gaming GPU while not gaming).
- mirekrusin 5mo agoJust like cloud is "cheaper" than colo/metal, right?
- lumost 5mo agoLaptops/desktops are cheaper per flop than any datacenter hardware by a good order of magnitude. The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin. You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x the same hardware sitting in the back of an office somewhere. While all of these premiums matter for 24/7 systems under active development, they don't really matter for ephemeral small scale workloads.
- general_reveal 5mo agoRound Robin the free tier APIs, should be effectively free. Just say say “sike” if discussing sensitive issues so the LLM never flags you.