3 ms·
If I had the capital I’d make an household inference appliance. No peripherals except Ethernet, integrated compute (cpu+gpu+mem) and secondary storage (+mobo,
by huragok 3mo ago
If I had the capital I’d make an household inference appliance.
No peripherals except Ethernet, integrated compute (cpu+gpu+mem) and secondary storage (+mobo, psu). No accoutrements, just the minimum amount of hardware to run a model as a utility.
Even the appliance faceplate would be a display showing stats like an old HiFi stereo.
Edit: something like a series of modules consisting of a RISC-V CPU + Vortex GPGPU + memory
- Aperocky 3mo agoYou're describing the mac mini/studio with some facelift.
- boredatoms 3mo agoYeah but like running linux hopefully
- throw1234567891 3mo agoso you have invent unified memory for linux first because that’s the limitation today
- Alpha3031 3mo agoFairly sure most iGPUs these days are zero-copy and can dynamically allocate memory so what does "unified memory" mean to you exactly? A wider bus would be nice but it's not exactly a groundbreaking new invention.
- throw1234567891 3mo agoI was actually pretty far off: > Unified memory in Linux creates a single address space accessible to both the CPU and GPU, eliminating the need to manually copy data between system RAM and video memory. It is enabled via NVIDIA's CUDA, AMD's ROCm/HIP, or generic kernel-level Heterogeneous Memory Management (HMM). So it does exist and is available for platforms that matter.
- vkazanov 3mo agoIt is interesting how apple claimed that "unified memory" is something special, and ppl believed them. Intel and AMD had been doing this for years already, and had linux support for it from day 1.
- throw1234567891 3mo agoCool. Apple was the only one who managed to ship a consumer device with UMA and RDMA support. 2TB VRAM max over RDMA.
- vkazanov 3mo agoI think the REALLY cool thing about apple's shared memory implementation is the ultra-wide memory bus. Otherwise, AMD is quite close to what Apple has, and Strix Halo is honestly incredible. Not sure what RDMA brings to the table.
- throw1234567891 3mo agoRDMA increases the inference performance by a significant percentage across devices connected via Thunderbolt 5.4x512 is like a 2TB machine.
- boredatoms 3mo agoThunderbolt RDMA is slower and higher latency than if apple just gave us PCIe, where we could put a (old) connectX card in for infiniband
- vardump 3mo agoRaspberry Pi and other SBCs, Android phones and practically all of the embedded devices with a display and microprocessor. All have unified memory. Linux runs just fine on all of those.
- throw1234567891 3mo agoDude, https://news.ycombinator.com/item?id=48858095 https://news.ycombinator.com/item?id=48858095
- vardump 3mo agoAh ok. I replied to ~45 minutes stale page.
- curt15 3mo agoOr just buy a MI300A server like https://www.servethehome.com/gigabyte-g383-r80-aap1-amd-instinct-mi300a-server-review/ https://www.servethehome.com/gigabyte-g383-r80-aap1-amd-inst...
- throw1234567891 3mo ago"Just". And then GPUs, and RAM? And cooling? Will you really appreciate it when sitting right next to it?
- huragok 3mo agoAbsolutely, but not under the control of Apple.
- robotswantdata 3mo agobuild a Xeon / epyc 4u server. 12 channel ram.
- permalac 3mo agoIs that the nvidia spark?
- imp0cat 3mo agoYes, and a lot of others. A bit too expensive for a home appliance though, isn't it?
- dracotomes 3mo agoIsn't that what what George Hotz is doing over at tiny? https://tinycorp.myshopify.com/ https://tinycorp.myshopify.com/
- huragok 3mo agoYes, but for inference. 45k is so far out of the budget of a professional unless you earn ridiculous money and have no dependents.
- kotberg 3mo ago[flagged]
- musha68k 3mo agoYes, just a big cool Cerebras wafer for the closet please.
- throw1234567891 3mo agoA single wafer comes with 44GB RAM, the reason why Cerebras is so interesting is because the architecture scales up to 1.6PB RAM.
- musha68k 3mo agoCentral heating / thinking.
- not-kinsale-joe 3mo agoSounds like reinventing the home server.
- not-a-llm 3mo agothe pheriphels support, or the appliance faceplate is tens of dollars, that not where you make the saving 95% of the price is going to be in GPU+CPU+RAM
- Tade0 3mo agoI think the closest to that in existence is the LLM ASIC designed by Taalas: https://taalas.com/products/ https://taalas.com/products/ Unfortunately their chatbot, while amazingly fast, doesn't know anything about the company running it. Anyway I wouldn't mind an ASIC running a diffusion language model locally. Even if eventually it would become dated. Beats outsourcing all that to a company that's running on VC money which in the future might either perish or worse - dominate the market and charge whatever they wish.
- auston 3mo agoHere you go: https://www.truffle.net/ https://www.truffle.net/
- joshmn 3mo ago"Login to order" That's a new one.
- grosswait 3mo agoI lasted about 25 seconds on that site. Way too much friction for me to endure just trying to figure out what it is
- wtetzner 3mo agoYeah, I don't know who thought that website was a good idea.
- tristor 3mo agoI feel like this is some sort of satire? There's no actual information or substance to anything on any page of that site.
- swader999 3mo agoIt could heat your home in the winter and your pool in the summer.
- catlikesshrimp 3mo agoIs warming a pool in the summer real where you live?
- surfaceofthesun 3mo agoYes. Solar thermal heaters on the roof are common in Florida and other parts of the south. Some people also use heat recovery devices attached to the AC condenser. Further north I've only seen natural gas heating (e.g. in very rich NYC exurbs). The amount of shade over the pool has a big effect.
- jdc0589 3mo agoI'm keeping an eye on Tenstorrent for this. Pricing seems like its going to end up being in between a super memory dense unified memory platform, and a purpose built GPU. Definitely on the edge of what would make sense at home, but its interesting.