DrPhish

6 ms·

31.
by DrPhish 2y ago
Making your own ggufs is trivial: https://rentry.org/tldrhowtoquant/edit It's a bit harder when they've provided the safetensors in FP8 like for the DS3 series, but these smaller distilled models appear to be
32.
by DrPhish 2y ago
Thanks for the reply! This is perhaps not so much a Hacker News type question since this place is very VC focused, but have you considered publishing any papers on your system? I think it would make a fascinating and valuable bit of researc
33.
by DrPhish 2y ago
Do you need realtime results, or is an ongoing queue of article analysis good enough? Have you considered running your own hardware with a frontier MoE model like deepseek v3? It can be done for relatively low cost on CPU depending on your
34.
Mikupad: An interface in a single HTML file for interacting with language models(github.com)
3 points by DrPhish 2y ago|0 comments
35.
by DrPhish 2y ago
I believe mikupad[0] supports showing logprobs from a llama.cpp backend. [0]: https://github.com/lmg-anon/mikupad
36.
by DrPhish 2y ago
I think we agree on almost all points, but I thought ollama-gui was an official gui, so I’m even more baffled as to what the draw is. Running as a llama.cpp as a service/API endpoint is trivial (I do just that). Maybe you can outline f
37.
by DrPhish 2y ago
It's very hard to put into words without coming off as being unfair to one side or the other, but the ollama project really does provide little-to-no _innovative_ value over simply running components of llama.cpp directly from the comm
38.
by DrPhish 2y ago
Thanks! I tried to use a multi-sport cross section of exercises and recovery routines to show some of the things you could do with it, but all the exercises are ones that are related to sports I do to keep it grounded in reality. I was thin
39.
by DrPhish 2y ago
I had a spreadsheet set up much like that in the past: it would bubble up the exercises I'd done the longest ago for a particular muscle group so I'd always get variety. It worked pretty good, but I found myself often avoiding unp
40.
by DrPhish 2y ago
Haha, I unironically considered this as well, but prefer the "always today, only today" clean automatic view and lack of a physical artifact to lose/forget. I travel for work often enough that being able to get to it on my ph
41.
by DrPhish 2y ago
I actually implemented an export/restore function at one point and ended up taking it out. I waffled heavily on that feature but in the end only losing the last recorded weight for each exercise didn’t feel like enough of a loss to war
42.
Show HN: Minimal, self-hosted exercise tracker(github.com)
127 points by DrPhish 2y ago|39 comments
43.
by DrPhish 2y ago
Out of curiosity, what was the showstopper on dwm?
44.
by DrPhish 2y ago
Yes I have the same engineering sample. I can only keep 48 cores fed with max memory bandwidth given current llama.cpp architectural constraints
45.
by DrPhish 2y ago
Exactly. Locally hosted must be maintained as a viable option by volunteers, or the audio llm equivalent of YouTube circa 2006 will gradually become YouTube circa 2024…
46.
by DrPhish 2y ago
I use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and ddr5-6000, but it is cost prohibitive with CPUs still over $10k each on eBay
47.
by DrPhish 2y ago
I’ve had great luck so far with GPT-SoVITS. With a custom trained Japanese model and clean reference audio the quality is outstanding. It is quite finicky to set up and use though. https://github.com/RVC-Boss/GPT-SoVITS
48.
by DrPhish 2y ago
Yes, it can be done. I'm running a 24-channel DDR5 dual-EPYC rig and get good speed on large MoE models. I only use the GPU for context processing. They're actually a best-case for CPU inference vs dense models. I usually run deep
49.
by DrPhish 2y ago
I run it at home at q8 on my dual Epyc server. I find it to be quite good, especially when you host it locally and are able to tweak all the settings to get the kind of results you need for a particular task.
50.
by DrPhish 2y ago
True. eBay dual-socket Epyc systems ala https://rentry.org/miqumaxx are a great case in point.
51.
405B at Home – CPU Budget Inference Build Guide(rentry.co)
1 points by DrPhish 2y ago|0 comments
52.
MiquMaxx: 405B CPU Inference Build Guide(rentry.co)
1 points by DrPhish 2y ago|0 comments
53.
by DrPhish 2y ago
Unrelated, but GPU coil whine can give unintentional ambience to LLM inference. In some llama.cpp versions on my home inference rig it would manifest as a sort of squelchy sound that would match the generation of characters on the screen, r
54.
by DrPhish 2y ago
LibreELEC is my go-to as well. I like to buy old thin-clients with bad storage for next to nothing and net-boot them diskless
55.
MiquMaxx: CPU Inference Hardware Build Guide(rentry.co)
2 points by DrPhish 2y ago|0 comments
56.
MiquMaxx: CPU Inference Build Guide(rentry.co)
3 points by DrPhish 2y ago|0 comments
57.
by DrPhish 2y ago
On the ODDTAXI front, I can recommend the currently running "Roots of Odd Taxi (RoOT)" series. Its a nice side-story that adds a lot of texture without making the original worse. Also, the "ODDTAXI in the Woods" movie ve
58.
by DrPhish 2y ago
“Debt is slavery” is how I’ve always thought about debt, and what I’ve taught my kids.
59.
by DrPhish 2y ago
I have also seen a single phase go out (bird poop into a transformer) and having redundant PSUs on different phases saved our bacon
60.
by DrPhish 3y ago
Thanks. Watching with bated breath!

More ›