tatef

4 ms·

1.
by tatef 1mo ago
Hey HN, along with many of you I’ve been burning through my Claude Code usage and have been wondering how I can save on costs and still make the most of it. I’m also a big local model enthusiast, but the truth is I don’t believe people shou
2.
Sharding a 70B model across 39 Intel laptops(github.com)
5 points by tatef 2mo ago|1 comments
3.
by tatef 2mo ago
Hey everyone, we’ve been working on a project called https://cascadia.to which allows you to shard LLMs across Intel-based machines and perform inference across CPUs/GPUs/NPUs. To get it functioning, we split models in
4.
by tatef 6mo ago
Hi HN! Coming to you today with Autolab, a framework I built that combines Ralph loops and Karpathy's autoresearch to design and run tests and make novel research discoveries. I hope one day a similar framework can be used with hardwar
5.
Using LLMs to make novel research discoveries(github.com)
1 points by tatef 6mo ago|1 comments
6.
by tatef 6mo ago
Hi HN, Wanted to share a project I've been working on aimed at helping solve the context problem called Memoryport. It works across LLM providers/apps, stores conversations locally, is fully OSS, and enables anyone to keep track o
7.
Add 500M tokens of context space to any LLM with <300ms latency(github.com)
3 points by tatef 6mo ago|1 comments
8.
by tatef 6mo ago
Thanks for sharing this! If you'd be interested in running the benchmark yourself with Hypura I'd happily merge into our stats. Otherwise will add to my todo list :)
9.
by tatef 6mo ago
Yes definitely. I use a M1 Max with 32gb of RAM daily and it's about on par from a performance standpoint with the new base M5 Pro 24gb. You can check the benchmarks in the repo if you're interested in seeing specific performance
10.
by tatef 6mo ago
Noted, thanks. I had LLM help positioning this message but I did the initial draft along with edits. Will keep in mind for the future.
11.
by tatef 6mo ago
Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU. There is no writing to SSDs on inference with this architecture.
12.
by tatef 6mo ago
Yes, exactly this.
13.
by tatef 6mo ago
I'm referencing it as being possible, however I didn't share benchmarks because candidly the performance would be so slow it would only be useful for very specific tasks over long time horizons. The more practical use cases are le
14.
by tatef 6mo ago
Yes, definitely agree. It's more of a POC than a functional use case. However, for many smaller MoE models this method can actually be useful and capable of achieving multiple tokens/sec.
15.
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon(github.com)
221 points by tatef 6mo ago|85 comments
16.
by tatef 6y ago
We aren't a package manager. We're a registry and CDN (of sorts). Blockchain is actually a huge solution to this problem for three very notable reasons. The first is that Deno module imports are url based, and we don't want c
17.
by tatef 6y ago
Actually, this raises a very good point. I'm Tate, a co-founder. Our publishing system works in a way that users will be able to publish malicious modules, yes, but our registry is not decentralized up to a certain point; let me elabor
18.
by tatef 6y ago
Indeed, this is a massive issue with any url based imports. Because Trex supports nest.land, this is not an issue. nest.land is actually a first-of-its-kind blockchain module registry and CDN. Because we use the blockchain for storing modul
19.
by tatef 6y ago
Trex is not a company; Trex is a product under the crewdevio organization on GitHub.
20.
by tatef 6y ago
In my opinion, Trex is actually working against what npm introduced to Node. Though I don't exactly know what "main problem" you're referring to, I can say that: 1) Trex is supporting multiple module registries, not just