huac

13 ms·

91.
by huac 3y ago
Finetuning teaches skills. RAG provides 'facts.'
92.
by huac 3y ago
Net effect: fewer clicks to source websites, fewer clicks on ads on said websites.
93.
by huac 3y ago
I feel like this is a linear optimization problem (maximize benchmark score for games and/or for work) constrained on price / part compatibility, not something which you can just feed into ChatGPT. Unfortunately, the compatibility
94.
by huac 3y ago
Is it possible to have a 'knob' here? some topics might need low latency even if most don't. My sense, reading this, is that while most topics / use cases will be fine on Warpstream, that some will not be.
95.
by huac 3y ago
I sense that most folks will end up using this via a Python wrapper, if at all, much like how HF has Transformers written in Rust but with a Python wrapper. This library is targeting production, a much smaller subset of engineers. See the &
96.
by huac 3y ago
I'm writing a blog post with some more reasoning but my view is that it can be useful for certain simpler tasks (eg unstructured -> structured, basic summarization) and not more complex things (eg generation). The tricky thing is th
97.
by huac 3y ago
you need an inference server. I am doing ~400 tokens/sec on 7B with a 4090 with multiple concurrent (streaming!) requests. it's reasonably straightforward for me to host this and serve public requests, but would likely just be a b
98.
by huac 3y ago
agree! and notably - there's a difference between pre- and post-filtering -- left as an exercise to the reader :)
99.
by huac 3y ago
dev experience is very very important, but I think it's so subjective -- if you are really pushing the boundaries, maybe you want a super powerful platform which can use all the bells and whistles, but if you want to hack a weekend pro
100.
by huac 3y ago
I would suggest that anyone trying a real comparison of vector DB's consider the following - necessary functions / use cases (eg prefiltering, dense search) - embeddings version management - anticipated embedding siz
101.
by huac 3y ago
You should test yourself on your own use case (eg vector dimension, prefiltering, throughput, target latency). In my testing, using identical HNSW configs between OS and a purpose built vector DB, I saw 10x+ better performance with the vect
102.
by huac 3y ago
did they announce that anywhere? it does appear like progress has slowed down quite a lot.
103.
by huac 3y ago
I've written a version of this in Golang (tied to OpenAI API, mostly): https://github.com/stillmatic/gollum/blob/main/dispatch.go Define a struct and tag it with golang's json comments. Then, g
104.
by huac 3y ago
no, because the compute intensity scales with the number of classes which you wish to classify to. if you have n classes, you need to do n gzip compressions at inference time. in the embedding world, you only call the embedding model once o
105.
by huac 3y ago
how would you build an index over the gzip encoded data? seems quite different from building indices over vector embeddings.
106.
by huac 3y ago
The gzip approach is much slower at inference time because you need to compute the gzip representation of the concatenated strings (query + target). Intuitively, this should be significantly more than a dot product of two embedding vectors.
107.
Show HN: Use Google Sheets to do realtime feature flagging(github.com)
2 points by huac 3y ago|0 comments
108.
by huac 3y ago
it's marketing :)
109.
by huac 3y ago
from somebody who bought 2500 of them: https://news.ycombinator.com/item?id=36313960
110.
by huac 3y ago
It would be really cool to support more vendors (eg Lambda Labs, Runpod, etc) - getting GPU availability feels like 'any port in a storm' sometimes :)
111.
by huac 3y ago
It is, in fact, a try-catch. https://github.com/BerriAI/reliableGPT/blob/main/reliablegpt...
112.
by huac 3y ago
I think it's inevitable that the LLM providers will build out more and more of the dev tools around their stack, such as this Azure service. The model providers can offer significantly better products via their direct access to the mod
113.
OpenAI is a hyperscaler like AWS(hua.substack.com)
1 points by huac 3y ago|0 comments
114.
by huac 3y ago
does that scale to high-dimensions? IIUC the photon mapping procedure typically uses a kd-tree to find closer/farther candidates, and if you consider something like ada embeddings (1536 dim), that would be very computationally intense.
115.
by huac 3y ago
I have gotten 10x speedups with SIMD on modern hardware - goes from fast to blazing. https://github.com/stillmatic/gollum/blob/main/vectorstore.g... FWIW - Weaviate's pure Go implementation of cosi
116.
by huac 3y ago
nice, if I'm reading this right, you do random projection as a first stage filter, then exact search over the filtered subset? that's clever, did you test precision/recall at all?
117.
by huac 3y ago
I'd be curious how this stacks up on ANN-benchmarks ( https://ann-benchmarks.com/ ). FWIW exhaustive search is still probably good enough for most use cases. IMO the exhaustive search should use a heap https:/&#x2F
118.
by huac 3y ago
Per card? DGX is 8 H100, so ~$300k per node?
119.
by huac 3y ago
Arguably better async primitives too. Python's main advantage at this point feels like a stronger ecosystem of packages calling into native code (eg numpy et al mostly call optimized C routines). If Typescript adds more of those librar
120.
by huac 3y ago
I don't quite understand the convenience, R2 and the other services are already quite easy to use. I wrote a simple wrapper that exposes the same API: https://github.com/stillmatic/storage-wrapper/blob/ma

More ›