omneity

8 ms·

91.
by omneity 1y ago
Qwen3 32B is a dense model, it uses all its parameters all the time. GPT OSS 20B is a sparse MoE model. This means it only uses a fraction (3.6B) at a time. It’s a tradeoff that makes it faster to run than a dense 20B model and much smarter
92.
by omneity 1y ago
Consumer GPUs in theory and by a large margin (10 5090s will eat an H100 lunch with 6 times the bandwidth, 3x VRAM and a relatively similar compute ratio), but your bottleneck is the interconnect and that is intentionally crippled to avoid
93.
by omneity 1y ago
It takes ~17-20GB on Q4 depending on context length & settings (running it as we speak) ~30GB in Q8 sure, but it's a minimal gain for double the VRAM usage.
94.
by omneity 1y ago
Conversations are always "reprocessed from scratch" on every message you send. LLMs are practically stateless and the conversation is the state, as in nothing is kept in memory between two turns.
95.
by omneity 1y ago
Would running vLLM in docker work for you, or do you have other requirements?
96.
by omneity 1y ago
Also dishonest since the reason Qwen 2.5 got so popular is not so much paper performance.
97.
by omneity 1y ago
Crazy timing! I shipped a similar abstraction for llms a bit over a week ago: https://github.com/omarkamali/borgllm pip install borgllm I focused on making it Langchain compatible so you could drop it in as a replaceme
98.
by omneity 1y ago
Eventually through experience and self-play with the technologies in question.
99.
by omneity 1y ago
Is it actually growing, or is it swarm-like assembly, such as a coral colony?
100.
by omneity 1y ago
I just posted incidentally about Wikipedia Monthly[0], a monthly dump of wikipedia broken down by language and cleaned MediaWiki markup into plain text, so perfect for a local search index or other scenarios. There are 341 languages in ther
101.
by omneity 1y ago
They did and then spent $5B purchasing Instagram and Whatsapp to recover.
102.
by omneity 1y ago
Asking the fox to guard the hen house..
103.
by omneity 1y ago
Hello HN! I've been developing a lot of LLM-based apps recently, and I got tired of writing the same glue code each time to switch providers, find the base URL again, or other strategies to deal with provider idiosyncracies, so I extra
104.
Show HN: BorgLLM, Zero-config LangChain Client, key rotation and rate limit mgmt(borgllm.com)
3 points by omneity 1y ago|1 comments
105.
by omneity 1y ago
My thoughts exactly. Or at least curate a sample of true positives and use zero shot classification with semantic embeddings + keywords.
106.
by omneity 1y ago
Looking at the post, the methodology (checking the presence of a few keywords, not even including "LLM") is very simple with a lot of potential false negatives. 20% is closer to the lower bound. It could also explain the lower vot
107.
by omneity 1y ago
Hello HN! I've been developing a lot of LLM-based apps recently, and I got tired of writing the same glue code each time to switch providers, find the base URL again, or other strategies to deal with provider idiosyncracies, so I extra
108.
Show HN: BorgLLM, Zero-config LangChain Client, key rotation and rate limit mgmt(pypi.org)
1 points by omneity 1y ago|1 comments
109.
by omneity 1y ago
I've been quite obsessed about ramping up (technically complex, not basic crud/wrappers) SaaS development with Gen AI tools, speeding things from months to weeks to days. But then I hit a snag: operations are the new bottleneck. H
110.
by omneity 1y ago
This is an incredible achievement for a solo developer. The dev is from the Deepseek team by the way.
111.
by omneity 1y ago
It will work, but at the scale needed for pretraining you are bound to have many quality issues that will destroy your student model, so your data cleaning process better be very capable. One way to think of it is that any little bias or un
112.
by omneity 1y ago
kinda? omarkama.li
113.
by omneity 1y ago
I am not entirely surprised by the relative equivalence for the sparse model. The combined bandwidth of 2x 5060 Ti ≃ 1x 3090. There are inefficiencies in multi-gpus that are more negligible at smaller dimensions, hence why the dense 32B mod
114.
by omneity 1y ago
That seems to be the gist of it. You cannot rely on serverless alone and you need one or many pre-warmed instances at all times. This distinction is rarely mentioned in serverless GPU spaces yet has been my experience in general.
115.
by omneity 1y ago
> Time-to-First-Token of approximately 19 seconds for a gemma3:4b model (this includes startup time, model loading time, and running the inference) This is my biggest pet-peeve with serverless GPU. 19 seconds is a horrible latency from t
116.
by omneity 1y ago
Minutes or hours at most depending on the model size and the training hardware.
117.
by omneity 1y ago
Very rougly you can consider the Bs of a model as GBs of memory then it depends on the quantization level. Say for an 8B model: - FP16: 2x 8GB = 16GB - Q8: 1x 8GB - Q4: 0.5x 8GB = 4GB It doesn't 100% neatly map like this but this gives
118.
by omneity 1y ago
Worth mentioning that a single H100 (80-96GB) is not enough to run R1. You're looking at 6-8 GPUs on the lower end, and factor in the setup and download time. An alternative is to use serverless GPU or LLM providers which abstract some
119.
by omneity 1y ago
With a lot of motivation, grit, coffee and reinventing the wheel every couple of weeks.
120.
by omneity 1y ago
Have you tried open weights vision models such as Qwen VL, MiniCPM, PaliGemma...? I'm also curious how usable are simpler vision models such as Florence in case you explored this direction.

More ›