kkielhofner

13 ms·

91.
by kkielhofner 2y ago
IBM is a sales and services org. Their customers aren’t going to build their own RAG and agent frameworks, vector DBs, data ingest pipelines, finetunes, high scale inference serving solutions, etc, etc. There’s an incredible amount of stuff
92.
by kkielhofner 2y ago
TOPS != TFLOPS RTX 4090 Tensor 1,321 TOPS according to spec sheet so roughly 35x. RTX 4090 is 191 Tensor TFLOPS vs M2 5.6 TFLOPS (M3 is tough to find spec). RTX 4090 is also 1.5 years old.
93.
by kkielhofner 2y ago
My project is ROCm (torch, more or less) and working with OLCF staff I've never heard of HIP in use but based on their training series it is supported[0]. Of course my personal experience isn't exhaustive and it can be inferred fr
94.
by kkielhofner 2y ago
I currently have a project with ORNL OLCF (on Frontier). The short answer is yes. Happy to answer any questions I can.
95.
by kkielhofner 2y ago
Very misleading, likely deliberately so. Per usual it’s working - here we are talking about it. It’s evaluating input prompt at that speed. Important metric and impressive on its own but “going X tokens per second” in the mind of anyone loo
96.
by kkielhofner 2y ago
> It may be an irrational hobby Refreshing to hear someone just come out and say it. I don't get it but hey, I'm into all kinds of weird stuff that doesn't make sense either. Just don't try to attempt some argument th
97.
by kkielhofner 2y ago
Nice! Out of curiosity what model(s) are you using to generate the embeddings?
98.
by kkielhofner 2y ago
Fair points but with all due respect completely misses the point and context. My comment was a reply to a new user interested in esphome on a post about esphome. You're talking about CircuitPython, 35KB web replies, PSRAM, UF2 bootload
99.
by kkielhofner 2y ago
Good point and batteries, solar, exterior environments, etc are what I would consider "advanced" use cases with significant additional challenges and considerations. All of my use-cases are interior environments with the most &quo
100.
by kkielhofner 2y ago
You can get a three pack of esp32 dev boards (with headers) for $6 from Aliexpress. For that you get: 1) Wifi. 2) Much more robust ecosystem, including esphome (the subject of this post).
101.
by kkielhofner 2y ago
I am a VERY low-effort hardware person (even soldering is um, not my favorite) but for years my approach has been: 1) Go to Amazon and buy a three pack of ESP32 dev boards with headers[0]. They're always some random seller, etc but I&#
102.
by kkielhofner 2y ago
IMO the issues Microsoft had weren't unexpected. If you look at the disaster that was their US military Hololens project it turns out a fairly significant portion of the population experiences motion sickness and other fundamental issu
103.
by kkielhofner 3y ago
Very wrong. Llama is usable by any company under 700M MAU.
104.
by kkielhofner 3y ago
> I pointed them towards vLLM, but it sounded like they were set on ollama I'm baffled how someone could be so set on Ollama. Being married to a tool is always weird to me and being set on the (very) wrong tool for the job even when
105.
by kkielhofner 3y ago
Your parent comment (by my reading) implied the H100 "does just fine" when it has 50MB SRAM. The reason Grok needs multiple racks of chips to serve up models that fit in a single H100 is because Grok chips are SRAM only while the
106.
by kkielhofner 3y ago
...and 80GB of very high speed VRAM.
107.
by kkielhofner 3y ago
Obviously I don't know the story but yeah... That founder and potential company are in for a rude awakening. > I don’t really think ollama scales to production workloads anyway Not even close. At the risk of gatekeeping in terms of
108.
by kkielhofner 3y ago
> First of all, they are not violating any license or terms in any form. IANAL but from what I understand likely debatable at least. You'll notice I said "sleazy" and didn't touch on license, potential legal issues, e
109.
by kkielhofner 3y ago
What's frustrating is they're good at the value they add: documentation, UI/UX, their model zoo thing, etc. This alone is something to be proud of and adds a lot of value. As of this writing they have 2,377 commits - there is
110.
by kkielhofner 3y ago
Exactly. Demand for FP64 is significantly lower than for ML/AI. Pascal isn’t incredibly cheap by comparison because it’s some secret hack. It’s cheap by comparison because most of the market (AI/ML) doesn’t want it. Speaking of wh
111.
by kkielhofner 3y ago
Pascal series are cheap because they are CUDA compute capability 6.0 and lack Tensor Cores. Volta (7.0) was the first to have Tensor Cores and in many cases is the bare minimum for modern/current stacks. See flash attention, triton, et
112.
by kkielhofner 3y ago
FWIW PyMuPDF doesn't do OCR. It extracts embedded text from a PDF, which in some cases is either non-existent or done with poor quality OCR (like some random implementation from whatever it was scanned with). This implementation bolts
113.
by kkielhofner 3y ago
> An AI chip with questionable software support. As many can attest with the random "ARM SBC of the day" the Raspberry Pi continues to reign supreme because the software and ecosystem is second to none in the ARM SBC space. You
114.
by kkielhofner 3y ago
For those who don't know - one year after launch. Meanwhile Nvidia will go as far as to back port Hopper support to CUDA 11.8 so it "just runs" the day of launch with everything you already have.
115.
by kkielhofner 3y ago
I work with Frontier and ORNL/OLCF. They have had and continue to have issues with AMD/ROCm but yes, they do of course get excellent support from AMD. The entire team at OLCF is incredible as well (obviously) and they do amazing w
116.
by kkielhofner 3y ago
It's been talked to death but non-CUDA implementations have their challenges regardless of use case. That's what first-mover advantage and > 15 years of investment by Nvidia in their overall ecosystem will do for you. But suppo
117.
Tiny Corp decides to make both AMD and Nvidia versions of its TinyBox AI server(tomshardware.com)
3 points by kkielhofner 3y ago|0 comments
118.
by kkielhofner 3y ago
I'm working on a project ATM at one of the DoE sites you're likely referring to... Maybe we'll bump into each other! Ah yes, pytorch: 1) Check issues, PRs, etc on torch Github. Considering market share ROCm has a multiple of
119.
by kkielhofner 3y ago
Genuine questions. What are your use cases? What do you do? How much experience? My personal experience shows CUDA to in fact be a very deep moat. In ~12 years CUDA and ~6 ROCm (since Vega) I’ve never met a professional who says otherwise,
120.
by kkielhofner 3y ago
> probably less concurrent demand This is a significant understatement. ChatGPT has an estimated 100m monthly active users. Groq gets featured on HN from time to time but is otherwise almost completely unknown. According to their stats t

More ›