3 ms·
Keep in mind that you need at least the jetson board and the carrier board to make it work. The official nvidia devkit device with both boards is actually way m
by tmp777 3y ago
Keep in mind that you need at least the jetson board and the carrier board to make it work. The official nvidia devkit device with both boards is actually way more expensive that this (amazon lists it at $1,999).
The jetson itself is pretty cool though, and nvidia has a bunch of tutorials / cool demos of running LLMs on it, e.g.: https://www.jetson-ai-lab.com/tutorial_live-llava.html https://www.jetson-ai-lab.com/tutorial_live-llava.html
- throwanem 3y ago$2000 for fast, self-contained CUDA inference doesn't seem too unreasonable. How's it bench next to a 4090 or two?
- tmp777 3y agoThe big upside here is memory: you get 64G, which means you can easily run 70B models at 4bits. You'd need 3x4090s for that. And because of how most inference engines work today, the performance of such setup will actually be slightly lower than 1x4090. You should be really comparing this to A6000, which has similar performance to 3090/4090 (depending on the gen) but with 48GB memory — A6000 is way more expensive. In terms of numbers, jetson agx orin is closer to 3090: - jetson agx orin 64gb has 275 TOPS - 3090 has 285 TOPS - 4090 has 1321 TOPS Another big advantage is power: jetson is getting these 275 tops at 60W, vs. 350W for 3090.
- ZiiS 3y agoIt is from the 3090 generation and 1/6th the TDP.