8 ms·
Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC
- amelius 3y agoWith only 8GB of VRAM, it can't be that good ...
- speed_spread 3y agoNewer models such as Phi2 run comfortably with 4GB and are good enough to be useful for casual interaction. Sticking with local inference, multiple small models tuned for specific usage scenarios is where it's at.
- navjack27 3y ago30 and 40 series only? My 2080 Ti scoffs at the artificial limitation
- phone8675309 3y agoDon't worry, they'll be happy to charge you $750 for an entry level card next generation that can run this.
- tekla 3y agoYes peasants, Nvidia requires you to buy the latest and greatest expensive luxury gear, and you will BEG for it.
- nickthegreek 3y agoYou can use an older 3-series card. No latest & greatest required.
- nickthegreek 3y agoA 4060 8gb is $300.
- haunter 3y agoCheapest 40xx is $288 https://pcpartpicker.com/products/video-card/#c=552&sort=price&page=1 https://pcpartpicker.com/products/video-card/#c=552&sort=pri... Chepest 8GB 30xx is $220 https://pcpartpicker.com/products/video-card/#sort=price&c=518 https://pcpartpicker.com/products/video-card/#sort=price&c=5...
- andy_xor_andrew 3y agoso they branded this "Chat with RTX", using the RTX branding. Which, originally, meant "ray tracing". And the full title of your 2080 Ti is the "RTX 2080 Ti". So, reviewing this... - they are associating AI with RTX (ray tracing) now (??) - your RTX card cannot chat with RTX (???) wat
- startupsfail 3y agoNo support for bf16 in a card that was released more than 5 years ago, I guess? Support starts with Ampere? Although you’d realistically need 5-6 bit quantization to get anything large/usable enough running on a 12GB card. And I think it’s just CUDA then, so you should be able to use 2080 Ti.
- 0x457 3y ago> I pull my PC with Intel 8086 out of closet > I try to run windows 10 on it > It doesn't work > pff, Intel cpu cannot run OS meant for intel CPUs wat Jokes aside, nvidia been using RTX branding for products that use Tensor Cores for a long-time now. Limitation due to 1st gen tensor cores not supporting precisions required.
- a13o 3y agoThe marketing whiff on ray tracing happened long ago. DLSS is the killer app on RTX cards, another 'AI'-enabled workload.
- nottorp 3y agoThat was my first question, does it display pretty ray traced images instead of answers?
- Havoc 3y agoRTX is a brand more than ray tracing. It is largely an arbitrary generational limit
- speckx 3y agoI, too, was hoping that my 2080 Ti from 2019 would suffice. =(
- operator-name 3y agoYeah, seems a bit odd because the TensorRT-LLM repo lists Turing as supported architecture. https://github.com/NVIDIA/TensorRT-LLM?tab=readme-ov-file#precision https://github.com/NVIDIA/TensorRT-LLM?tab=readme-ov-file#pr...
- mdrzn 3y agoRequirement: NVIDIA GeForce™ RTX 30 or 40 Series GPU or NVIDIA RTX™ Ampere or Ada Generation GPU with at least 8GB of VRAM"
- strangecasts 3y agoUnfortunately the download is taking its time - which kind of base model is it using and what techniques (if any) are they using to offload weights? Since the demo is 35 GB, my first assumption was it's bundling a ~13B parameter model, but if the requirement is 8 GB VRAM, I assume they're either doing quantization on the user's end or offloading part of the model to the CPU. (I also hope that Windows 11 is a suggested and not a hard requirement)
- operator-name 3y agoFor some reason it's actually bundling both LLaMA 13b (24.5GB) and Ministral 7b (13.6GB), but only installed Ministral 7b. I have a 3070ti 8GB, so maybe it installs the other one if you have more VRAM?
- ReFruity 3y agoI have 3070 and when I choose LLaMA in config it just changes it back to Mistral on launch
- nottorp 3y ago8 Gb minimum? So they're excluding the new 3050 6 Gb that is only powered from pcie?
- vdaea 3y agoI suppose this app despite running locally will also be heavily censored. Is there some local chatbot application like this, for Windows, that isn't hell to set up and that is not censored?
- dvngnt_ 3y agoit uses Mistral or Llama 2
- theshrike79 3y agohttps://lmstudio.ai https://lmstudio.ai You can use it to directly search any models, download and run 100% locally. M-series Macs are the simplest, they Just Work. Even faster if you tick the GPU box. Windows needs the right kind of GPU to get workable speed.
- thejohnconway 3y agoCan that interact with files on your computer, like they show in the video?
- operator-name 3y agoLast time I used it, LM Studio doesn't include RAG.
- mrdatawolf 3y agoI just installed it yesterday and you are right it does not seem to have RAG but you can use something like anythingLLM to do the rag work and ut has built in integration with studio LM.
- purpleflame1257 3y agoYou can start with Kobold.cpp which should handhold you through the process.
- fortran77 3y agoTrying it now. Doesn't seem censored to me.
- McAtNite 3y agoI’m struggling to understand the point of this. It appears to be a more simplified way of getting a local LLM running on your machine, but I expect less technically inclined users would default to using the AI built into Windows while the more technical users will leverage llama.cpp to run whatever models they are interested in. Who is the target audience for this solution?
- papichulo2023 3y agoDoes windows uses the pc's gpu or just cpu or cloud?
- robotnikman 3y agoIf they are talking about the Bing AI, just using whatever OpenAI has in the cloud
- McAtNite 3y agoI’m referring to CoPilot which for your average non technical user who doesn’t care whether something is local or not has the huge benefit of not requiring the purchase an expensive GPU.
- zamadatix 3y agoNever underestimate people's interest in running something which lets them generate crass jokes about their friends or smutty conversation when hosted solutions like CoPilot could never allow such non-puritan morals. If this delivers on being the easiest way to run local models quickly then many people will be interested.
- seydor 3y agoWindows users who haven't bought an Nvidia card yet
- SirMaster 3y agoThis lets you run Mistral or Llama 2, so whomever has an RTX card and wants to run either of those models? And perhaps they will add more models in the future?
- tuananh 3y agothis is exactly what i want: a personal assistant. a personal assistant to monitor everything i do on my machine, ingest it and answer question when i need. it's not there yet (still need to manually input url, etc...) though but it's very much feasible.
- Xeyz0r 3y agoBut it sounds kinda creepy don't you think?
- spullara 3y agoit is all local so, no?
- autoexec 3y agoIt generates responses locally, but does your data stay local? It's fine if you only ever use it on a device that you leave offline 100% of the time, but otherwise I'd pay close attention to what it's doing. Nvidia doesn't have a great track record when it comes to privacy (for example: https://news.ycombinator.com/item?id=12884762 https://news.ycombinator.com/item?id=12884762).
- operator-name 3y agoThe source is available, minus the installer. You could always use the base repo after verifying it: https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/trt-llm-rag-windows
- gmueckl 3y agoYou'd be the one controlling the off-switch and the physical storage devices for the data. I'd think that this fact takes most of the potential creep out. What am I not seeing here?
- Capricorn2481 3y ago
- yuck39 3y agoInteresting. Since you are running it locally do they still have to put up all the legal guardrails that we see from Chat GPT and the like?
- dist-epoch 3y agoYes, because otherwise there would be news articles "NVIDIA installs racist/sexist/... LLM on users computers"
- phone8675309 3y agoGaming company Gaming LLM Checks out
- mchinen 3y agoGiven that you can pick llama or mistral in the NVIDIA interface, I'm curious if this is built around ollama or reimplementing something similar. The file and URL retrieval is a nice addition in any case.
- deleted 3y ago[deleted]
- RockRobotRock 3y agoWhy can't this run on older devices?
- Legend2440 3y agoIt's an LLM, older devices don't have the juice. Newer devices only barely have the juice.
- RockRobotRock 3y agoWhat does that mean, though? Is it a VRAM thing? I have a 20 series card with 11 GB and okay performance in CUDA for things like OpenAI Whisper. I think it could run it, albeit slowly.
- imtringued 3y agoIt's always a VRAM thing from this point on. Compute will always be abundant in relation to memory capacity and bandwidth. The only places were this doesn't count is I low power situations such as embedded, where you might intentionally choose a small model to save power.
- RockRobotRock 3y agoThanks for the insight. Time for an upgrade.
- htrp 3y agoAre there benchmarks on how much faster TensorRT vs native torch/cuda?
- operator-name 3y agoI found some official benchmarks for enterprise GPUs, but no comparison data. I couldn't find any benchmarks for commercial GPUs. https://nvidia.github.io/TensorRT-LLM/performance.html https://nvidia.github.io/TensorRT-LLM/performance.html
- fisf 3y agohttps://nvidia.github.io/TensorRT-LLM/performance.html https://nvidia.github.io/TensorRT-LLM/performance.html It was one of the fastest backends last time I checked (with vLLM and lmdeploy being comparable), but the space moves fast. It uses cuda under the hood, torch is not relevant in this context.
- operator-name 3y agoThis looks quite cool! It's basically a tech demo for TensorRT-LLM, a framework that amongst other things optimises inference time for LLMs on Nvidia cards. Their base repo supports quite a few models. Previously there was TensorRT for Stable Diffusion[1], which provided pretty drastic performance improvements[2] at the cost of customisation. I don't forsee this being as big of a problem with LLMs as they are used "as is" and augmented with RAG or prompting techniques. [1]: https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT [2]: https://reddit.com/r/StableDiffusion/comments/17bj6ol/hows_your_results_with_tensorrt/ https://reddit.com/r/StableDiffusion/comments/17bj6ol/hows_y...
- operator-name 3y agoHaving installed this, this is an incredibly then wrapper around the following github repos: https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/TensorRT-LLM https://github.com/NVIDIA/TensorRT-LLM It's quite a thin wrapper around putting both projects into %LocalAppData%, along with a miniconda environment with the correct dependnancies installed. Also for some reason the LLaMA 13b (24.5GB) and Ministral 7b (13.6GB) but only installed Ministral? Ministral 7b runs about as accurate as I remeber, but responses are faster than I can read. This seems at the cost of context and variance/temperature - although it's a chat interface the implementation doesn't seem to take into account previous questions or answers. Asking it the same question also gives the same answer. The RAG (llamaindex) is okay, but a little suspect. The installation comes with a default folder dataset, containing text files of nvidia marketing materials. When I tried asking questions about the files, it often cites the wrong file even if it gave the right answer.
- kkielhofner 3y agoThe wrapping of TensorRT-LLM alone is significant. I’ve been working with it for a while and it’s… Rough. That said it is extremely fast. With TensorRT-LLM and Triton Inference Server with conservation performance settings I get roughly 175 tokens/s on an RTX 4090 with Mistral-Instruct 7B. Following commits, PRs, etc I expect this to increase significantly in the future. I’m actually working on a project to better package Triton and TensorRT-LLM and make it “name and model and press enter” level usable with support for embeddings models, Whisper, etc.
- krunck 3y agoDoes this communicate with other people's (cloud) computers at all?
- spullara 3y agoSo you download a 35G zipfile, then extract that, then run the setup where you can decide not to install one of the models. Why doesn't it just download the model you want to use? Who did this? Oh and now the install failed with no error message. Lovely.
- 9front 3y agoIn my case the installer finished okay. It pulled in miniconda, nvidia-toolkit and "Mistral 7b int4". Upon launch it opens a gradio-chat session. For the datasets it supports .txt, .pdf, .doc files. There's a "YouTube URL" option and "AI model default". When asked "What is ChatWithRTX" the reply was: Chat with RTX is not mentioned in the given context information. It is possible that it is a separate event or feature that is related to the NVIDIA Lounge at LTX 2023 or QuakeCon 2023, where attendees can interact with a virtual assistant or chatbot powered by NVIDIA's GeForce RTX technology. However, without more information, it is impossible to provide a more detailed answer. Reference files: portal-prelude-rtx-inside-the-game-interview.txt
- 9front 3y agoIt's a 35.1GB download!
- politelemon 3y agoIt's the L in LLM!
- bugbuddy 3y agoThis is amazing and it shows that Nvidia is at least 3 decades ahead of the competitors. Imagine this turning into a powerful agent that can answer everything about your life. It will revolutionize life as we know it. This is why Nvidia stock is green and everything else is red today. I am glad that I went all in on the green team. I wished I could get more leveraged at this point.
- dingnuts 3y agocalm down NVIDIA marketing department, 30 years is a long time
- gruturo 3y ago3 decades might be how long it takes until this is running locally in your glasses, although we may hit some hard limit in silicon before we get there at all. But AI models are already running on tablets (not necessarily on Nvidia hardware) and I expect some phone to ship with them within a year (maybe as a stunt, I guess it would be a few years more before this is practical).
- v4lheru 3y agoi have 3050 and its failing for me. do i have to install it on Windows Drive - cause i don't have that much space there?
- haunter 3y agoDo you have an 8GB 3050?
- mleroy 3y agoI actually thought AMD would release something like that. But somehow they don't seem to see their chance.
- imtringued 3y agoOn what hardware with what software? Do you think people got an MI300 lying around? AMD's GPUs simply weren't meant for GPGPU.
- randerson 3y agoThe Creative Labs sound cards of the early 90's came with Dr. Sbaitso, an app demoing their text-to-speech engine by pretending to an AI psychologist. Someone needs to remake that!
- davedunkin 3y agoLevels is making something like that. https://twitter.com/levelsio/status/1756396158652432695 https://twitter.com/levelsio/status/1756396158652432695 I would really want it to have that Dr. Sbaitso voice, though, telling me how to be fitter, happier, more productive.
- dcist 3y agoYes, I remember Dr. Sbaitso very, very well. I spent many hours with it as a kid and thought it was tons of fun. To be frank, Dr. Sbaitso is why I was underwhelmed when chatbots were hyped in the early 2010s. I couldn't understand why anyone would be excited about 90s tech.
- jhbadger 3y agoReally just 1960s tech. Dr. Sbaitso was really just a version of ELIZA with the only new part being the speech synthesis.
- Mistletoe 3y agoChatting with ALICE is what has tempered my ChatGPT hype. It was neat and seemed like magic, but I think it was in the 90s when I tried it. I'm sure for new people it feels like an unprecedented event to talk to a computer and it seem sentient. https://www.pandorabots.com/pandora/talk?botid=b8d616e35e36e881 https://www.pandorabots.com/pandora/talk?botid=b8d616e35e36e... Like other bogus things like tarot or horoscopes, it's amazing what you can discover when you talk about something, it asks you questions, and what you want or desire eventually floats to the surface. And now people are even more lonely... >Human: do you like video games >A.L.I.C.E: Not really, but I like to play the Turing Game.
- lysp 3y agoIt's actually been ported to web - along with a lot of other dos based games. https://classicreload.com/dr-sbaitso.html https://classicreload.com/dr-sbaitso.html
- anon115 3y agoSystem Requirements Platform Windows GPU NVIDIA GeForce™ RTX 30 or 40 Series GPU or NVIDIA RTX™ Ampere or Ada Generation GPU with at least 8GB of VRAM RAM 16GB or greater OS Windows 11 Driver 535.11 or later ----yeah not in this lifetime baby...
- redder23 3y ago> and all you need is an RTX 30- or 40-series GPU with at least 8GB of VRAM Smells like artificial restriction to me. I have a 2080 Ti with 8GB of VRAM that is still perfectly fine for gaming. I play in 3440x1440 and the modern games need DLSS/FSR on quality for nice 60++ - 90 FPS. That is perfectly enough for me and I have not had a game, even UE5 games where I really thought I really NEED a new one. I bet that card is totally capable of running that chatbot. They do the same with frame generation. There they even require you a 40 series card. That is ridiculous to me as these cards are so fast that you do not even need any frame generation. The slower cards are the ones that would benefit from it most so they just lock it down artificially to boost their sales.
- magicalhippo 3y ago> 2080 Ti with 8GB of VRAM Sure you don't mean 11GB[1]? Or did they make other variants? FWIW I have a 2080 Ti with 11GB, been considering upgrading but thinking I'll wait til 5xxx. [1]: https://www.techpowerup.com/gpu-specs/geforce-rtx-2080-ti.c3305 https://www.techpowerup.com/gpu-specs/geforce-rtx-2080-ti.c3...
- redder23 3y agoYes of course, I have a 11GB of VRAM. My next card will be an AMD one. I like that they are open sourcing most of their stuff and I think they play better with Linux Wine/Proton. FSR 3 also not artificially restricts cards and runs even on the competition. I read today about at open source API that takes CUDA calls and runs them on AMD or everywhere. I am sure there will be some cool open source projects that do all kinds of things if I ever even need them.
- xyst 3y agoChatbots are hot again. 2010-2011, vibes.
- smcleod 3y agoMS Windows only it seems
- temp_user 3y agoSo I am looking for the Linux version of this...
- temp_user 3y agoSo, there should be something equivalent for Linux right? Ill be thankful to the person that points me to the right github repository, I am new to local LLM.
- alecco 3y agohttps://github.com/NVIDIA/TensorRT-LLM https://github.com/NVIDIA/TensorRT-LLM
- deleted 3y ago[deleted]
- errorsz 3y ago[flagged]
- errorsz 3y ago[flagged]
- errorsz 3y ago[flagged]