11 ms·
Talk-Llama
- bribri 3y agoElevenlabs voice is amazing but it's so expensive. You can easily spend $20 on a single conversation.
- washadjeffmad 3y agoIf you have adequate local resources, TorToiSe has pretty remarkable quality, especially after a little training.
- deleted 3y ago[deleted]
- avereveard 3y agoAren't there text to talk solution that can receive a stream of text so one doesn't have to wait for llama to finish production before getting the answer talked out? I guess it'd only work if the model can keep the buffer filled fast enough so the tts engine doesn't stall.
- taneq 3y agoJust have llama.cpp emit an “um”, “uhh” etc. when the buffer’s down to one word. :D
- umtksa 3y agoyou laught at the end but I love this solution
- ryanklee 3y agoHumans have loved the same solution since we first started talking, as well
- mirekrusin 3y agoDon’t forget to mix it with “apparently”, “you know what I’m saying”, “I mean”, “you know” etc.
- ryanklee 3y agoI can't tell if you're disparaging the usage or not (truly, I can't tell), but such utterances exist because they serve a real function. Disfluency is an integral part of speech.
- vidarh 3y agoI think it's a good idea, if done well. It could also potentially be combined with dynamically adjusting speed of the speech, and reducing or increasing the use of shortcuts and contractions, making word replacements. I know wish for a model built to be a low-computation filter that takes text in and produces padded text out intended for TTS and annotated with pauses or sounds and extra words that maintains the same meaning but provides the ability to dynamically adjust the level of verbosity to maintain a fixed rate of words per minute.
- taneq 3y agoI always thought of them as the human equivalent of hard drive noises. <brrrrr brrbrrbr>
- taneq 3y agoarray_rand($verbal_fry[$locale]) /* :D */
- HPsquared 3y agoTiming and emphasis work better if you know where the sentence is going; otherwise you sound like one of those translators at the UN with the flat stream of words.
- numpad0 3y agoI suppose it could be buffered, and used to only shorten the wait in cases with long responses, rather than aiming for perfect and push out words as they come. Besides, GPTs sound like translators anyway.
- moffkalast 3y agoI mean.. 99% of the current TTS engines wouldn't know timing and emphasis if it hit them. Besides humans do this all the time, we start saying words before we even have, uhh, any idea how we're gonna end the sentence and for the most part it, uhh, works out. Should be doable.
- unshavedyak 3y agoAlso there's a big difference between waiting for a buffer of outputs and waiting for all the outputs.
- HPsquared 3y agoThe difference is that if you are the one saying the sentence, you at least have some idea where it's going. Receiving one word at a time from a different source isn't the same.
- moffkalast 3y ago> you at least have some idea where it's going Unless you're Michael Scott.
- ccoreilly 3y agoElevenLabs and Gemelo.AI are services that both support text input streaming for exactly this use-case. I am not aware of any open-source Incremental TTS (this is the term used in research afaik) model but you can already achieve somthing similar by buffering tokens and sending them to the TTS model on punctuation characters.
- selfhoster11 3y agoElevenLabs only has streaming output available. I've had a look at both recently and ElevenLabs doesn't have streaming input listed as a feature. Would be cool if it had it, though. You could probably approximate this on a sentence level, but you would need to do some normalisation to make the speech sound even.
- column 3y agowhy use this instead of "memgpt run" ?
- taneq 3y agoI dunno, why would you use memgpt run?
- garblegarble 3y agoA major benefit (for me) is that I can build it with "make" and know that it just works. Now perhaps this is a skills issue on my part (because I'm not a Python dev), but I've had endless trouble with Python-based ML projects, with some requiring I use/avoid specific 3.x versions of Python, each project's install instructions seemingly using a different tool to create a virtual environment to isolate dependencies, and issues tracking down specific custom versions of core libraries in order to allow the use of GPU/Neural Engine on Apple Silicon. The whisper and llama.cpp projects just build and run so easily by comparison
- loudmax 3y agoI've had the same experience. One of the things I like most about llama.cpp is the relatively straightforward build process, at least when compared to the mess of Python library requirements you run into if you want to experiment with ML models. Having said that, I have the sense that the ML ecosystem is coalescing around using `venv` as a standard for Python dependencies. Most of the build instructions for Python ML projects I've seen recently begin with setting up the environment using venv, and in my experience, it works fairly reliably. I don't particularly like downloading gigabytes of dependencies for each new project, but that mess of dependencies is what's powering the rapid pace of prototypes and development.
- nwoli 3y agoA few months until this is effectively outlawed if the open weights proposal in 270 days comes into existence
- turned-upright 3y agoCan you elaborate please?
- eurekin 3y agoI think this is relevant: https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/ https://www.whitehouse.gov/briefing-room/presidential-action...
- infecto 3y agoSoliciting Input on Dual-Use Foundation Models with Widely Available Model Weights. When the weights for a dual-use foundation model are widely available — such as when they are publicly posted on the Internet — there can be substantial benefits to innovation, but also substantial security risks, such as the removal of safeguards within the model. To address the risks and potential benefits of dual-use foundation models with widely available weights, within 270 days of the date of this order, the Secretary of Commerce, acting through the Assistant Secretary of Commerce for Communications and Information, and in consultation with the Secretary of State, shall... I believe that is the relevant section, which I am hoping they realize how dumb this is going to be.
- kristopolous 3y agothis has really strong eliza vibes.
- boiler_up800 3y agoWhat’s the best chat interface for llama? I have a 3090 and would love to get one of the models running in my terminal for quick coding tasks.
- ahoka 3y agoThis is the easiest to setup: https://faraday.dev/ https://faraday.dev/ I think Wizard is the “meta” for technical questions now.
- ForkMeOnTinder 3y agoIs this open source? I don't see a Linux download link and I want to try building it for Linux myself
- reqo 3y agoHere is an open source project that supports voice as well: https://github.com/cogentapps/chat-with-gpt https://github.com/cogentapps/chat-with-gpt I think it is meant to be used with ElevenLabs and OpenAI API, but might be easy to configure for use with local Whisper.cpp + llama?
- ForkMeOnTinder 3y agoollama is dead easy to use. It's a single binary that downloads models on demand, like docker downloads images. pacman -S ollama ollama serve ollama run llama2:13b 'insert prompt' https://ollama.ai/ https://ollama.ai/
- riddleronroof 3y agoollama wraps llama.cpp into a docker container, correct? Besides that it seeks like a go server for chat?
- _ea1k 3y agoThe ollama shell is really nice.
- 3y ago
- d3nj4l 3y agoThis makes me wonder, what's the equivalent to ollama for whisper/SOTA OS tts models? I'm really happy with ollama for locally running OS LLMs, but I don't know of any project that makes it that simple to set up whisper locally.
- lhl 3y agoFor SRT, here are some front-ends: https://www.reddit.com/r/OpenAI/comments/163hzhe/recommended_whisper_ai_with_gui_and_gpu_support/ https://www.reddit.com/r/OpenAI/comments/163hzhe/recommended... Also I saw this thing called WhisperScript that looks pretty slick: https://github.com/openai/whisper/discussions/1028 https://github.com/openai/whisper/discussions/1028 That being said, WhisperX isn't that hard to setup. My step by step from a couple months ago: https://llm-tracker.info/books/logbook/page/transcription-test#bkmrk-setup https://llm-tracker.info/books/logbook/page/transcription-te...
- ccoreilly 3y agoWhisper is an STT model, you can use whisperx to transcribe audios locally via the CLI or whisper-turbo.com that runs in the browser. For TTS coqui has the best UX and models for a lot of languages although quality is not on par with commercial TTS providers.
- jcuenod 3y agoI've just been looking for SOTA TTS. I found coqui.ai and elevenlabs.io (and a bunch of others). They're good (and better than older TTS), but I am not fooled by any of them. Do you have recommendations?
- selfhoster11 3y agoGemelo was the other one listed. I doubt you'll get anything sounding more natural than ElevenLabs with the following settings: * Model: Multilingual v2 * All options and sliders to boost similarity: set to max/yes * Stability slider: experimentally set to a value where the model sounds natural enough without destabilising sound output
- lbltavares 3y agoI'm getting a "floating point exception" when running ./talk-llama on arch and debian. Already checked sdl2lib and ffmpeg (because of this issue: https://github.com/ggerganov/whisper.cpp/issues/1325 https://github.com/ggerganov/whisper.cpp/issues/1325) but nothing seems to fix it. Anyone else?
- oakhaven 3y agoI was struggling with the same error on PopOS 22.04, and this helped: https://github.com/ggerganov/whisper.cpp/issues/352#issuecomment-1369024452 https://github.com/ggerganov/whisper.cpp/issues/352#issuecom... I'm not sure what changed, but basically I purged ffmpeg and libsdl2-dev and the `make` in the root of the repo. Then I installed libsdl2 and ffmpeg and `make talk-llama`. It's quite slow on 4 core i7-8550U and 16 GB of RAM. basically, in the root of the repo: $ sudo apt purge ffmpeg $ make clean $ git pull $ make $ sudo apt install libsdl2-dev $ make talk-llama $ ./talk-llama -mw ./models/ggml-small.en.bin -ml ../llama.cpp/models/llama-2-13b.Q4_0.gguf -p "t0mk" -t 8\n\n HTH
- ggerganov 3y agoHeh, funny to see this popup here :) The performance on Apple Silicon should be much better today compared to what is shown in the video as whisper.cpp now runs fully on the GPU and there have been significant improvements in llama.cpp generation speed over the last few months.
- A4ET8a8uTh0 3y agoYou are kinda famous now man. Odds are, people follow your github religiously.
- MuffinFlavored 3y agoIs ggerganov to LLM what Fabrice Bellard is to QuickJS/QEMU/FFMPEG?
- actionfromafar 3y agoThat's a big burden to place on anyone.
- boesboes 3y ago13 minutes between this and the commit of a new demo video, not bad :D And impressive performance indeed!
- tomtom1337 3y agoIs it just me, or is the gpu version actually slower to respond?
- tomtom1337 3y agoAh, forget the other message, I watched the videos in the wrong order! And I can’t delete or edit using the Hack app!
- deleted 3y ago[deleted]
- 3y ago
- onemoresoop 3y agoCould anyone explain the capability of this in plain English? Can this learn and retain context of a chat and build on some kind of long term memory? Thanks
- i-use-nixos-btw 3y agoI'm not an LLM expert by any means but here is my take. It's Speech Recognition -> Llama -> Text to Speech, running on your own PC rather than that of a third party. The limitations on the context of the LLM are that of the model being used, e.g. Llama 2, Wizard Vicuna, whatever is chosen, in whatever compatible configuration is set by the user regarding context window etc, and given a preliminary transcript (as the LLM doesn't "reply" to the user in a sense, it just predicts the best continuation of a transcript between the user and a useful assistant, resulting in it successfully pretending to be a useful assistant, thus being a useful assistant - it's confusing). I can imagine that it's viable to get that kind of behaviour by modifying the pipeline. If the architecture was instead Speech Recognition -> Wrapper[Llama] -> Text 2 Speech, where "Wrapper" is some process that lets Llama do its thing but hooks onto the input text to add some additional processing, then things could get interesting. The wrapper could analyse the conversation and pick out key aspects ("The person's name is Bob, male, 35, he likes dogs, he likes things to be organised, he wants a reminder at 5pm to call his daughter, he is an undercover agent for the Antarctic mafia, and he prefers to be spoken to in a strong Polish accent") and perform actions based on that: - Set a reminder at 5pm to call his daughter (through e.g. HomeAssistant) - Configure the text-2-speech engine to use a Polish accent - Modify the starting transcript for future runs: - Put his name as the human's name within the underlying chat dialogue - Provide a condensed representation of his interests and personality within the preliminary introduction to the next chat dialogue This way there's some interactivity involved (through actions performed by some other tool), some continuity (by modifying the next chat dialogue) and so on.
- hhjinks 3y agoI've been wondering about how feasible it is to simulate long term memory by running multiple LLMs at the same time. One of them would be tasked with storing and retrieving long term memories from disc, so it'd need to be instructed about some data structure where memories were persisted, and then you'd feed it the current context, instructing it to provide a way to navigate the memory data structure to any potentially relevant memories. Whatever data was retrieved could be injected into the prompt to the next LLM, which would just respond to the given prompt. No idea what sort of data structure could work. Perhaps a graph database could be feasible, and the memory prompt could instruct it to write a query for the given database.
- birdyrooster 3y agoI don’t want to talk to anything in my terminal. It’s a shitty interface for that.
- horsellama 3y agocan this be used to talk with local documents? say I have a research paper in pdf, can I ask llama questions about it?
- simonw 3y agoNo. This is a demo of directly prompting a model using voice-to-text. If your model has a long enough context (models like MistralLite can do 32,000 tokens now, which is about 30 pages of text) you could run a PDF text extraction tool and then dump that text into the model context and ask questions about it with the remaining tokens. You could also plug this into one of the ask-questions-of-a-long-document-via-embedding-search tools.
- wahnfrieden 3y agoHow does this choose when to speak back? (Like is it after a pause, or other heuristics.) I tried looking through the source to find this logic.
- beiller 3y agoIt waits for sufficient silence to determine when to stop recording the voice and send it to the model. There is other modes in the source as well and methods of setting the length of silences in order to chunk up and send bits at a time, but I imagine that is either work in progress or not planned for this demo.
- wahnfrieden 3y agoThanks I was surprised they didn’t combine this work with the streaming whisper demo. So I guess I will implement that for iOS/macos (streaming whisper results in realtime without waiting on an audio pause, but as you say using the audio pauses and other signals like punctuation in the result to determine when to llm complete; makes me also wonder about streaming whisper results in to the llm incrementally before ready for completion)
- beiller 3y agoIt may be using the streaming demo. The reason I know to answer your question is that I had modified the streaming demo myself for personal use before. I think there is bugs in the silence detection code (as of a few months back, maybe fixed now). Maybe what we are seeing in this demo is just the "silence detection" setting to be waiting for very long pauses, I believe its configurable.
- wahnfrieden 3y agoI added libfvad
- cjbprime 3y agoWould it be possible to reduce lag by streaming groups of ~6 tokens at a time to the TTS as they're generated, instead of waiting for the full LLM response before beginning to speak it?
- ggerganov 3y agoYes, I was planning to do this back then, but other stuff came up. There are many different ways in which this simple example can be improved: - better detection of when speech ends (currently basic adaptive threshold) - use small LLM for quick response with something generic while big LLM computes - TTS streaming in chunks or sentences One of the better OSS versions of such chatbot I think is https://github.com/yacineMTB/talk https://github.com/yacineMTB/talk. Though probably many other similar projects also exist by now.
- rjtavares 3y ago> use small LLM for quick response with something generic while big LLM computes Can't wait for poorly implemented chat apps to always start a response with "That's a great question!"
- generalizations 3y agoI keep wondering if a small LLM can also be used to help detect when the speaker has finished speaking their thought, not just when they've paused speaking.
- drunkenmagician 3y ago
- rgbrgb 3y agovery sick demo! if anyone wants to work on packaging this up for broader (swiftUI/macos) consumption, I just added an issue https://github.com/psugihara/FreeChat/issues/30 https://github.com/psugihara/FreeChat/issues/30
- human183729153 3y agoWhat are currently the best/go-to approaches to detect the end of an utterance? This can be tricky even in conversations between humans, requiring semantic information about what the other person is saying. I wonder if there’s any automated strategy that works well enough.
- selfhoster11 3y agoVoice activity detection (referred to as VAD for short) is what you're looking for. Personally, I'd prefer a hold-down hotkey a la press-to-speak for more immediate UI feedback, but both work pretty well.
- wxnx 3y agoThis is cool. I hooked up Llama to an open-source TTS model for a recent project and there was lots of fun engineering that went into it. On a different note: I think the most useful coding copilot tools for me reduce "manual overhead" without attempting to do any hard thinking/problem solving for me (such as generating arguments and types from docstrings or vice-versa, etc.). For more complicated tasks you really have to give copilot a pretty good "starting point". I often talk to myself while coding. It would be extremely, extremely futuristic (and potentially useful) if a tool like this embedded my speech into a context vector and used it to as an additional copilot input so the model has a better "starting point". I'm a late adopter of copilot and don't use it all the time but if anyone is aware of anything like this I'd be curious to hear about it.
- BigRedDog1669 3y agoTacoma
- danielEM 3y agoAll I need is ... Vulkan support :) pls, pls, pls :)
- SillyUsername 3y agoDoes anybody have a quick start for building it all in Windows for this? I could probably check it out as a VS project and build but I'm going to bet since it's not documented it's going to have issues specifically because the Linux build instructions are the only ones that are a first class citizen...
- deleted 3y ago[deleted]