18 ms·
Open-sourcing AudioCraft: Generative AI for audio
- pmarreck 3y agoWouldn't Pandora's possibly vast library associating textual descriptions with music be the ideal training data for something like this?
- chrisjj 3y agoYes... except that Pandora's library does not include the music.
- s1k3s 3y agoThe demos are great. Could someone explain what’s in it for Meta open sourcing all these models?
- bick_nyers 3y agoIf anything it makes them appear to be one of the best places to work at to do research. Could be them playing the long game.
- Saturdays 3y agotheoretically what's in it for them is that people will build content faster and with less barriers for eventual consumption on their platforms
- jdadj 3y agoCommoditize Your Complement? https://gwern.net/complement https://gwern.net/complement
- maximus-decimus 3y agoWhat is it a complement to though?
- jwestbury 3y agoContent is a complement to social media.
- CrypticByte87 3y agoMeta has several of the biggest UGC platforms, and in this case the complement is content itself. Reels with autogenerated (and royalty free) background music is the obvious example but I'm sure there are more. Maybe creative for ads as well?
- raincole 3y agoIn short term it's social media, because people will share whatever they generate on social media. But I don't think it's a very strong incentive to invest in AI for Meta.
- jononor 3y agoTo Metaverse access. Filling the metaverse with engaging interactive 3D content is an insane job with 2020 technology. It requires a huge amount of a range of skilled labor to create 3D models, soundtracks, NPC dialog, visuals et.c. to make a compelling experience. In 2030 that may have been reduced to that everyone with creativity and Internet access can do it. Sure, most of it will be silly things - but so is social media today, does not make it any less of a commercial success. And there will be be millions of semi-pro creators to create the things with higher production value, like with videography today.
- gostsamo 3y agoThey want to comoditize the offerings of OpenAI, Google, MS, and Apple. Also, they gain mindshare and good will after years of bad publicity. Some back contributions might help them improve the models for free. If they just keep their models, people won't be interested and will build over ChatGPT or Bard.
- vasili111 3y agoThey will own most popular open models so they can dictate the direction in open source AI.
- jamil7 3y agoNot a fan of Meta, but haven't they generally been pretty forward with open sourcing their tech?
- unnouinceput 3y agoThe same move that Microsoft did back in 90's to kill Netscape. Make your product the one available to masses, next generation of users will be using your product.
- zyang 3y agoI was just thinking how Google made Android free to check Microsoft. This is Meta checking Google.
- roody15 3y agoChecking Google or OpenAI (or both?)
- ipaddr 3y agoChecking OpenAI. Google is still playing checkers.
- mcbuilder 3y agoI just can't get how bad Google is doing. They have a ton of top researchers, papers, money, just no good LLMs. It's like OpenAI was first to the punch, and everyone else just saw $$$. Meta was smart to go down this open source road, as the masses will start training their llamas one way or another. Personally I believe the "intelligence" aspect will asymptote, so even having exclusive access to a "super AI" (i.e. hypothetical 1T parameter model like a GPT5) won't be that much of a step behind the lesser AIs, and as soon as you grant access to the masses they will start to use some transfer learning to make their "lesser" models better. AI applications though still need a lot of work. The models are smart or general purpose enough to be useful to the average person out of the box.
- rvnx 3y agoThe problem also is that Google is making lot of grandiose announcements about tools and models that nobody can see nor use. This is a serious credibility problem in the long-term.
- ipaddr 3y agoThey haven't opened sourced much. Open models/closed weights restrictive non-commercial license is something I guess. They are trying to kill the market before they get left out.
- kypro 3y agoA competitive opensource project basically destroys the pricing power of all closed-source alternatives. If you're a company and wanted to integrate an LLM into your product if the choice is between several equally good models, but one is free and open-source which would you pick? Aside from keeping competition at bay, this move also gives Meta leverage because ecosystems are now being built around their projects. If these models see wide-scale adoption they could later launch AudioCraft+ as a licensed version with some extra features for example. Alternatively, they might offer support or hosting for their open source projects. Right now though I think the primary benefit of these open sourced models is to attract talent. If Meta is seen as one of the leaders in AI then researchers will want to work for them simply for the prestige. Arguably one of the reasons Meta has been behind so many awesome projects like PyTorch and React over the last decade was because they were seen as the cool place for recently graduated, but talented software engineers to work in ~2010.
- nprateem 3y agoIf people love hanging out with chatgpt or bard, they won't be wasting their precious little eyeballs on FB/Insta
- hereonout2 3y agoWas asking myself the same earlier, I'm sure it is largely to do with publicity and the fact that selling these services is not their core business. At the very least releasing this stuff probably won't damage their core business but will take the sheen off of some other big names. I wondered though, generative AI is hurling us into a world where we'll need more mechanisms to sort real from fake, provenance will play a large part, and meta's platforms could be part of the answer. i.e. content linked to actual verifiable people.
- klapinat0r 3y agoSomewhat relevant, Yann LeCun insisted the research should be open sourced. At least in an academic sense. He touches on it briefly in this podcast episode: https://www.therobotbrains.ai/who-is-yann-lecun https://www.therobotbrains.ai/who-is-yann-lecun
- squidsoup 3y agoThe demos are, unsurprisingly, soulless muzak. This contributes nothing to our culture.
- deleted 3y ago[deleted]
- o_____________o 3y agoSkip to the samples: https://audiocraft.metademolab.com/audiogen.html https://audiocraft.metademolab.com/audiogen.html
- operator-name 3y agoAnd https://audiocraft.metademolab.com/musicgen.html https://audiocraft.metademolab.com/musicgen.html The samples included in the press release are quite impressive to my ears, but the other samples (especially from AudioGen) have a hint of artificially. As usual the music is quite repetitive, but I'm looking forward to tools that simplify changing the prompt whilst it generates over a window. I can only imagine the consequences for royalty free music. Edit: the "Text-to-music generation with diffusion-based EnCodec" samples are quite impressive.
- padjo 3y agoQuite impressive although if these are the cherry picked examples the average output must be pretty weak! Nothing catchy about most of these examples and the reggae one is pretty lame.
- justinclift 3y agoWonder how far off the whole "generate music based on your existing music library" thing is going to be? That'll make musicians happy with big tech as well, just like artists are. *sigh*
- zitterbewegung 3y agoWhy not do generate music you like which wouldn’t need you to upload your library and would have RLHF baked in.
- ElFitz 3y agoSomething like the algorithm TikTok uses. First probing by offering a variety of content that should match based on what little information you have on the user (ip location, locale, etc). Then use the user’s action to iteratively refine your classification, until you end up with something tailor-made.
- zitterbewegung 3y agoUh more like Reddit with up and down and also how long you listen.
- LewisVerstappen 3y agoThe Record labels are far, far more litigious than the art community.
- PaulDavisThe1st 3y agoThey can't litigate a person doing this at home, and never redistributing. I suppose they might try, anyway.
- kmeisthax 3y agoThe RIAA pioneered copyright enforcement at the individual level back in the 2000s, they absolutely would try to sue downstream AudioCraft users.
- gavman 3y ago> MusicGen, which was trained with Meta-owned and specifically licensed music, generates music from text-based user inputs, while AudioGen, which was trained on public sound effects, generates audio from text-based user inputs. Meta is really clearly trying to differentiate themselves from OpenAI here. Open source + driving home "we don't use data we haven't paid for / don't own".
- itsyaboi 3y agoBully "Open"AI into rebranding.
- deleted 3y ago[deleted]
- kmeisthax 3y agoThis is purely a function of everyone remembering the RIAA's decade-long campaign to prevent people from taking the music they had rightfully stolen. As far as I'm aware LLaMA was trained on "publicly available data"[0], not "licensed data". Furthermore, MusicGen's weights are licensed CC-BY-NC, which is effectively a nonlicense as there is no noncommercial use you could make of an art generator[1]. This is not only a 'weights-available' license, but it's significantly more restrictive than the morality clause bearing OpenRAIL license that Stability likes to use[2]. [0] https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [1] https://github.com/facebookresearch/audiocraft/blob/main/LICENSE_weights https://github.com/facebookresearch/audiocraft/blob/main/LIC... [2] These are also very much Not Open Source™ but the morality clauses in OpenRAIL are at least non-onerous enough to collaborate over.
- eropple 3y ago> MusicGen's weights are licensed CC-BY-NC, which is effectively a nonlicense as there is no noncommercial use you could make of an art generator How do you figure? Have you never just...made stuff to make stuff?
- Culonavirus 3y agoHere's a different question: Can you use the audio output this produces for anything else other than "research purposes"?
- sangnoir 3y agoYou can - as long as it's not commercial. It's a broad definition, but a good eule of thumb is if you're not directly making money out of the generated audio. They may still cone for you if you're making money indirectly, so consult a lawyer.
- blackkettle 3y agoI can see some fantastic uses for this in generating complex acoustic environments to layer over TTS or real recordings for speech-to-text model training. I wonder if that is occupying some kind of gray-area. For example you have 1000hrs of clean speech from the librispeech corpus. It would be trivial to use this tool and available weights to generate background noise, environmental noise and the like, and then layer this with the clean speech to cheaply train a much more robust model. The environmental audio you create would never be directly shared or sold, but it would impact the overall quality of the STT model that you train from the combined results.
- deleted 3y ago[deleted]
- vasili111 3y agoIs there place where I can check how it works? Like give my input and get output audio?
- operator-name 3y agoThe model cards from the repo[0] link to Colab and HF spaces. [0]: https://github.com/facebookresearch/audiocraft#models https://github.com/facebookresearch/audiocraft#models
- emporas 3y agoAudiocraft+, don't forget the plus, on github has a collab notebook based on audiocraft, and a webui to use. It is pretty awesome!
- trojan13 3y agoFinally, a way to fulfill my childhood dream of composing a symphony of rubber ducks honking. Bach would be proud. /edit On a more serious node. I already see the 24/7 lofi girl streaming generated music. The sample[1] on lofi sounds pretty good. [1]https://dl.fbaipublicfiles.com/audiocraft/webpage/public/assets/audios/musicgen_mono/long_sample_000.mp3 https://dl.fbaipublicfiles.com/audiocraft/webpage/public/ass... "Lofi slow bpm electro chill with organic samples"
- squidsoup 3y ago> Finally, a way to fulfill my childhood dream of composing a symphony of rubber ducks honking. Samplers have been around since the 70s.
- painted-now 3y agoI also like some of the generated examples. Can I haz full version of Bach + `An energetic hip-hop music piece, with synth sounds and strong bass. There is a rhythmic hi-hat patten in the drums.` please? (https://dl.fbaipublicfiles.com/audiocraft/webpage/public/assets/audios/musicgen_chroma_mono/bach_an_energetic_hip-hop_music_piece,_with_synth_sounds_and_strong_bass._there_is_a_rhythmic_hi-hat_pattern_in_the_drums.wav https://dl.fbaipublicfiles.com/audiocraft/webpage/public/ass...) ?
- jasonjmcghee 3y agoThe difference between MBD-EnCodec and EnCodec is pretty interesting. MBD variant sounds more like a professional studio recording, while the EnCodec feels like a richer sound. Curious if I’m alone in that. (At the bottom https://audiocraft.metademolab.com/musicgen.html https://audiocraft.metademolab.com/musicgen.html) For what it’s worth though, the voice based examples sound dramatically better with MBD https://audiocraft.metademolab.com/encodec.html https://audiocraft.metademolab.com/encodec.html
- operator-name 3y agoMBD definitely sounds like it was recorded in dead room, whereas plain EnCodec has been mixed but includes some artificial noise.
- wg0 3y ago"What a time to be alive!"
- TheAceOfHearts 3y agoAudioGen seems really fascinating. I have some dumb questions. While the datasets used for training AudioGen aren't available, is there any kind of list where one can review the tags or descriptions of the sounds on which the model was trained? Otherwise how do you know what kinds of sounds you can reasonably expect AudioGen to be capable of generating? And what happens if you request a sound which is too obscure or something not found in the dataset? What are AudioGen's capabilities regarding spatial positioning? First example: can it generate a siren that starts in front and moves left to right and complete a full circle around the listener? Second example: can it do the same siren but on the Y axis, so it start at the front, it goes over the listener and then it goes under them to complete the circle?
- westurner 3y agoGenerative AI > Modalities > [Music,]: https://en.wikipedia.org/wiki/Generative_artificial_intelligence#Modalities https://en.wikipedia.org/wiki/Generative_artificial_intellig...
- dragonwriter 3y agoThe license of the model weights is CC-BY-NC, which is not an open source license. The code is MIT, though.
- archontes 3y agoIt's unlikely that model weights can be copyrighted, as they're the result of an automatic process.
- dragonwriter 3y ago> It’s unlikely that model weights can be copyrighted, as they’re the result of an automatic process. If they can’t for that reason alone, then the model is a mechanical copy of the training set, which may be subject to a (compilation) copyright, and a mechanical copy of a copyright-protected work is still subject to the copyright of the thing of which it is a copy. OTOH, the choices made beyond the training set and algorithm in any particular training may be sufficient creative input to make it a distinct work with its own copyright, or there may be some other basis for them not being copyright protected. But the mechanical process one alone just moves the point of copyright on the outcome, it doesn’t eliminate it.
- archontes 3y agoThat isn't necessarily true. You're saying that the model weights would be a derivative work. Derivative work is insufficiently transformative to fully distinguish the output from the input. In this instance, it's very likely that the process is sufficiently transformative. A set of model weights look nothing like the Mona Lisa, nor can they be directly transformed back into it. What it is NOT is the product of a creative process on the part of a human, and is thus ineligible for copyright. It is as though we are able to distill meaning using an automatic process. Copyright doesn't protect meaning, only expression, and it only protects expressions that were generated by humans. The network itself might be patentable if it isn't obvious. If a network was copyrighted and it was found that the function of the network was inseparable from its expression, it would no longer be eligible for copyright on that basis. I think people often misunderstand the application of copyright to GPL code in this way. Now, will any of this stop people from CLAIMING copyright? No. It'll have to be fought out in courts.
- pcwelder 3y agoDiffusion model is now SOTA in audio and image generation. Has anyone given it a shot on texts? Audio is more similar to language than images because of more stronger time dependency. The paper says the critical step they took for making diffusion model work for audio was splitting the frequency bands and applying diffusion separately to the bands (because full band model had limitations due to poor modeling of correlations between low frequency and high frequency features). I think something could be done on text side as well.
- polygamous_bat 3y agoThere are two problems with this. Diffusion models work on a single rule of thumb: if you keep adding small, noisy gaussian steps to a "nice" distribution many times, you get uniform gaussian at the end. So, for text: a) what is the equivalent of a small, noisy step? and b) what is the equivalent of a uniform gaussian in language space? If you can solve a and b, you can make diffusion work for text, but there hasn't been any significant progress there afaik.
- joshstrange 3y agoI'm looking forward to playing with the M1 Mac apps/cli-tools that will probably come out for this in the next week or so! Being able to run this stuff locally is a lot of fun.
- illwrks 3y agoAre the M1 macs capable enough? I'm eyeing and upgrade in the coming months and I'm curious if a MacBook would be suitable
- joshstrange 3y agoI've run Stable Diffusion locally (both from the cli and later using GUI wrappers) and that used my GPUs, I've also run Llama locally but I believe that was on the CPU (I used both llama.cpp, cli, and Ollama, gui). So to sum it up: yes? Or at least it's good enough for me.
- illwrks 3y agoGreat thanks!
- deleted 3y ago[deleted]
- Palmik 3y agoMaybe this will finally lead to high-quality open-weights solution for TTS generation.
- deleted 3y ago[deleted]
- reducesuffering 3y ago"Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence. And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs." -Yuval Noah Harari
- ironborn123 3y agoMaybe this us vs them mentality is the biggest bottleneck. If instead you consider that this new form of 'alien' intelligence is actually a descendant of human intelligence, that we are raising a new species which will inherit what humans have built (ideally only the good parts) and then improve upon it further.. It may sound grandiose, but that perspective changes everything.
- Pannoniae 3y ago"generating new music in the style of existing music" will probably be a huge field soon. I can't wait for it to happen, it's a low-cost way of producing even more music to listen to.
- gilmore606 3y ago> I can't wait for it to happen, it's a low-cost way of producing even more music to listen to. I can't really understand this. I'm a DJ and a huge music nerd, and I spend a lot of time every week discovering new music from the past 100 years and all over the world, and I'm constantly struck by _how much of it there is_. I've spent weeks just digging through psych-funk records from West Africa from the 1970s. How can you have the impression we're so desperate for more music that we need computer programs to generate it for us?
- Pannoniae 3y agoYes but it's not really fungible. My favourite artist is Fats Waller and they don't make anything like that anymore. Most people are only interested in some category of music, not all of it.
- cooper_ganglia 3y agoMusic is self-expression. I don’t always identify entirely with others. I always identify with my self. Having music generated for you on such a personalized level is an attractive prospect. I don’t think this replaces “100% organic, human-made” music, though. I think there’ll always be a reason to listen to music made by other people. But I think this changes the landscape of how and why people create music to begin with. It certainly will devalue existing music, since everyone has something they may prefer that they can generate instantly. I think generative AI is a terrible technology for artists who want to make money from their art, but in my personal opinion, I strive for a world where art isn’t a transaction, but a gift of human expression and connection. A world where art is appreciated for the emotion, stories, and ideas it conveys rather than the monetary value it holds. Generative AI might disrupt the traditional economic models in the art world, but it also opens up new opportunities for creative exploration and personal expression. It’s a challenging evolution, but one that could potentially democratize art, making it more accessible and personal than ever before! Bring on the Renaissance: Part 2!
- davidw 3y agoIs there a way to try this out? I didn't see one, but didn't look too hard.
- smallerfish 3y agoYes. Installation instructions on the front of the repo, then click on the model readme for sample getting started code (10 lines of python and you get output.)
- smallerfish 3y agoThis is Spotify's route to profitability - the Netflix model of generating their own "content" (/music), and not having to pay the labels. Premium plans for us music nerds who want a human at the other end, regular plans for plebs who just want to fill the silence with something agreeable.
- CharlesW 3y agoAlthough I think AI-generated and AI-augmented (using voice cloning, etc.) artists are a given, for Spotify to stop paying labels they'd have to be able to remove all non-Spotify content from their streaming catalog. That doesn't seem like a possibility in our lifetimes. (Also, Spotify hasn't even been able to build a sustainable business on podcasts, which they copy to their closed platform for free.) It's an interesting thought experiment, though. I can imagine that "environmental audio" companies like Muzak have about 5 years left before they either adapt or die. What other kinds of companies are in trouble?
- smallerfish 3y agoTheir current pay structure is royalties, i.e. per listen. If they can route their audience to mostly AI generated content in time (say, 5-10 year transition), and it's just as good for most people, then they can negotiate much lower prices with the labels. We all grumble about Netflix being full of junk, but most of us are still subscribers, despite a sparse catalog of big name movies.
- hobofan 3y ago> then they can negotiate much lower prices with the labels Or alternatively, if the labels are not stupid, they'll negotiate for a higher price per listen (or similar), as they are still as essential to the service as before.
- skybrian 3y agoAs an amateur musician I’m wondering if there are any of these audio generators that you can give a tune or chord progression to riff on. ABC format maybe? There are lots of folk tunes on thesession.org. Could you generate a rhythm track? Ideally you could make songs one track at a time, by giving it a mix of the previous tracks and asking it to make another track for an instrument. Or, give it a track and ask it to do some kind of effect on it. Another interesting use might be generating sound samples for a sampled instrument.
- emporas 3y agoIf you mean to give a source of melody of 30sec and extend that melody into a full song, yes MusicGen can do that. There are two ways to extend a song based on a melody: 1) give a sample, and continue the song from that sample as close as possible, and 2) give a melody as an inspiration. They both work in varying degrees of success. Audiocraft on github, issues or discussion sections have a lot of questions answered.
- chrisjj 3y agoEvidence? None of the demos suggest that is true.
- emporas 3y agoDid they change the base model? If not, then the audiocraft_plus, is based on audiocraft, and it creates music of length close to 5 minutes. I don't know if audiocraft_plus incorporates all three modalities of the release, MusicGen, AudioGen, and EnCodec. It uses MusicGen for sure, all four models, small, medium, large and melody. https://github.com/GrandaddyShmax/audiocraft_plus https://github.com/GrandaddyShmax/audiocraft_plus
- chrisjj 3y agoBit is that "extension" to 5mkns more than just the repeat of e.g. 15secs heard in the demos?
- praveenhm 3y agodoes this model help in TTS(text-to-speech), badly needed only free option is bark and tortise TTS right now.
- aedocw 3y agoCoqui-TTS with vtck/vits is very good right now. Not as good as eleven labs or coqui studio, but for fast open TTS it's pretty good, in case you're not familiar with it. It will be great when there's eventually something open that competes with the closed models out there.
- praveenhm 3y agoExcellent, I will take look into this.
- RobotToaster 3y agoCC-BY-NC Isn't an open source licence, it violates point six of the open source definition https://opensource.org/osd/ https://opensource.org/osd/
- hackernewds 3y agowho gets to declare what is the "open source definition" and why?
- BearhatBeer 3y ago[dead]
- frognumber 3y agoIn my opinion, the Free Software Foundation, ironically, since they invented the movement, with open source starting out as a tacky rip-off with the ethics stripped out. After decades, open source converged on free software. More popular opinion is OSI: https://en.wikipedia.org/wiki/Open_Source_Initiative https://en.wikipedia.org/wiki/Open_Source_Initiative They were founded by the persons who (claimed to have) invented the term in order to steward it. It's the same definition as the FSF.
- xdennis 3y agoThe people who created the term: the Open Source Initiative. Before, people most often used "free software" as defined by the free software movement, but some disliked this term because it's confusing (most think "free" means no money) and perceived to be anti-commercial. The term "open source software" was chosen and given a precise definition. It's dishonest, then, for people to use the term "open source software" with a different interpretation when it was specifically chosen to avoid confusion.
- OrangeMusic 3y ago> It's dishonest, then, for people to use the term "open source software" with a different interpretation I disagree. You're saying that they "invented" the term, but it's a very generic term. The source is open, it's open source. I bet people were using the term before they claim they invented it. In that context, it is very fine to use a different definition and in fact here's my definition, and I guess most people (maybe not on HN) share it: if the source is visible by the general public, it's open source. For what you mean, I use "FLOSS".
- bottlepalm 3y agoAnyone feel like with the flood of AI generated content there's a risk of the past being 'erased'. Like in 10 years we won't be able to tell if any information from the past is real or fake - sounds, pictures, videos, etc.. Like we need to start cryptographically signing all content now if there's any hope of being able to verify it as 'real' 10 years from now.
- seydor 3y agoThe past ended in 2022
- shon 3y agoThis ^^
- apabepa 3y agoOr is the past endlessly rehashed with AI generated content?
- bottlepalm 3y agoAgree. Any video/image/text created post-2022 is now suspect of being AI generated (even this comment). And without any 'registering' of pre-2022 content, we can easily lose track and not really know what from pre-2022 is authentic or not. Maybe it's not a big deal to 'lose' the past, maybe landfills will be mined for authentic content.
- seydor 3y agoThere should be a canonical copy of the 2022 internet that is verifiable. Archive.org is not enough
- jeffwass 3y agoI’ve been wondering about this and real video evidence (eg dashcam or cctv) being refuted in court for inability to show it’s not deepfaked.
- crazygringo 3y ago
- jeffbee 3y agoThe fact that it generates a song for the prompt "Earthy tones, environmentally conscious ... organic instrumentation" goes a long way to proving that English words no longer mean anything particularly.
- camillomiller 3y agoI think "song" should go in quotes
- TrackerFF 3y agoI've played guitar 25 years, and it's funny how the music community has been using all kind of words to describe music or tone. Describing certain tone as "hairy", "mushy", "wooly", "airy", "buttery", etc. is just very common.
- jeffbee 3y agoSure, there's jargon. But these words don't describe the music. They describe words associated with the kinds of people who would listen to it (according to the biases of the language model). As a description of the music, it's meaningless. If a person was asked to name some "environmentally conscious" music they could just as easily veer over to hardcore straight edge.
- squeaky-clean 3y agoIt works if you don't need a specific type of music but just something to fit a vibe. For example in an indie game or film project. Maybe level 3 is a jungle area with a plant based species of NPCs and the music should be "Earthy tones, environmentally conscious ... organic instrumentation"
- chrisjj 3y agoThat sort of presumes those words had any effect on the output. We might know more had it generated a song as you said, but in fact it generated only an instrumental.
- bestcoder69 3y agoAnyone know if there are ways, as-is, to speed this up on Apple Silicon? This setup takes 5 minutes: - Mac Studio M1 Max 64GB memory - running musicgen_app.py - model: facebook/musicgen-medium - duration: 10s
- mk_stjames 3y agoI see from the musicgen.py- >.if torch.cuda.device_count(): >. device = 'cuda' >. else: >. device = 'cpu' So pytorch will fall back to CPU on a Apple Silicon. Ideally it would use Metal for acceleration (MPS) instead of just plain 'CPU', but if you replace CPU with MPS you'll probably run into a few bugs due to various Autocast errors and I think some other incompatibility with Pytorch 2.0. At least that is what I ran into last time I tried to speed this up on an M1. It's possible there are fixes.
- bestcoder69 3y agoSame here (mps errors). I tried after the initial musicgen release. I’ll have to check again, but I remember AFAICT my hardware wasn’t getting saturated, so maybe there’s headroom for mac cpu performance. And of course in the meantime I’ll be refreshing the ggml github every day
- camillomiller 3y agoAll very interesting, but how would a musician ever be interested in creating the result of "Pop dance track with catchy melodies, tropical percussions, and upbeat rhythms, perfect for the beach"? This stuff will create a lot of Muzak for sure. Actually turning into anything useful for musician? I honestly doubt it, and I'm happy if it stays that way. Saying that engineers don't understand the arts is a bit of a trite generalization, but reading the way Meta markets these "music making" contraptions is really cringe inducing. Have you ever, at least, listened to some music?
- zapdrive 3y agoIncoming strike by American Federation of Musicians in 3, 2, 1.... How many jobs would this thing take away? One of the biggest time consuming in any video production is post production audio including background music, audio, Foley etc. This will automate almost of it!
- chrisjj 3y agoThis would take away all the jobs producing such low-quality low-fi artifact-laden background music... if any existed.
- aimor 3y agoThis is great, I've been wanting sound effect generation for years. I spent a lot of time trying to get WaveNet working well, eventually just dropped the project after mediocre results. With AudioGen I'm generating a sample in less than a second.
- johnwheeler 3y agoHow does meta plan to make money from this open source?
- tibanne 3y agoCan I generate a reading by someone if I have a lot of their voice samples with this? Or is there a better tool for doing such a thing?
- lm28469 3y agoelevenlab and dozen others already do that
- parekhnish 3y agoGenerative AI for images and music produce pixels and waveform data, respectively. I wonder if there is research into "procedural" data; so in this case, it would be SVG elements and, perhaps, MIDI data respectively. I know training data would be much more harder to get, (notwithstanding legal ramifications), but I think that creating structured, procedural data will be much more interesting than just the final, "raw" output!
- IAmGraydon 3y agoI've thought about this too. The instruments themselves can be synthesized for extremely high quality audio. All we need is the musical structure - the MIDI.
- RyanAdamas 3y agoFor my two cents; the goal of our human pursuit is to advance our technologies and systems to the point we can all live carefree lives where the focus of our pursuits is self defined. It is not to "own more rights" to shit. Sorry, but no one actually owns what they create, that's the point of creation. And if you take issue with the term create, that only reinforces my point. We're all influence machines, input and output, the future should not be about preserving some peoples rights to limit our collective advancements over their personal wants. Tough shit.
- deleted 3y ago[deleted]
- lm28469 3y ago> For my two cents; the goal of our human pursuit is to advance our technologies and systems to the point we can all live carefree lives where the focus of our pursuits is self defined. Our overlords didn't get the memo
- samstave 3y agoI disagree - the human pursuit is artificially, or organically, besting its own pattern recognition wet-ware. If we can off load the mundane (survival) aspect of our pattern recognition engine, then maybe we can use those cycles in lofty pursuits - this is the victorian fallacy. - Break everything down - its all patterns all the way, and how we process them... we are letting AI take on an aspect of ourself (pattern recog ;; tokens;; and prediction) That is, if applied to self-preservation, is the essence of sentience. (I think! therefore, I am, and I will prevent you from making me NOT)
- whycome 3y agoCulture is based on the creation of those that came before. Genres of music are created as people try to mimic styles of those before…one could argue that they “trained themselves on the previous dataset”. No one creates anything in a vacuum. They utilize things that we collectively have contributed. Hell, language and writing is the open source thing that we collectively own that ppl use to create their stuff. The creation came after going to schools that we collectively pay for. And travelling on shared roads. It’s the standing on the shoulders of giants — except its really just stacked people all adding their bits.
- avereveard 3y agohttps://www.audiogen.co/ https://www.audiogen.co/ related? unrelated? big fight coming up?
- JHonaker 3y agoGet ready for the next generation of Muzak
- wdb 3y agoI am curious if it can generate audio for a country. For example, the sirens sound in the sample doesn't like I siren I would recognise. Sounds like an American one?
- samstave 3y agoThats an interesting q ; What about the ring tone, busy tone, disconnected tone for any country over time. 2600 vibes (pun)
- aimor 3y agoI tried it out (american, british, korean, italian, japanese) and couldn't really get any control. Sometimes the american siren would sound different, but asking for a siren of a specific country would just give the american sound. Maybe better prompting would help. I used "isolated american ambulance siren no traffic".
- naillo 3y agoI wish people made unconditional predictive models for music instead of text-to-music ones. Would be so cool to give an input 'inspiration' track that it 'riffs' a continuation to. That's usually what I want, just continue this track it's too short that's what I want to hear more of. (That said this is super cool though.)
- janalsncm 3y agoTheoretically this is very possible using their techniques. They tokenize the audio and learn next tokens autoregressively. Instead of text tokens -> audio tokens as input, just tokenize a prior song and continue it.
- bulbosaur123 3y agoOh my god, some of these tracks actually SLAP. Like for real. The last bastion of human creativity is about to be defeated.
- IAmGraydon 3y agoWhich ones slap? I want them to, but what I'm hearing is only OK. I think this could generate some interesting starting points for me when I'm stuck, though.
- peteforde 3y agoI just ran all of the cited installation steps, which appear to have been successful... but I am now experiencing a profound sense of "now what?" There doesn't appear to be any new CLI executables installed, and the documentation links to an API but there's no clues on how to actually process a prompt. What am I missing? Alternatively, I wouldn't mind using it in a Notebook but so far this thread doesn't link to anything so ambitious (yet?)
- speedgoose 3y agoThe main gradio app has been moved to the demos folder. python demos/musicgen_app.py Otherwise you can check the jupyter notebooks in the same folder.
- peteforde 3y agoThanks! This will be even more helpful if you could share a hint about where this was installed to. I carefully went through the output generated by the "pip install -U audiocraft" command, and there were no clues provided. Disclosure: I am not a Python developer, so I apologize if this is a master-of-the-obvious question for Python folks. However, if there was ever a scenario where a line or two of post-install notes would be useful, it's stuff like this.
- speedgoose 3y agoYou may have to clone the repository to get the demos folder. Otherwise it's perhaps somewhere depending on how you use python (global and often broken environment, virtual environments, conda hell, etc…). I feel like Python folks are on average terrible at distributing software. So many projects have some python script to install the dependencies, still assume you use conda, or don't bother to specify the dependencies versions. Thankfully it's often the same patterns and after some time you understand what to do based on the error messages. But I wish they could use something like NPM or Cargo. Even something like Maven would be an improvement.
- xolox 3y agoTip: If you run `pip uninstall audiocraft` it should render an interactive confirmation prompt that explains which files it is about to remove, this should help to understand where the library was installed (of course you don't have to confirm the prompt, just press Control-C or answer "No" to the prompt once you have your answer). Disregarding the tip above, determining where the library was installed requires a bit of context, for example your platform (Windows versus UNIX) and the fact that newer pip releases default to "pip install --user" when not running with super-user privileges, whereas older pip releases did not default to "pip install --user". Assuming you are using Linux and using an up-to-date pip release and you ran the "pip install -U audiocraft" command without super-user privileges, then the library was most likely installed in ~/.local/lib/pythonX.Y/site-packages (where X.Y is the version of Python that was used by the pip command you ran).
- CSSer 3y agoDoes anyone else hear a kind of background static in these samples? It almost sounds like part of the track is more compressed in terms of dynamic range than other parts, which doesn't make any sense to me. I'm trying to decide if this is my own confirmation bias at work or not.
- cushpush 3y agoNice results so far. "Perfect for the beach" is a very funny description of music, because it has nothing to do with the acoustic qualities, so consider these descriptions to be anthropocentric! (As if they could be anything else) It is less about describing the actual sounds you want and more about describing the quality or vibe of the atmosphere. This is markedly different than incremental composition, maybe we can call it "composition by collage." Puts on COLLAGE shirt like in Animal House
- squidsoup 3y ago"Imagine a professional musician being able to explore new compositions without having to play a single note on an instrument." A musician will always reach for an instrument as their compositional tool - keyboard and mouse producers are not musicians.
- brennanm 3y agoThis makes no sense. How is pressing a key on a piano different from pressing a key on an electronic piano?
- squidsoup 3y agoI was referring to a computer keyboard, not an electric piano. I can't see how any musician would see this appealing as a compositional tool. Music is its own language - expressing a musical idea with a text prompt is antithetical to the process of making music.
- RaiyanYahya 3y agoWould really be interested to run this locally
- MadDemon 3y agoWe built a Mac/Windows app around the original MusicGen so people can experiment with it on their own machine with a simple UI (https://samplab.com/text-to-sample https://samplab.com/text-to-sample).
- ouraf 3y agoIt will be a licensing nightmare, just like Llama1 was. But clever devs can study it to make better software and pressure them for a better license on the next release. Worked for LLaMA 2
- waffletower 3y agoI think it is a mistake to acquiesce and let copyright owners bully AI model trainers over model data inputs. The endgame of this practice is a "pay per thought" society. This is separate from speculation regarding machine sentience -- as interfaces improve AI models will serve more and more as direct human extensions of mind. While copyright duration is a separate issue, and the current durations are appalling, copyright violations should focus strictly upon the output of models and how they are utilized. There are so many melodies in my head that I have not nor will I ever pay for, (some of which I would love to remove). AI models need also to have the same unfettered access to the commons as we do. Infringement occurs on the outputs -- application of copyright restrictions on model inputs is a violation of Fair Use and a definite money grab.
- momirlan 3y agooh no, more muzak !
- makestuff 3y agoThese models are going to end up being used for advertising. Soon pretty much every ad you see will be generative AI based. It makes A/B testing way easier as you no longer need a creative person to modify the ad or change something subtle about it. For example, the generative voice might change to a different speaker or something, and the AI can generate thousands of different voices to see which one is most effective.
- deleted 3y ago[deleted]
- e1ghtSpace 3y agoWe might even end up with the most effective ones being the weirdest. Where people clicked though because the ad was so weird.
- BryanLegend 3y agoHow many parameters do these models have?
- jokethrowaway 3y agoA pretty neat background noise generator. "From text to audio with ease" I hoped for a second we would get a good quality model to do text to speech - damn, I guess it's back to bruteforcing bark.ai or waiting for tortoise (or more realistically, just paying elevenlabs)
- Somteencead 3y agoThe Encodec project's focus on AI-generated sound effects offers practical applications across various industries, from gaming to film production. It's exciting to think about the possibilities of AI enhancing audio design and creating immersive auditory experiences.