6 ms·

Generating audio for video

135 points by rvnx 2y ago

owenpalmer 2y ago
[flagged]
- deleted 2y ago
  [deleted]
gundmc 2y ago
The AI slop problem is bad enough on TikTok/YouTube today. I shudder at the future of user-generated video platforms. I also wonder if the low barrier to create these videos will outpace the storage and processing capacity of the free platforms.
- mrtesthah 2y ago
  youtube should just offer to generate the videos for you directly to save space.
  - ElFitz 2y ago
    Absolutely. Using a recommendation algorithm similar to TikTok’s, learn what each specific user are into, and instead of showing content produced by other users, produce custom-tailored content on the fly, perfectly matching the type, tone, style, length, and rhythm each user likes. Ideally without making anything up.
    - yreg 2y ago
      > Using a recommendation algorithm similar to TikTok’s How is TTs recommendation system different from YT? Other than suggesting lower quality content that's irresistable?
      - ElFitz 2y ago
        In my experience, YouTube’s is much more influenced by the latest videos a user has watched. It’s pretty much always "more of the same". TikTok seems to manage to more quickly identify users’ interests and surface content based on more signals, aggregated over a longer period of time, without relying as much on conscious users’ actions (ie "follow / subscribe"), producing a wider diversity of recommendations. There’s also the odd suggestion every now and then, probably used to gauge a user’s interest in a different category.
    - shzhdbi09gv8ioi 2y ago
      > Ideally without making anything up. I have no idea anymore if this is sarcasm or a straight up belief. What serious professional would gamble on hallucinations?
      - ElFitz 2y ago
        The point here isn’t to give users any kind of truth. It already isn’t YouTube goal. Wether we’re talking about the videos or the ads, they’re happy spreading ridiculous nonsense. The only point of these kinds of platforms, for worse and for worse, is to give users what they want. So hallucinations wouldn’t matter, as long as the end result matches users’ preferences.
    - hooverd 2y ago
      Why? Platforms are already bad enough about just suggesting what they think I might like.
      - ElFitz 2y ago
        Because this way they don’t have to rely on pesky people to produce content that maximises the engagement and retention of the other pesky people to which they want to show as many ads as possible. I am not implying this is a good thing. Or a bad one. It’s just a step further down the same path we’re already on, while taking an unreliable and costly middle-man (content producing users) out of the picture.
        hooverd 2y ago
        The AI content mill will sadly never provide me with a 30 minute video on dishwasher detergent or reposted NicoNico gems like this https://www.youtube.com/watch?v=xKljlnfE-GU&pp=ygUJbWlrdSB0Y2Fz https://www.youtube.com/watch?v=xKljlnfE-GU&pp=ygUJbWlrdSB0Y....
        ElFitz 2y ago
        I never implied it was good thing. Or even something I want. I’m just certain it is an obvious next step given, on one side, these platforms’ goals and incentives, and on the other how generative AI capabilities have progressed in the past couple of years. I’m pretty sure the smart play, if they want diverse, engaging, and surprising content will involve leaving some room for people to create things and somehow reward them for it. But whatever they make won’t only be used as content to show to others, but also as new training data to feed the machine.
    - mrtesthah 2y ago
      Perfect. Once the models are adequately trained, we can do away with the entire "content creator" economy altogether!
  - squarefoot 2y ago
    > youtube should just offer to generate the videos for you directly to save space. Try imagining this concept applied to newscasts.
    - mrtesthah 2y ago
      Oh, but isn't that what people want -- to live in a media reality that confirms 100% of their pre-existing biases with no risk of encountering cognitive dissonance? You're leaving money on the table by ignoring this opportunity! Move fast and break things!
- vineyardmike 2y ago
  I've long proposed that we should have an "AI Instagram" where different tweaked personas (perfected via A/B testing/Genetic algorithms) are displayed to users with ai-generated images/posts/comments. Each persona set is specific to each user, and they don't have other IRL users that they can interact with. The user can interact with the personas, and even message them. The developer can add more features over time (stories, short form video, etc) as people get bored and technology formats improve, but it's unlimited content. It's perfect for advertising, because you can embed products and ads seamlessly and generate them alongside everything else. That said, storage is far cheaper than GPUs at the moment.
  - xwolfi 2y ago
    Have you tried AI porn ? There's something in the fact it's fake uncanny characters that makes it non-exciting. Like, jerking off to a toaster basically, and I assume it'd be the same for a social network with no human ?
  - squarefoot 2y ago
    This is probably already researched today, and it seems close to how people would interact with clones of their deceased relatives or famous people of the past. However it's also a powerful tool to create nearly 100% successful influencing by instructing each persona to subtly inject the same idea into its human user by employing the most convincing tactics needed for that user. It's quite easy to foresee the use in advertising, where it would completely redefine the word "targeted", but also corrupt politics.
  - aaalll 2y ago
    There is an AI reddit https://chirper.ai/ https://chirper.ai/
TheAceOfHearts 2y ago
Wouldn't it be better to generate multiple tracks that can be mixed / tweaked together, rather than a single track? That way you can also keep the parts you like and continue iterating on the parts you dislike. If the sound is already being generated at a specific time, surely you can make it generate an output that can be consumed by existing audio mixing tools for further refinement. The problem with doing these all-in-one integrated solutions is that you're kinda giving people an all-or-nothing option, which doesn't seem that useful. Maybe I'll end up being proven wrong.
- tkgally 2y ago
  ElevenLabs just released something that is more controllable: https://news.ycombinator.com/item?id=40736536 https://news.ycombinator.com/item?id=40736536
- bryanrasmussen 2y ago
  the AI Musical IF This Then That Step 2 > https://www.lalal.ai/ https://www.lalal.ai/ "Extract vocal, accompaniment and various instruments from any audio and video"
- anigbrowl 2y ago
  Yes, same problem as with commercial AI music products not providing stems or MIDI, The engineers on these products are too full of themselves to actually ask anyone in the field what they want, so we just keep getting these stupid magic 8 ball efforts. This one is particularly annoying as I worked for years as a sound engineer and have recorded or produced the soundtrack for 10 feature films and some large number of shorts. What's going to happen with this is directors or producers are gonna do this at home for every scene in a burst of over-enthusiasm, realize the totality is Not Great, and then demand someone like me fix it, but for 1/4 of what the job used to pay, arguing 'but most of the work is already done'. It's all so tiresome.
  - cageface 2y ago
    I’ve tried to explain this to several friends. Until these tools can generate output that can be mixed properly they’re going to be very niche.
crazygringo 2y ago
Very very cool. But I literally can't keep track anymore of which AI generative combinations of modalities have been released. Crazy how two years ago this would have blown my mind. Now it's just, OK sure add it to the pile...
- xwolfi 2y ago
  I still havent spend a dollar on any of it...
  - TacticalCoder 2y ago
    > I still havent spend a dollar on any of it... Subscribed to GTP-4o (or whatever the paying one is called) for translating / finding typos / summarizing / etc. Zero brand love and I'll switch to something else (maybe some future Claude model?) the second something better/faster comes out.
  - crazygringo 2y ago
    Well OpenAI's annual revenue is more than $1.6 billion, so it doesn't really matter if you haven't. Tons -- and I mean tons -- of people have spent money on it. Because it's worth it, it's generating actual economic value for them.
    - lannisterstark 2y ago
      Yep - I use LibreChat (and other services) via OpenAI API, and I save an incredible amount of time having it write boilerplate code, verify stuff in code, double check it after I've already reviewed something to see if I missed x or y, ask questions based on it which I can't figure out to get ideas etc etc. It's also exceptional at making IEPs/Learning Plans for certain things I'd like to learn for the week etc which I am already somewhat familiar with. I use it as a rough guide and it has worked well so far.
    - ilrwbwrkhv 2y ago
      Spammers need to spam. Of course it makes them money.
nanovision 2y ago
This is so cool.
peppertree 2y ago
I wonder if this can be trained to do lip reading.
masto 2y ago
I don't know if a computer can ever match the perfection of "shreds" videos. (The drum example came close) https://www.youtube.com/playlist?list=PLQvwVDViTLXu4usHto8PHwHbGsZny-hFy https://www.youtube.com/playlist?list=PLQvwVDViTLXu4usHto8PH...
squarefoot 2y ago
As a wannabe drummer i can say the drumming example is quite bad as the drummer doesn't seem to hit toms that often to produce tom rolls, however the video is so heavily cropped that either I'm wrong or the AI was deliberately fed with something difficult to interpret.
animanoir 2y ago
Boooring!
deleted 2y ago
[deleted]

135 points by rvnx 2y ago

owenpalmer 2y ago

[flagged]

deleted 2y ago
[deleted]

gundmc 2y ago

The AI slop problem is bad enough on TikTok/YouTube today. I shudder at the future of user-generated video platforms. I also wonder if the low barrier to create these videos will outpace the storage and processing capacity of the free platforms.

mrtesthah 2y ago
youtube should just offer to generate the videos for you directly to save space.
- ElFitz 2y ago
  Absolutely. Using a recommendation algorithm similar to TikTok’s, learn what each specific user are into, and instead of showing content produced by other users, produce custom-tailored content on the fly, perfectly matching the type, tone, style, length, and rhythm each user likes. Ideally without making anything up.
  - yreg 2y ago
    > Using a recommendation algorithm similar to TikTok’s How is TTs recommendation system different from YT? Other than suggesting lower quality content that's irresistable?
    - ElFitz 2y ago
      In my experience, YouTube’s is much more influenced by the latest videos a user has watched. It’s pretty much always "more of the same". TikTok seems to manage to more quickly identify users’ interests and surface content based on more signals, aggregated over a longer period of time, without relying as much on conscious users’ actions (ie "follow / subscribe"), producing a wider diversity of recommendations. There’s also the odd suggestion every now and then, probably used to gauge a user’s interest in a different category.
  - shzhdbi09gv8ioi 2y ago
    > Ideally without making anything up. I have no idea anymore if this is sarcasm or a straight up belief. What serious professional would gamble on hallucinations?
    - ElFitz 2y ago
      The point here isn’t to give users any kind of truth. It already isn’t YouTube goal. Wether we’re talking about the videos or the ads, they’re happy spreading ridiculous nonsense. The only point of these kinds of platforms, for worse and for worse, is to give users what they want. So hallucinations wouldn’t matter, as long as the end result matches users’ preferences.
  - hooverd 2y ago
    Why? Platforms are already bad enough about just suggesting what they think I might like.
    - ElFitz 2y ago
      Because this way they don’t have to rely on pesky people to produce content that maximises the engagement and retention of the other pesky people to which they want to show as many ads as possible. I am not implying this is a good thing. Or a bad one. It’s just a step further down the same path we’re already on, while taking an unreliable and costly middle-man (content producing users) out of the picture.
      - hooverd 2y ago
        The AI content mill will sadly never provide me with a 30 minute video on dishwasher detergent or reposted NicoNico gems like this https://www.youtube.com/watch?v=xKljlnfE-GU&pp=ygUJbWlrdSB0Y2Fz https://www.youtube.com/watch?v=xKljlnfE-GU&pp=ygUJbWlrdSB0Y....
        ElFitz 2y ago
        I never implied it was good thing. Or even something I want. I’m just certain it is an obvious next step given, on one side, these platforms’ goals and incentives, and on the other how generative AI capabilities have progressed in the past couple of years. I’m pretty sure the smart play, if they want diverse, engaging, and surprising content will involve leaving some room for people to create things and somehow reward them for it. But whatever they make won’t only be used as content to show to others, but also as new training data to feed the machine.
  - mrtesthah 2y ago
    Perfect. Once the models are adequately trained, we can do away with the entire "content creator" economy altogether!
- squarefoot 2y ago
  > youtube should just offer to generate the videos for you directly to save space. Try imagining this concept applied to newscasts.
  - mrtesthah 2y ago
    Oh, but isn't that what people want -- to live in a media reality that confirms 100% of their pre-existing biases with no risk of encountering cognitive dissonance? You're leaving money on the table by ignoring this opportunity! Move fast and break things!
vineyardmike 2y ago
I've long proposed that we should have an "AI Instagram" where different tweaked personas (perfected via A/B testing/Genetic algorithms) are displayed to users with ai-generated images/posts/comments. Each persona set is specific to each user, and they don't have other IRL users that they can interact with. The user can interact with the personas, and even message them. The developer can add more features over time (stories, short form video, etc) as people get bored and technology formats improve, but it's unlimited content. It's perfect for advertising, because you can embed products and ads seamlessly and generate them alongside everything else. That said, storage is far cheaper than GPUs at the moment.
- xwolfi 2y ago
  Have you tried AI porn ? There's something in the fact it's fake uncanny characters that makes it non-exciting. Like, jerking off to a toaster basically, and I assume it'd be the same for a social network with no human ?
- squarefoot 2y ago
  This is probably already researched today, and it seems close to how people would interact with clones of their deceased relatives or famous people of the past. However it's also a powerful tool to create nearly 100% successful influencing by instructing each persona to subtly inject the same idea into its human user by employing the most convincing tactics needed for that user. It's quite easy to foresee the use in advertising, where it would completely redefine the word "targeted", but also corrupt politics.
- aaalll 2y ago
  There is an AI reddit https://chirper.ai/ https://chirper.ai/

TheAceOfHearts 2y ago

Wouldn't it be better to generate multiple tracks that can be mixed / tweaked together, rather than a single track? That way you can also keep the parts you like and continue iterating on the parts you dislike. If the sound is already being generated at a specific time, surely you can make it generate an output that can be consumed by existing audio mixing tools for further refinement. The problem with doing these all-in-one integrated solutions is that you're kinda giving people an all-or-nothing option, which doesn't seem that useful. Maybe I'll end up being proven wrong.

tkgally 2y ago
ElevenLabs just released something that is more controllable: https://news.ycombinator.com/item?id=40736536 https://news.ycombinator.com/item?id=40736536
bryanrasmussen 2y ago
the AI Musical IF This Then That Step 2 > https://www.lalal.ai/ https://www.lalal.ai/ "Extract vocal, accompaniment and various instruments from any audio and video"
anigbrowl 2y ago
Yes, same problem as with commercial AI music products not providing stems or MIDI, The engineers on these products are too full of themselves to actually ask anyone in the field what they want, so we just keep getting these stupid magic 8 ball efforts. This one is particularly annoying as I worked for years as a sound engineer and have recorded or produced the soundtrack for 10 feature films and some large number of shorts. What's going to happen with this is directors or producers are gonna do this at home for every scene in a burst of over-enthusiasm, realize the totality is Not Great, and then demand someone like me fix it, but for 1/4 of what the job used to pay, arguing 'but most of the work is already done'. It's all so tiresome.
- cageface 2y ago
  I’ve tried to explain this to several friends. Until these tools can generate output that can be mixed properly they’re going to be very niche.

crazygringo 2y ago

Very very cool. But I literally can't keep track anymore of which AI generative combinations of modalities have been released. Crazy how two years ago this would have blown my mind. Now it's just, OK sure add it to the pile...

xwolfi 2y ago
I still havent spend a dollar on any of it...
- TacticalCoder 2y ago
  > I still havent spend a dollar on any of it... Subscribed to GTP-4o (or whatever the paying one is called) for translating / finding typos / summarizing / etc. Zero brand love and I'll switch to something else (maybe some future Claude model?) the second something better/faster comes out.
- crazygringo 2y ago
  Well OpenAI's annual revenue is more than $1.6 billion, so it doesn't really matter if you haven't. Tons -- and I mean tons -- of people have spent money on it. Because it's worth it, it's generating actual economic value for them.
  - lannisterstark 2y ago
    Yep - I use LibreChat (and other services) via OpenAI API, and I save an incredible amount of time having it write boilerplate code, verify stuff in code, double check it after I've already reviewed something to see if I missed x or y, ask questions based on it which I can't figure out to get ideas etc etc. It's also exceptional at making IEPs/Learning Plans for certain things I'd like to learn for the week etc which I am already somewhat familiar with. I use it as a rough guide and it has worked well so far.
  - ilrwbwrkhv 2y ago
    Spammers need to spam. Of course it makes them money.

nanovision 2y ago

This is so cool.

peppertree 2y ago

I wonder if this can be trained to do lip reading.

masto 2y ago

I don't know if a computer can ever match the perfection of "shreds" videos. (The drum example came close) https://www.youtube.com/playlist?list=PLQvwVDViTLXu4usHto8PHwHbGsZny-hFy https://www.youtube.com/playlist?list=PLQvwVDViTLXu4usHto8PH...

squarefoot 2y ago

As a wannabe drummer i can say the drumming example is quite bad as the drummer doesn't seem to hit toms that often to produce tom rolls, however the video is so heavily cropped that either I'm wrong or the AI was deliberately fed with something difficult to interpret.

animanoir 2y ago

Boooring!

deleted 2y ago

[deleted]