8 ms·

What I've Learned in the Past Year Spent Building an AI Video Editor

168 points by burningion 2y ago

lorddr 2y ago
[dead]
henrybrown1709 2y ago
[dead]
deleted 2y ago
[deleted]
mips_avatar 2y ago
I agree that building AI on top of the video editor is probably a mistake. Maybe the format of the representation of the video can be something better than a series of matrices of pixel values.
- arjunaaqa 2y ago
  Absolutely true ! Re-imagined AI first products will kill AI patched up legacy products. Always.
  - brianjking 2y ago
    I think this is sometimes true, and certainly after a ton of failure first.
  - dangitman 2y ago
    [dead]
35mm 2y ago
As someone who has worked as a video editor, the most helpful AI tool would be prompt based editing. For example “find all the interview sections where people are talking about x and make a sequence”. OpusClip claims to have this but it’s behind a waitlist.
- wheatgreaser 2y ago
  that seems really hard
- yunohn 2y ago
  Not a personal jab, but I am astounded how every day, HN is full of discussion around how articles, newsletters, podcasts, and videos need to be aggregated and summarized for actual consumption. Repeat ad infinitum in both directions. In my experience, I’ve always listened to live discussions or read long form blog posts, specifically for the story and obscure points being made. Summaries never capture that and always miss nuances.
  - mjburgess 2y ago
    It has a lot to do with the kinds of articles that appear on HN and across the internet. And also, that spending time on something requires being interested in it, and so, there's a larger audience for summaries. I think, in general, most people have areas of interest to them where it would not occur to them to summarise what they're having fun engaging with.
  - pjc50 2y ago
    Not sure about articles, but people keep recommending multi-hour-long podcasts and videos far beyond the ability of any employed person to keep up with what they might want, so a summary is a useful tool to extract the salient points and possibly consider if something meets the threshold of being better than all the other hour-long things I might want to spend my free hour on. It sometimes feels like media has bifurcated into hyper-dense (let me explain a whole field of law in a 30 second tiktok) versus hyper-fluffy (documentary with 30 minutes of material spread out into six episodes, with a recap before and after each commercial break), depending on whether the target audience has a job or not.
    - reportgunner 2y ago
      Sounds like you're suffering from FOMO if you feel the need to consume summaries of multi-hour content you don't have time to consume.
sfmike 2y ago
what do you think of this versus the ai that is hiring actors that are then reused as models in the videos via script
- burningion 2y ago
  Author here. I imagine that being one of the components you can "plug in" to what I'm building. Imagine taking in a prompt, which describes the video you'd like generated. At render time you pass along variables which get injected to describe the specifics for your audience. We can then adjust the video edit according to that audience, including mixing generated and non-generated content.
lukaqq 2y ago
Impressive blog! I am building a professional web video editor - https://chillin.online https://chillin.online and trying to embed various AI workflows into it. Your article has given me a lot of inspiration. Thank you!
- b-lee 2y ago
  Looks so interesting..
SCUSKU 2y ago
Love the author sharing their winding journey as well as the tools and things they learned along the way. You can tell the author did grow a lot through this process, and through the year. Great stuff, thanks for sharing these great tips :D
Narciss 2y ago
Good work on pushing through. It’s like you say, building anything is an achievement.
- ericmcer 2y ago
  Seriously, every person needs the opportunity to really throw themselves into creating something for a year. I think so many people walk around thinking "if only I had time/money/space/whatever I could do something amazing". It is really humbling to actually try it and realize how difficult making anything original is. You also realize that... you just might not be talented haha.
1oooqooq 2y ago
did i miss something or this is "video editing was too hard so i just made a Wikipedia reading bot that generates drivel for Instagram and TikTok at the same time"?
- burningion 2y ago
  Author here. This is a genuine concern of mine! I don't want to build something that generates slop. Rather, I think whenever we change the costs / process of things, new possibilities open up. As an example, last night I re-watched Starship Troopers for the six-hundredth time. I'm a huge fan of Paul Verhoeven. What if I could watch a custom edit of Starship Troopers on demand, and this edit surprised me with something new? I don't know exactly how this would look, but maybe it's interesting? It's tough to predict the future and how things will change. But I'd rather be participating in its creation, trying to make it better.
  - AlienRobot 2y ago
    >What if I could watch a custom edit of Starship Troopers on demand, and this edit surprised me with something new? I don't know exactly how this would look, but maybe it's interesting? Is this what you want to do? https://www.youtube.com/watch?v=6sUR6ylVH7E https://www.youtube.com/watch?v=6sUR6ylVH7E
  - j45 2y ago
    There is a high time cost to actual video editing and managing the details well, quite different than generating one template/perception of it, which is more where social media slop is at.
  - scudsworth 2y ago
    >What if I could watch a custom edit of Starship Troopers on demand, and this edit surprised me with something new? I don't know exactly how this would look, but maybe it's interesting? this is not interesting whatsoever actually
    - burningion 2y ago
Arelius 2y ago
Because I don't see it mentioned elsewhere, I wanted to plug OpenTimelineIO, as a lot of the industry is building support around it as a format right now, and it would be great for any new video editor to support. https://opentimelineio.readthedocs.io/en/stable/ https://opentimelineio.readthedocs.io/en/stable/
mcdow 2y ago
It seems like we are currently in the "skeuomorphic" product design era for AI products. Which is to say we are building the same products but with AI tacked on. I appreciate that you are approaching this problem from first principles and attempting to break from the model of the previous generation. Kudos.

168 points by burningion 2y ago

lorddr 2y ago

[dead]

henrybrown1709 2y ago

[dead]

deleted 2y ago

[deleted]

mips_avatar 2y ago

I agree that building AI on top of the video editor is probably a mistake. Maybe the format of the representation of the video can be something better than a series of matrices of pixel values.

arjunaaqa 2y ago
Absolutely true ! Re-imagined AI first products will kill AI patched up legacy products. Always.
- brianjking 2y ago
  I think this is sometimes true, and certainly after a ton of failure first.
- dangitman 2y ago
  [dead]

35mm 2y ago

As someone who has worked as a video editor, the most helpful AI tool would be prompt based editing. For example “find all the interview sections where people are talking about x and make a sequence”. OpusClip claims to have this but it’s behind a waitlist.

wheatgreaser 2y ago
that seems really hard
yunohn 2y ago
Not a personal jab, but I am astounded how every day, HN is full of discussion around how articles, newsletters, podcasts, and videos need to be aggregated and summarized for actual consumption. Repeat ad infinitum in both directions. In my experience, I’ve always listened to live discussions or read long form blog posts, specifically for the story and obscure points being made. Summaries never capture that and always miss nuances.
- mjburgess 2y ago
  It has a lot to do with the kinds of articles that appear on HN and across the internet. And also, that spending time on something requires being interested in it, and so, there's a larger audience for summaries. I think, in general, most people have areas of interest to them where it would not occur to them to summarise what they're having fun engaging with.
- pjc50 2y ago
  Not sure about articles, but people keep recommending multi-hour-long podcasts and videos far beyond the ability of any employed person to keep up with what they might want, so a summary is a useful tool to extract the salient points and possibly consider if something meets the threshold of being better than all the other hour-long things I might want to spend my free hour on. It sometimes feels like media has bifurcated into hyper-dense (let me explain a whole field of law in a 30 second tiktok) versus hyper-fluffy (documentary with 30 minutes of material spread out into six episodes, with a recap before and after each commercial break), depending on whether the target audience has a job or not.
  - reportgunner 2y ago
    Sounds like you're suffering from FOMO if you feel the need to consume summaries of multi-hour content you don't have time to consume.

sfmike 2y ago

what do you think of this versus the ai that is hiring actors that are then reused as models in the videos via script

burningion 2y ago
Author here. I imagine that being one of the components you can "plug in" to what I'm building. Imagine taking in a prompt, which describes the video you'd like generated. At render time you pass along variables which get injected to describe the specifics for your audience. We can then adjust the video edit according to that audience, including mixing generated and non-generated content.

lukaqq 2y ago

Impressive blog! I am building a professional web video editor - https://chillin.online https://chillin.online and trying to embed various AI workflows into it. Your article has given me a lot of inspiration. Thank you!

b-lee 2y ago
Looks so interesting..

SCUSKU 2y ago

Love the author sharing their winding journey as well as the tools and things they learned along the way. You can tell the author did grow a lot through this process, and through the year. Great stuff, thanks for sharing these great tips :D

Narciss 2y ago

Good work on pushing through. It’s like you say, building anything is an achievement.

ericmcer 2y ago
Seriously, every person needs the opportunity to really throw themselves into creating something for a year. I think so many people walk around thinking "if only I had time/money/space/whatever I could do something amazing". It is really humbling to actually try it and realize how difficult making anything original is. You also realize that... you just might not be talented haha.

1oooqooq 2y ago

did i miss something or this is "video editing was too hard so i just made a Wikipedia reading bot that generates drivel for Instagram and TikTok at the same time"?

burningion 2y ago
Author here. This is a genuine concern of mine! I don't want to build something that generates slop. Rather, I think whenever we change the costs / process of things, new possibilities open up. As an example, last night I re-watched Starship Troopers for the six-hundredth time. I'm a huge fan of Paul Verhoeven. What if I could watch a custom edit of Starship Troopers on demand, and this edit surprised me with something new? I don't know exactly how this would look, but maybe it's interesting? It's tough to predict the future and how things will change. But I'd rather be participating in its creation, trying to make it better.
- AlienRobot 2y ago
  >What if I could watch a custom edit of Starship Troopers on demand, and this edit surprised me with something new? I don't know exactly how this would look, but maybe it's interesting? Is this what you want to do? https://www.youtube.com/watch?v=6sUR6ylVH7E https://www.youtube.com/watch?v=6sUR6ylVH7E
- j45 2y ago
  There is a high time cost to actual video editing and managing the details well, quite different than generating one template/perception of it, which is more where social media slop is at.
- scudsworth 2y ago
  >What if I could watch a custom edit of Starship Troopers on demand, and this edit surprised me with something new? I don't know exactly how this would look, but maybe it's interesting? this is not interesting whatsoever actually
  - burningion 2y ago

Arelius 2y ago

Because I don't see it mentioned elsewhere, I wanted to plug OpenTimelineIO, as a lot of the industry is building support around it as a format right now, and it would be great for any new video editor to support. https://opentimelineio.readthedocs.io/en/stable/ https://opentimelineio.readthedocs.io/en/stable/

mcdow 2y ago

It seems like we are currently in the "skeuomorphic" product design era for AI products. Which is to say we are building the same products but with AI tacked on. I appreciate that you are approaching this problem from first principles and attempting to break from the model of the previous generation. Kudos.