11 ms·

Comixify: Transform video into a comics

77 points by hardmaru 8y ago

hardmaru 8y ago
link to their paper: https://arxiv.org/abs/1812.03473 https://arxiv.org/abs/1812.03473
- lsh 8y ago
  doesn't have much more than a link back to the site. google cache: https://webcache.googleusercontent.com/search?q=cache:c8dVMlkJspYJ:https://comixify.ii.pw.edu.pl/+&cd=1&hl=en&ct=clnk&gl=au https://webcache.googleusercontent.com/search?q=cache:c8dVMl... edit: my mistake, it has the full PDF there
neotek 8y ago
The site seems to be suffering from the HN hug of death, unfortunately.
vackosar 8y ago
no subtitles or audio integration yet
latexr 8y ago
Site is slow right now, but got results after some waiting. From this video[1] (one of their given demos), it generated this result[2]. From this other video[3] (another of their demos), it generated this result[4]. All used the default settings. Was frankly disappointed. They mention “neural style transfer” and whatnot, but the results looks only slightly better than what you’d get by applying default photoshop filters from a decade ago, probably longer. ImageMagick can also achieve something similar[5]. The extracted frames, including their sizes and positioning, don’t look like anything special. The final layout was the same for both tries, meaning they’re probably not doing anything to detect/convey a better sense of the story. In that sense, the result is as much a comic as any other random sequence of photographs (comics don’t need to be drawings) divided into squares. It’s a neat demo, but I’m failing to understand why it is part of a larger academic paper. Seems like the whole effect could have been achieved over a decade ago with a bash script glueing ffmpeg and imagemagick. [1]: https://youtu.be/CvvAftMZYKM https://youtu.be/CvvAftMZYKM [2]: https://imgur.com/a/rbrQBbx https://imgur.com/a/rbrQBbx [3]: https://youtu.be/pvAhRcUofDk https://youtu.be/pvAhRcUofDk [4]: https://imgur.com/a/kZm2M5w https://imgur.com/a/kZm2M5w [5]: http://www.fmwconcepts.com/imagemagick/cartoon/index.php http://www.fmwconcepts.com/imagemagick/cartoon/index.php
- andrusza2 8y ago
  Hi, I am one of co-authors of the paper. Thank you for your interest in our demo. Regarding the points you raise in your comment: 1. Layout You are right that we use static layout for all comics we generate. Changing that has a high priority on our todo list, we are currently working on more dynamic layout generation that takes into account what we know about the story and which frames are more important than others. 2. Style transfer The method we use gives the most pleasant results of all obtained with various style transfer methods applied to comics - see our previous work on comparison between them here: https://arxiv.org/abs/1809.01726 https://arxiv.org/abs/1809.01726. 3. Story sense (>probably not doing anything to detect/convey a better sense of the story<) We use Video Summarization model in our pipeline, however as mentioned above, we use static layouts and therefore this is not always apparent. Overall, this demo is only a starting point. We are working on adding automatic speech recognition and text generation to introduce dialogs into our comics. And we have many other improvements planned. Stay tuned and if you have any other comments feel free to send us an email to any of the addresses from the paper: 243678@pw.edu.pl, 280814@pw.edu.pl, 261406@pw.edu.pl, t.trzcinski@ii.pw.edu.pl.
liquidmetal 8y ago
Something similar we did for an internal Microsoft hackathon in 2017. https://utkarshsinha.com/blog/microsoft-oneweek-hackathon-2018/ https://utkarshsinha.com/blog/microsoft-oneweek-hackathon-20...
ZeroGravitas 8y ago
This reminded me of this: BBC iPlayer for Kindle http://www.frisnit.com/iplayer-for-kindle/ http://www.frisnit.com/iplayer-for-kindle/

77 points by hardmaru 8y ago

hardmaru 8y ago

link to their paper: https://arxiv.org/abs/1812.03473 https://arxiv.org/abs/1812.03473

lsh 8y ago
doesn't have much more than a link back to the site. google cache: https://webcache.googleusercontent.com/search?q=cache:c8dVMlkJspYJ:https://comixify.ii.pw.edu.pl/+&cd=1&hl=en&ct=clnk&gl=au https://webcache.googleusercontent.com/search?q=cache:c8dVMl... edit: my mistake, it has the full PDF there

neotek 8y ago

The site seems to be suffering from the HN hug of death, unfortunately.

vackosar 8y ago

no subtitles or audio integration yet

latexr 8y ago

Site is slow right now, but got results after some waiting. From this video[1] (one of their given demos), it generated this result[2]. From this other video[3] (another of their demos), it generated this result[4]. All used the default settings. Was frankly disappointed. They mention “neural style transfer” and whatnot, but the results looks only slightly better than what you’d get by applying default photoshop filters from a decade ago, probably longer. ImageMagick can also achieve something similar[5]. The extracted frames, including their sizes and positioning, don’t look like anything special. The final layout was the same for both tries, meaning they’re probably not doing anything to detect/convey a better sense of the story. In that sense, the result is as much a comic as any other random sequence of photographs (comics don’t need to be drawings) divided into squares. It’s a neat demo, but I’m failing to understand why it is part of a larger academic paper. Seems like the whole effect could have been achieved over a decade ago with a bash script glueing ffmpeg and imagemagick. [1]: https://youtu.be/CvvAftMZYKM https://youtu.be/CvvAftMZYKM [2]: https://imgur.com/a/rbrQBbx https://imgur.com/a/rbrQBbx [3]: https://youtu.be/pvAhRcUofDk https://youtu.be/pvAhRcUofDk [4]: https://imgur.com/a/kZm2M5w https://imgur.com/a/kZm2M5w [5]: http://www.fmwconcepts.com/imagemagick/cartoon/index.php http://www.fmwconcepts.com/imagemagick/cartoon/index.php

andrusza2 8y ago
Hi, I am one of co-authors of the paper. Thank you for your interest in our demo. Regarding the points you raise in your comment: 1. Layout You are right that we use static layout for all comics we generate. Changing that has a high priority on our todo list, we are currently working on more dynamic layout generation that takes into account what we know about the story and which frames are more important than others. 2. Style transfer The method we use gives the most pleasant results of all obtained with various style transfer methods applied to comics - see our previous work on comparison between them here: https://arxiv.org/abs/1809.01726 https://arxiv.org/abs/1809.01726. 3. Story sense (>probably not doing anything to detect/convey a better sense of the story<) We use Video Summarization model in our pipeline, however as mentioned above, we use static layouts and therefore this is not always apparent. Overall, this demo is only a starting point. We are working on adding automatic speech recognition and text generation to introduce dialogs into our comics. And we have many other improvements planned. Stay tuned and if you have any other comments feel free to send us an email to any of the addresses from the paper: 243678@pw.edu.pl, 280814@pw.edu.pl, 261406@pw.edu.pl, t.trzcinski@ii.pw.edu.pl.

liquidmetal 8y ago

Something similar we did for an internal Microsoft hackathon in 2017. https://utkarshsinha.com/blog/microsoft-oneweek-hackathon-2018/ https://utkarshsinha.com/blog/microsoft-oneweek-hackathon-20...

ZeroGravitas 8y ago

This reminded me of this: BBC iPlayer for Kindle http://www.frisnit.com/iplayer-for-kindle/ http://www.frisnit.com/iplayer-for-kindle/