13 ms·

Video Vectorization

319 points by xanthine 6y ago

deleted 6y ago
[deleted]
londons_explore 6y ago
Is there market demand for this? In a world where videos are still sent around as multi-megabyte gif files, and audio clips are still distributed with a random slideshow on YouTube, I think a lot of users aren't so bothered about efficiency - they just want the simplest thing that works. Lots of music is most efficiently stored as MIDI, yet how many songs on iTunes are midi? For video, raster is king because it works for everything.
- vhiremath4 6y ago
  It's a good point. Internet speeds are also quickly increasing in areas getting online for the first time.
- xanthine 6y ago
  They've listed some uses for it — animated videos, e-learning (the likes of Khan Academy). While I doubt that animation would shift to vector video any time soon, it indeed would help e-learning a lot, speaking from the side of the content distributors (less size). The only benefit for end users might be faster streaming, but yeah, I agree that the average user can't be bothered about it.
  - deleted 6y ago
    [deleted]
  - egypturnash 6y ago
    Lots of animation is done in vector tools to start with. If this format goes anywhere it would probably be pretty easy to add exporters to Toon Boom, Animate, OpenToonz, and whatever else people are currently using - I left that industry a decade and a half ago so I'm not too up on what everyone's using right now.
- xanthine 6y ago
  >Lots of music is most efficiently stored as MIDI, yet how many songs on iTunes are midi? While storing music as its component instruments is indeed size efficient, converting existing music into MIDIs using some ML/AI algorithm is not yet effective. Vectorly seems to convert _existing_ raster videos into vector ones using some combination of computer vision algorithms.
  - rjmunro 6y ago
    Playback of a MIDI file also doesn't return the original sound quality, it plays the same notes with generic sampled instruments. For high quality samples of some instruments, this can be OK, but for others, it sounds terrible. As for lyrics, you have to use a speech synth... https://www.youtube.com/watch?v=bJjxy4V00r0 https://www.youtube.com/watch?v=bJjxy4V00r0
- kevingadd 6y ago
  IMO the biggest problem is that even for enterprise/large-scale-website scenarios, the combination of the product cost, the cost of downloading the decoder on each machine, and the increased cpu/gpu load (resulting in lower battery life) can drown out any advantage from the size reduction. The fact that it only works for a subset of video content is another problem. Switching to H265 or AV1 is way more compelling in practice because it works for everything, it's hardware accelerated, the decoders are off-the-shelf, and the quality is significantly improved. A modern GPU can both encode and decode H265.
- black_puppydog 6y ago
  If you could drastically reduce the size of videos, self-hosting video distribution pages for e.g. training material would suddenly become feasible, if not for individuals then for small companies.
  - kevingadd 6y ago
    If you try to do it, you'll find that self-hosting for these scenarios is surprisingly cheap. You might be able to just use YouTube, anyway.
    - black_puppydog 6y ago
      Well, youtube is exactly not self-hosted. For one reason or another, you might really not want to have to deal with youtube (or vimeo for that matter) and it would be great to have the possibility to opt out of using such a service.
  - laserpistus 6y ago
    Yep, we serve around 50.000 hours of video in a month with a similar format and have ~0 extra bandwidth fees.
- jpalomaki 6y ago
  In some parts of the world mobile data is expensive compared to income and this is likely limiting consumption of services like online video. Random link with some pricing information for Africa [1] claims prices are up from $0.50/GB. [1] https://kenyanwallstreet.com/mobile-data-pricing-2020-report-somalia-has-the-cheapest-mobile-data-in-africa/ https://kenyanwallstreet.com/mobile-data-pricing-2020-report...
- gokhan 6y ago
  Netflix and others would be interested in mega bandwidth gains for animated content. They usually own both ends of the stream so they can deploy whatever they want.
- laserpistus 6y ago
  We (scrimba.com) actually do something similar for e-learning within programming. And it definitely makes sense for both students and as a business. - Our videos use about 1/100 bandwidth of a video. This matters for rendering (fans running when watching HD?) and to people who live in places with shitty ISPs like in Germany - It is rendered and thereby crisp as a Pringle - No codec needed as we render it as html - When you pause a video you actually have the text there, not pixels. For programming that allows you to copy and change things! - We store the context it was recorded in (dev environment) so you can change and run code in the "video" itself. This changes the pedagogy you can do in a video to be more interactive and hands on.
- dimatura 6y ago
  It's not really the same. True, MIDI won't work for say, iTunes, because it doesn't capture a lot of the information needed to recreate the audio of your average song. But for certain types of video, you can capture everything you need to recreate it faithfully with a vector format, at much higher compression rates. I think a better analogy is SVG -- which is not as popular as raster formats in the web, but it certainly has its place. (As an aside, MIDI is a useful format in its own right, and is still alive and well in the music-making world, even if from a technical perspective it's sort of outdated in 2020.)
vhiremath4 6y ago
Woah this is incredible. Even with the increased CPU demands, this is a very large bandwidth savings. >In practice however, DRM, streaming , analytics and ad placement also require javascript logic to function in web runtimes, so in real word settings web-video playback can and does use a non-trivial amount of CPU time. I'm a little skeptical of this claim. No idea how much CPU is used for DRM, but I can't imagine it's on the order of multiple percentage points.
- chmod775 6y ago
  > Woah this is incredible. Even with the increased CPU demands, this is a very large bandwidth savings. GPUs happen to be very good at drawing shapes. So there's probably a lot of room for improvement, probably even enough that these kinds of videos could be faster than decoding "normal" video (for which modern CPUs and GPUs have dedicated silicon). This doesn't mean that I'm not entirely skeptical of this, especially because this probably will fail spectacularly for animation that isn't just vector graphics, which is most animation.
  - kevingadd 6y ago
    GPUs are good at drawing shapes, but that doesn't actually mean that using the GPU to rasterize high-quality polygons will be fast. In practice most production solutions for this run on a combination of CPU and GPU and use complex shaders to achieve acceptable quality. If you're rendering regular alpha-blended or opaque triangles that's very fast, but they're not going to be anti-aliased and stuff like cubic splines will need to be triangulated on the CPU or GPU. Libraries like Pathfinder and Slug are the current state of the art and they are very complicated. In comparison, video acceleration on most modern GPUs is done using dedicated silicon that typically has its own performance budget (so it doesn't eat into the GPU time to run shaders or your compositor).
    - chrismorgan 6y ago
      Pathfinder and its ilk would breeze through this sort of thing, and are still improving constantly too, because so comparatively little work has been done in the space. Guessing very roughly based on the figures in https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fast-2d-rendering.html https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fas... (and Pathfinder is integrating some of the techniques there), I’d estimate that piet-gpu and soon Pathfinder, running on Intel integrated graphics, could comfortably draw each frame of this Simpsons sample video in well under one millisecond (for an effective maximum of 6% GPU/CPU usage), probably even under a tenth of a millisecond (0.6%). (Admittedly you could easily create a more complicated video which would be much more draining, e.g. paris-30k frames would struggle to hit 120fps on integrated graphics. Generally speaking it’s easier to make pathological cases in vector encodings than raster. But let’s forget about that.) At that point, the hardware separation of video decoder and other graphics stuff (which is a strong point in general, because it’s so computationally expensive) is much less important. You speak of Pathfinder and Slug as being “very complicated”, and indeed they are; but video decoding is hardly any less complicated, and much more computationally expensive.
katmannthree 6y ago
It sounds cool but given that the demo looks like this [0] on my bog-standard Windows 10 PC with Chrome (and Firefox and Edge) I'm assuming they've still got some bugs to work out... If it's working for anyone here I'd love to see a screen capture of the proper rendering. [o]: https://i.imgur.com/YO42u2C.png https://i.imgur.com/YO42u2C.png
- hackstack 6y ago
  I just got a black box on my Windows system. Looks pretty good in Firefox on the Mac though: https://imgur.com/a/rqPgQFZ https://imgur.com/a/rqPgQFZ Edit: Here it is in video form https://imgur.com/a/B0UOigX https://imgur.com/a/B0UOigX
  - katmannthree 6y ago
    Thank you! That video really is impressive.
- themmes 6y ago
  Interesting, maybe it's missing a coordinate, could it be missing the repeating first coordinate? Surprising to see that surfaces of the same color are made up so many polygons, I guess that can be made even more efficient?
- willis936 6y ago
  Aha. Reminds me of what I saw for two weeks when I wrote an svg renderer in MATLAB. I only got it to work with a subset of the standard and was reliant on the conventions of inkscape export for parsing. You can do a lot with just beziers, circles, and straight lines.
- aembleton 6y ago
  https://imgur.com/3Cgs7Z8 https://imgur.com/3Cgs7Z8 Works well on Firefox 78.0.2 on Ubuntu 20.04
xanthine 6y ago
Their android SDK (no release version yet) is available at their Github repo, and so are their bulk upload tools (for talking to their servers using, I guess, a pay to use API).
acd 6y ago
I think your technology would be useful for restoring old animated videos. Plus it would be useful for The catchy intro animations used on startups to demonstrate their technology. Also would be nice to use when you have bad internet connection speed to watch e-learning material animations. For e-learning you may need a hybrid M3u like Playlist approach with video for the presenter and vector graphics for the screen casts. Manga videos would probably also compress well. Children animated videos. What if you would reduce the color space vectorize of ordinary video to for example 8 colors and smooth out the noise to make large flat surfaces could you compress it with vectors?
- kevingadd 6y ago
  The average anime or children's cartoon won't compress well with this approach unless it's authored with it in mind - if you look at the average anime episode it's full of raster effects, layered gradients, 3d perspective, and mixed in 3DCG. Worse still the anime is typically sent to the broadcaster in an already-lossy format so the vectorizer will have to handle macroblocking and noise. Here are two random screen captures from anime airing this season: https://pbs.twimg.com/media/EdP8gc4WAAAbkWy?format=jpg&name=large https://pbs.twimg.com/media/EdP8gc4WAAAbkWy?format=jpg&name=... https://pbs.twimg.com/media/EdPwGmyXoAIjlGw?format=jpg&name=large https://pbs.twimg.com/media/EdPwGmyXoAIjlGw?format=jpg&name=... Suffice it to say that vectorizing this content would be difficult and the vector representation wouldn't be particularly small.
chmod775 6y ago
While this is pretty cool, this naive approach will fail spectacularly for animation that isn't just vector graphics, which is most animation. This might have a future as part of a regular video codec, being used when there's mostly vector graphics on screen (or just for those areas that are vector graphics).
royjacobs 6y ago
Reminds me of the Spaceballs demos State of the Art and Nine Fingers [1]. Released in 1993 iirc. [1] https://www.youtube.com/watch?v=PPoYzwib7JQ https://www.youtube.com/watch?v=PPoYzwib7JQ
teddyh 6y ago
Isn’t this basically the same thing which was famously used to achieve full motion video in the 1992 Amiga demo “State of the Art”¹ and improved one year later in the followup “9 Fingers”²? 1. https://www.pouet.net/prod.php?which=99 https://www.pouet.net/prod.php?which=99 https://www.youtube.com/watch?v=J2r7-ygXOzo https://www.youtube.com/watch?v=J2r7-ygXOzo 2. https://www.pouet.net/prod.php?which=100 https://www.pouet.net/prod.php?which=100 https://www.youtube.com/watch?v=tGetanBEKK8 https://www.youtube.com/watch?v=tGetanBEKK8
- emmanueloga_ 6y ago
  Apparently, there's not much info out there about how 9 Fingers was made. I found this [1]. A genetic algorithm takes quite a while to come up with a pic, I wonder how long it took them in 1992! [2] Kind of related, this series of articles on the vector encoder for Another World [3]. 1: http://www.mos6502.com/friday-commodore/9-fingers-the-infamous-amiga-demo/ http://www.mos6502.com/friday-commodore/9-fingers-the-infamo... 2: https://chriscummins.cc/s/genetics/ https://chriscummins.cc/s/genetics/ 3: https://fabiensanglard.net/another_world_polygons/index.html https://fabiensanglard.net/another_world_polygons/index.html
  - metadata 6y ago
    I can give you a glimpse into the past. Amiga had a very slow CPU (7 MHz) and each instruction took between 4 and 40-ish cycles. However, it had 3 special purpose co-processors. For these demos, Blitter (very fast bit copy and memory fill functionality) and Copper (change palette and even resolution on a specific vertical scan location) were essential. The technique is the same one we used to make 3D games - you would draw an outline (taking care to write a exactly single pixel on boundaries, your line-drawing routines had to be written to take that into account) and let Blitter fill this. Amiga had 6 overlay planes you could use for up to 64 colors (well, it was actually up to 4096 colors with caveats, but let's ignore that for a moment). So, if you write a filled outlines (time shifted) in those 6 planes, you'd get this nice blurred effect. My guess is they manually wrote pixel coordinates for the images, connected them with lines, filled with Blitter, repeated for each frame. It's how things were done back then. Without Internet we had a lot of free time (I was a kid in high school). Those special purpose chips were magical. At the time of IBM XT and its green on black terminals, Amiga was a space shuttle in comparison. For example, my game used to have upper part of the screen in lower resolution (where we painted vector graphics), lower part in higher (where we had game stats).
villgax 6y ago
Not ready yet, just hyping SVG videos for now
emmanueloga_ 6y ago
It just boggles my mind that the front page of a page that offers a "patented vector-transcoder converts video to a vector format, reducing bitrates" doesn't include such video on its front page.
- jlmorton 6y ago
  Well, it's not at the top of the page, but they did include a short proof-of-concept video of the Simpsons at the bottom of the page. https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.html https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.h...
  - WrtCdEvrydy 6y ago
    That is broken on my computer :(
    - Manfred 6y ago
      I suspect that they only focused on particular versions of Chrome during development because it's a proof of concept.
      - WrtCdEvrydy 6y ago
        Yeah, but it is Windows 10 on the latest chrome... pictures other people have linked look cool though.
        beernutz 6y ago
        Seems to work fine on Win 10 and latest chrome for me. I wonder if it might be related to turning on/off the GPU rendering in chrome maybe?
      - akdor1154 6y ago
        works on Firefox here for me
      - 6y ago
villgax 6y ago
Even their job postings on LinkedIn look like somebody anger-wrote it. For an Image Processing role, they've written no ML/DL engineers which is obvious but still, as if Computer Vision isn't any way linked to whatever it is they are trying to do with compression.
bkm 6y ago
One overlooked usecase is RDP and VNC-like concepts. Right now Appetize (App streaming service) uses raster graphics. Vector would allow high-FPS streaming of Apps, making running Apps in the cloud a realistic option.
- kevingadd 6y ago
  Most modern application UI can't really be encoded as vectors, there's lots of raster stuff in there plus bitmaps. Many mobile apps right now are a big pile of PNGs and/or raster filters (like drop shadows, etc). So you end up actually needing to ship scene graphs along with bitmaps. You can certainly leverage that, but it also increases the resource demands (and thus power demands) on the client.
- GordonS 6y ago
  I thought RDP was already a bit more intelligent than "only" sending raster frames, sending things like window dimensions and such to be rendered on the client?
  - bob1029 6y ago
    RDP is probably more intelligent than most people are aware. Try remoting into a machine over RDP and playing a youtube video occupying a large portion of the screen. You may be surprised to find that it plays nearly perfectly, even over non-ideal network conditions (wifi). Try this same exercise with VNC and you will experience a frame maybe once every other second. I am not sure exactly the heuristics involved, but RDP is certainly switching between modes of operation based on what kind of visual information is on the screen.
    - ttsda 6y ago
      RDP also works more reliably over mobile connections than VNC, in my experience
- IshKebab 6y ago
  Nah, raster encoders can do a pretty great job on vector-style data, whereas vector encoders can't do anything with raster-style data. In other words what happens when you play a video or view a photograph? Are you going to auto-detect that and switch modes? You'll end up just recreating a modern video codec. The best thing to do is just use a modern video codec and make sure it works well with text and sharp edges.
nautical 6y ago
If I understand this correctly, will it also allow to send updates to video on the fly ? Example : Change video from X to Y time to new vectors [V1 ..] ?
black_puppydog 6y ago
This reminds me of a paper from 2005 by Daniel Sýkora et al. [1] which tries something very similar, with the specific use case of animation video. The authors describe it best in the abstract I think: > Video Codec for Classical Cartoon Animations with Hardware Accelerated Playback > We introduce a novel approach to video compression which is suitable for traditional outline-based cartoon animations. In this case the dynamic foreground consists of several homogeneous regions and the background is static textural image. For this drawing style we show how to recover hybrid representation where the background is stored as a single bitmap and the foreground as a sequence of vector images. The idea of using prior knowledge about the nature of the content to decide on an encoding scheme makes intuitive sense to me, though I'm not a codec person so I don't know how feasible it would be to make these ideas into the hardware-accelerated codecs we know from other methods. Of course, these methods would make the most sense when used directly by the animation studios during export, not as an afterthought. But I'll take what we can get. By the way, the corpus of Sýkora's works [2] is really really impressive in my opinion. He gave a talk in my institute while I was researching methods around neural style, and his take on parametric models, paired with the quality (and speed!) of his results, really left a mark, if not to say they made me seriously question wtf I was doing there. His work is strictly tailored to a professional animation / video production setting, so it seems extremely applicable compared to the toy-like nature of neural style methods. That is not to say he doesn't know about those. His team's recent papers actually fruitfully combine the two. [1]: https://link.springer.com/chapter/10.1007/11595755_6 https://link.springer.com/chapter/10.1007/11595755_6 [2]: https://dcgi.fel.cvut.cz/home/sykorad/ https://dcgi.fel.cvut.cz/home/sykorad/
- dharma1 6y ago
  Daniel's video style transfer tech was used in this - it's really cool. https://ebsynth.com/ https://ebsynth.com/
karteum 6y ago
In terms of stream format, it seems that BIFS / MPEG 4 part 11 originaly aimed at the same purpose (probably in a more efficient manner than textual SVG), isn't it ? https://en.wikipedia.org/wiki/MPEG-4_Part_11 https://en.wikipedia.org/wiki/MPEG-4_Part_11
okaleniuk 6y ago
It's awesome, but it also brings back memories of the flash-animation.
- cypressious 6y ago
  Exactly. The funny thing is that Flash animations where the more straight-forward way. You create vector animations and export them as such. Obviously, this is more universal. You can take any input as opposed to something that was vectorized to begin with.
sneak 6y ago
Patenting algorithms is supposed to be impossible. Companies that ignore this and patent the “system and method” for implementing algorithms are being jerks.
- villgax 6y ago
  They're using this to get VC clout & footing in job posts since they don't have any revenue the only way of convincing people for them is to say we have a 'patent'. Almost as if a shark-tank episode wrote their pitches.
rhn_mk1 6y ago
Maybe this finally brings back the crispy scaling quality of Flash videos. They seem to be released in lossy raster formats these days.
imtringued 6y ago
Can this technique be used to generate a CAD sketch from a photograph?
- detritus 6y ago
  Technically, probably, although there are exisitng and likely better vector conversions for none-animated purposes. The problem with CAD is that whilst a conversion to a rough sketch will be possible, 'CAD' implies very high accuracy or exactness, and the amount of work required to render a converted sketch accurate will likely trump that by just doing it manually in the first place. I suppose in future some ML-aided setup that takes a drawing in AND focuses on dimensional accuracy might fare better than this video-centric solution.
wodenokoto 6y ago
As a kid, I used to trace frames from the Simpsons in Macromedia Flash (later Adobe, later discontinued) as a way of creating high-res images, so reading this article really hit home for me! While the output of this algorithm (just like my traces) isn't as faithful to the source material as, say, H.264 is, the result looks great and has an amazing style. This might be a great target for mobile-first webtoons.
fefe23 6y ago
AFAIK MPEG-4 experimented with encoding 3d objects but it never took off. As usual for MPEG they did not specify how to get the 3d data from a scene but how to encode them, actually how to decode them, so that innovation could happen on the encoding side. The idea is so obvious that I would be astounded if this company gets anywhere. I'd wager many research teams already attempted this and were never heard from again. Also note that video compression is pretty impressive these days. A typical 2 hour 1080p movie compresses down to a handful of GiB. Compare that to a typical 1080p action game which is easily ten times that big, because storing all the meshes and textures takes a lot of space, it turns out.
- bob1029 6y ago
  I think the fundamentals are already in place to support direct encoding of 3 spatial dimensions (or more) using essentially the same ideas as in JPEG/MPEG: https://en.wikipedia.org/wiki/Discrete_cosine_transform#3-D_DCT-II_VR_DIF https://en.wikipedia.org/wiki/Discrete_cosine_transform#3-D_... Not sure what the pros/cons this kind of approach might bring, or if it's already being used in some areas. I find it really hard to believe that 3d modeling software ecosystem has not experimented with the idea of lossy compression. Does the loss of high frequency information cause more adverse outcomes in 3d models than in 2d models? Seems like this could be useful in games or other creative areas where mathematical precision of the final compressed artifact is not essential.
fredley 6y ago
I actually love the slightly 'off' aesthetic on the simpsons video, and I think there's some interesting creative space where this algorithm is deliberately de-tuned for interesting results.
dharma1 6y ago
This why Flash was such a nice format for some things (not going into the pitfalls of Flash - but in some ways it was fantastic). I guess these days we have animated SVG, and https://lottiefiles.com/ https://lottiefiles.com/ is getting some traction - but these require you to export in a specific format of course, you can't just convert/trace a bitmap movie with these. And SVG or Lottie aren't designed for longer/streaming vector animations, and they don't carry synchronised or streaming audio - Flash did all of those things. Vectorisation of bitmap images does have some artifacts, as is evident in the Simpsons demo on this website - when possible, you should export in a vector movie format directly from the vector animation software. It is kind of depressing that we don't have an open standard for vector movies (with sound), over a decade after Flash was killed. Sometimes it feels like technology stops or moves backwards.
- choeger 6y ago
  Actually I wonder why we should have a vector video format in the first place. Would it not be much simpler to render a series of vector images and their transformations directly to a standard video code? I mean all the modern video codecs have vector elements afaik, and most have motion prediction. One could just leave out the whole messy inference and directly tell the decoder "here is a rectangular shape, moving with that speed in this direction", no?
  - nitrogen 6y ago
    Flash vector animations are infinitely scalable from a very tiny file size. The vectors in modern codecs are motion vectors. They don't generally have any concept of drawing primitives like splines, polygons, fills, gradients, etc., and they are raster formats so they have a fixed native pixel size. The idea of a vector codec isn't new though, and prior attempts have been made, though I don't remember enough details to find a reference. Just for anyone who might not be familiar, "vector" in this context refers to the graphics you would generate in a tool like Inkscape, where you can define complex geometric shapes with just a few datapoints. "Raster" graphics are what Gimp works with, where each pixel is specified (effectively) individually.
  - 6y ago
vslira 6y ago
That's really interesting, excited to follow the project and see more!
iworkfromhome 6y ago
This is really cool. So the future video inspiration that will be built using only the code, without shooting again. Because everything can only be made with code. Cool!
laserpistus 6y ago
We are using a similar concept in production. Rendered videos for online education within programming. It is a limited usecase of course, but brings the same benefits: 1/100 bandwidth; crisp rendered text/content instead of rasterized; Easy editing; and the content is searchable / indexable. Using html gives us some additional benefits in that we can combine rendered with rasterized content. We also get access to a lot more advanced functionality through the browser context.
- bufferoverflow 6y ago
  But how do you convert a video to text? Or do you record straight to text?
  - laserpistus 6y ago
    We provide the recording tool, so it is recorded straight to rich data.
gruez 6y ago
>Our first vectorized proof of concept for animations is a 17 second clip of the Simpsons located here. Keep in mind, our technology is still at a very early stage, and this is much optimization work left to be done. >https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.html https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.h... There isn't a raster version to compare to, but that looks noticeably worse than what I'd expect from a raster version. There's a lot of artifacting when there's motion, and the linework looks.... off. >https://files.vectorly.io/demo/khan-20kbps/index.html https://files.vectorly.io/demo/khan-20kbps/index.html The khan academy one looks much better, although there's still some minor artifacting, eg. when the mouse comes close to the "O" in "O_2" changes a bit.
- flgr 6y ago
  Hm, unless I miscalculated the bit rate they use for the Simpsons demo is about equivalent to YouTube's 240p rendition (which uses 257 kbps; they use 250 kbps). Looking at https://www.youtube.com/watch?v=a1nmZq1KEHk https://www.youtube.com/watch?v=a1nmZq1KEHk at 240p I think their Simpsons rendition looks better. But that's probably because of the low resolution. Makes me wonder how well H264 would perform with the same bit rate but a higher resolution. I think it'd be best to have a 1 to 1 comparison between the two right on the vectorly site. I'm thinking that they'd have done that if they believed they were already ready to outperform, but that's a bit of speculation.
  - xhkkffbf 6y ago
    I think their demos are pretty good. But it would be nice to offer some comparison of the bandwidth savings. That being said, they're just starting and I'm sure they could find some fat to squeeze out of the data stream.
- pixelhorse 6y ago
  Most hand drawn animation wont't actually vectorize well, especially the ones from the hand drawn era, where every cel is a real painting: https://caseantiques.com/item/lot-738-the-simpsons-animation-cel-homer-bart-2/ https://caseantiques.com/item/lot-738-the-simpsons-animation... Look at all that line detail. It creates subtle gradients in the final film, which don't readily transfer to vector graphics. In this case, I think with some better pixel-based filtering before vectorization, they could've gotten a better result. However, the much bigger problem is that the vast majority of animation uses paintings for backgrounds that aren't just solid lines and colors.
cphoover 6y ago
Why use computer vision when likely most of these animations come from some software that can/could output a vectorized video format? You wouldn't have any conversion artifacts that way.
- mattigames 6y ago
  They do not, the vast amount are PNG files where the creator of the video themselves didn't have access to the vectors. It also is completely unrealistic to convince everyone to use a new video format, not to mention the gigantic amount of videos from the past (e.g. cartoon tv shows from the 90s) that no longer have any source material available and all that remains are digitalized copies.
zcw100 6y ago
Sounds like https://github.com/fogleman/primitive https://github.com/fogleman/primitive but for video
gardaani 6y ago
Domain specific video formats could be the future. They can reduce bandwidth requirements and offer features not available in pixel videos. One good example is https://asciinema.org https://asciinema.org , which plays back terminal sessions. The text in the "video" is selectable!
- neoncontrails 6y ago
  Wow. Could you embed these files in a GitHub README.md, for instance, or does the playback require a specific decoder? I've long wondered how developers manage to capture snazzy terminal demos of their projects, as the Linux tools I've tried using for this purpose generate bafflingly huge files (~750mb for roughly a minute of content).
  - Kuinox 6y ago
    Sadly no, people include a preview image that link to asciinema
  - sksavant 6y ago
    Apparently, you can after rendering the asciicast file to SVG: https://github.com/marionebl/svg-term-cli https://github.com/marionebl/svg-term-cli
tabtab 6y ago
If the vectors also relatively smoothly morph in time, then monitor-side interpolation (including motion smoothing) wouldn't be needed and directors would have more control over how interpolation is done. They've complained about monitors trying to do too much. It seems pixels are becoming obsolete for video and movies.
jcims 6y ago
Curious how ML would train on vectorized video instead of rasterized.
bane 6y ago
I imagine is a posterization preprocessing step would make this simpler and we could have very low bandwidth "video". If this could be done in real time, it would dramatically lower the bandwidth required for two-way video chat.

319 points by xanthine 6y ago

deleted 6y ago

[deleted]

Is there market demand for this? In a world where videos are still sent around as multi-megabyte gif files, and audio clips are still distributed with a random slideshow on YouTube, I think a lot of users aren't so bothered about efficiency - they just want the simplest thing that works. Lots of music is most efficiently stored as MIDI, yet how many songs on iTunes are midi? For video, raster is king because it works for everything.

vhiremath4 6y ago
It's a good point. Internet speeds are also quickly increasing in areas getting online for the first time.
xanthine 6y ago
They've listed some uses for it — animated videos, e-learning (the likes of Khan Academy). While I doubt that animation would shift to vector video any time soon, it indeed would help e-learning a lot, speaking from the side of the content distributors (less size). The only benefit for end users might be faster streaming, but yeah, I agree that the average user can't be bothered about it.
- deleted 6y ago
  [deleted]
- egypturnash 6y ago
  Lots of animation is done in vector tools to start with. If this format goes anywhere it would probably be pretty easy to add exporters to Toon Boom, Animate, OpenToonz, and whatever else people are currently using - I left that industry a decade and a half ago so I'm not too up on what everyone's using right now.
xanthine 6y ago
>Lots of music is most efficiently stored as MIDI, yet how many songs on iTunes are midi? While storing music as its component instruments is indeed size efficient, converting existing music into MIDIs using some ML/AI algorithm is not yet effective. Vectorly seems to convert _existing_ raster videos into vector ones using some combination of computer vision algorithms.
- rjmunro 6y ago
  Playback of a MIDI file also doesn't return the original sound quality, it plays the same notes with generic sampled instruments. For high quality samples of some instruments, this can be OK, but for others, it sounds terrible. As for lyrics, you have to use a speech synth... https://www.youtube.com/watch?v=bJjxy4V00r0 https://www.youtube.com/watch?v=bJjxy4V00r0
kevingadd 6y ago
IMO the biggest problem is that even for enterprise/large-scale-website scenarios, the combination of the product cost, the cost of downloading the decoder on each machine, and the increased cpu/gpu load (resulting in lower battery life) can drown out any advantage from the size reduction. The fact that it only works for a subset of video content is another problem. Switching to H265 or AV1 is way more compelling in practice because it works for everything, it's hardware accelerated, the decoders are off-the-shelf, and the quality is significantly improved. A modern GPU can both encode and decode H265.
black_puppydog 6y ago
If you could drastically reduce the size of videos, self-hosting video distribution pages for e.g. training material would suddenly become feasible, if not for individuals then for small companies.
- kevingadd 6y ago
  If you try to do it, you'll find that self-hosting for these scenarios is surprisingly cheap. You might be able to just use YouTube, anyway.
  - black_puppydog 6y ago
    Well, youtube is exactly not self-hosted. For one reason or another, you might really not want to have to deal with youtube (or vimeo for that matter) and it would be great to have the possibility to opt out of using such a service.
- laserpistus 6y ago
  Yep, we serve around 50.000 hours of video in a month with a similar format and have ~0 extra bandwidth fees.
jpalomaki 6y ago
In some parts of the world mobile data is expensive compared to income and this is likely limiting consumption of services like online video. Random link with some pricing information for Africa [1] claims prices are up from $0.50/GB. [1] https://kenyanwallstreet.com/mobile-data-pricing-2020-report-somalia-has-the-cheapest-mobile-data-in-africa/ https://kenyanwallstreet.com/mobile-data-pricing-2020-report...
gokhan 6y ago
Netflix and others would be interested in mega bandwidth gains for animated content. They usually own both ends of the stream so they can deploy whatever they want.
laserpistus 6y ago
We (scrimba.com) actually do something similar for e-learning within programming. And it definitely makes sense for both students and as a business. - Our videos use about 1/100 bandwidth of a video. This matters for rendering (fans running when watching HD?) and to people who live in places with shitty ISPs like in Germany - It is rendered and thereby crisp as a Pringle - No codec needed as we render it as html - When you pause a video you actually have the text there, not pixels. For programming that allows you to copy and change things! - We store the context it was recorded in (dev environment) so you can change and run code in the "video" itself. This changes the pedagogy you can do in a video to be more interactive and hands on.
dimatura 6y ago
It's not really the same. True, MIDI won't work for say, iTunes, because it doesn't capture a lot of the information needed to recreate the audio of your average song. But for certain types of video, you can capture everything you need to recreate it faithfully with a vector format, at much higher compression rates. I think a better analogy is SVG -- which is not as popular as raster formats in the web, but it certainly has its place. (As an aside, MIDI is a useful format in its own right, and is still alive and well in the music-making world, even if from a technical perspective it's sort of outdated in 2020.)

vhiremath4 6y ago

Woah this is incredible. Even with the increased CPU demands, this is a very large bandwidth savings. >In practice however, DRM, streaming , analytics and ad placement also require javascript logic to function in web runtimes, so in real word settings web-video playback can and does use a non-trivial amount of CPU time. I'm a little skeptical of this claim. No idea how much CPU is used for DRM, but I can't imagine it's on the order of multiple percentage points.

chmod775 6y ago
> Woah this is incredible. Even with the increased CPU demands, this is a very large bandwidth savings. GPUs happen to be very good at drawing shapes. So there's probably a lot of room for improvement, probably even enough that these kinds of videos could be faster than decoding "normal" video (for which modern CPUs and GPUs have dedicated silicon). This doesn't mean that I'm not entirely skeptical of this, especially because this probably will fail spectacularly for animation that isn't just vector graphics, which is most animation.
- kevingadd 6y ago
  GPUs are good at drawing shapes, but that doesn't actually mean that using the GPU to rasterize high-quality polygons will be fast. In practice most production solutions for this run on a combination of CPU and GPU and use complex shaders to achieve acceptable quality. If you're rendering regular alpha-blended or opaque triangles that's very fast, but they're not going to be anti-aliased and stuff like cubic splines will need to be triangulated on the CPU or GPU. Libraries like Pathfinder and Slug are the current state of the art and they are very complicated. In comparison, video acceleration on most modern GPUs is done using dedicated silicon that typically has its own performance budget (so it doesn't eat into the GPU time to run shaders or your compositor).
  - chrismorgan 6y ago
    Pathfinder and its ilk would breeze through this sort of thing, and are still improving constantly too, because so comparatively little work has been done in the space. Guessing very roughly based on the figures in https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fast-2d-rendering.html https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fas... (and Pathfinder is integrating some of the techniques there), I’d estimate that piet-gpu and soon Pathfinder, running on Intel integrated graphics, could comfortably draw each frame of this Simpsons sample video in well under one millisecond (for an effective maximum of 6% GPU/CPU usage), probably even under a tenth of a millisecond (0.6%). (Admittedly you could easily create a more complicated video which would be much more draining, e.g. paris-30k frames would struggle to hit 120fps on integrated graphics. Generally speaking it’s easier to make pathological cases in vector encodings than raster. But let’s forget about that.) At that point, the hardware separation of video decoder and other graphics stuff (which is a strong point in general, because it’s so computationally expensive) is much less important. You speak of Pathfinder and Slug as being “very complicated”, and indeed they are; but video decoding is hardly any less complicated, and much more computationally expensive.

katmannthree 6y ago

It sounds cool but given that the demo looks like this [0] on my bog-standard Windows 10 PC with Chrome (and Firefox and Edge) I'm assuming they've still got some bugs to work out... If it's working for anyone here I'd love to see a screen capture of the proper rendering. [o]: https://i.imgur.com/YO42u2C.png https://i.imgur.com/YO42u2C.png

hackstack 6y ago
I just got a black box on my Windows system. Looks pretty good in Firefox on the Mac though: https://imgur.com/a/rqPgQFZ https://imgur.com/a/rqPgQFZ Edit: Here it is in video form https://imgur.com/a/B0UOigX https://imgur.com/a/B0UOigX
- katmannthree 6y ago
  Thank you! That video really is impressive.
themmes 6y ago
Interesting, maybe it's missing a coordinate, could it be missing the repeating first coordinate? Surprising to see that surfaces of the same color are made up so many polygons, I guess that can be made even more efficient?
willis936 6y ago
Aha. Reminds me of what I saw for two weeks when I wrote an svg renderer in MATLAB. I only got it to work with a subset of the standard and was reliant on the conventions of inkscape export for parsing. You can do a lot with just beziers, circles, and straight lines.
aembleton 6y ago
https://imgur.com/3Cgs7Z8 https://imgur.com/3Cgs7Z8 Works well on Firefox 78.0.2 on Ubuntu 20.04

xanthine 6y ago

Their android SDK (no release version yet) is available at their Github repo, and so are their bulk upload tools (for talking to their servers using, I guess, a pay to use API).

acd 6y ago

I think your technology would be useful for restoring old animated videos. Plus it would be useful for The catchy intro animations used on startups to demonstrate their technology. Also would be nice to use when you have bad internet connection speed to watch e-learning material animations. For e-learning you may need a hybrid M3u like Playlist approach with video for the presenter and vector graphics for the screen casts. Manga videos would probably also compress well. Children animated videos. What if you would reduce the color space vectorize of ordinary video to for example 8 colors and smooth out the noise to make large flat surfaces could you compress it with vectors?

kevingadd 6y ago
The average anime or children's cartoon won't compress well with this approach unless it's authored with it in mind - if you look at the average anime episode it's full of raster effects, layered gradients, 3d perspective, and mixed in 3DCG. Worse still the anime is typically sent to the broadcaster in an already-lossy format so the vectorizer will have to handle macroblocking and noise. Here are two random screen captures from anime airing this season: https://pbs.twimg.com/media/EdP8gc4WAAAbkWy?format=jpg&name=large https://pbs.twimg.com/media/EdP8gc4WAAAbkWy?format=jpg&name=... https://pbs.twimg.com/media/EdPwGmyXoAIjlGw?format=jpg&name=large https://pbs.twimg.com/media/EdPwGmyXoAIjlGw?format=jpg&name=... Suffice it to say that vectorizing this content would be difficult and the vector representation wouldn't be particularly small.

chmod775 6y ago

While this is pretty cool, this naive approach will fail spectacularly for animation that isn't just vector graphics, which is most animation. This might have a future as part of a regular video codec, being used when there's mostly vector graphics on screen (or just for those areas that are vector graphics).

royjacobs 6y ago

Reminds me of the Spaceballs demos State of the Art and Nine Fingers [1]. Released in 1993 iirc. [1] https://www.youtube.com/watch?v=PPoYzwib7JQ https://www.youtube.com/watch?v=PPoYzwib7JQ

teddyh 6y ago

Isn’t this basically the same thing which was famously used to achieve full motion video in the 1992 Amiga demo “State of the Art”¹ and improved one year later in the followup “9 Fingers”²? 1. https://www.pouet.net/prod.php?which=99 https://www.pouet.net/prod.php?which=99 https://www.youtube.com/watch?v=J2r7-ygXOzo https://www.youtube.com/watch?v=J2r7-ygXOzo 2. https://www.pouet.net/prod.php?which=100 https://www.pouet.net/prod.php?which=100 https://www.youtube.com/watch?v=tGetanBEKK8 https://www.youtube.com/watch?v=tGetanBEKK8

emmanueloga_ 6y ago
Apparently, there's not much info out there about how 9 Fingers was made. I found this [1]. A genetic algorithm takes quite a while to come up with a pic, I wonder how long it took them in 1992! [2] Kind of related, this series of articles on the vector encoder for Another World [3]. 1: http://www.mos6502.com/friday-commodore/9-fingers-the-infamous-amiga-demo/ http://www.mos6502.com/friday-commodore/9-fingers-the-infamo... 2: https://chriscummins.cc/s/genetics/ https://chriscummins.cc/s/genetics/ 3: https://fabiensanglard.net/another_world_polygons/index.html https://fabiensanglard.net/another_world_polygons/index.html
- metadata 6y ago
  I can give you a glimpse into the past. Amiga had a very slow CPU (7 MHz) and each instruction took between 4 and 40-ish cycles. However, it had 3 special purpose co-processors. For these demos, Blitter (very fast bit copy and memory fill functionality) and Copper (change palette and even resolution on a specific vertical scan location) were essential. The technique is the same one we used to make 3D games - you would draw an outline (taking care to write a exactly single pixel on boundaries, your line-drawing routines had to be written to take that into account) and let Blitter fill this. Amiga had 6 overlay planes you could use for up to 64 colors (well, it was actually up to 4096 colors with caveats, but let's ignore that for a moment). So, if you write a filled outlines (time shifted) in those 6 planes, you'd get this nice blurred effect. My guess is they manually wrote pixel coordinates for the images, connected them with lines, filled with Blitter, repeated for each frame. It's how things were done back then. Without Internet we had a lot of free time (I was a kid in high school). Those special purpose chips were magical. At the time of IBM XT and its green on black terminals, Amiga was a space shuttle in comparison. For example, my game used to have upper part of the screen in lower resolution (where we painted vector graphics), lower part in higher (where we had game stats).

villgax 6y ago

Not ready yet, just hyping SVG videos for now

emmanueloga_ 6y ago

It just boggles my mind that the front page of a page that offers a "patented vector-transcoder converts video to a vector format, reducing bitrates" doesn't include such video on its front page.

jlmorton 6y ago
Well, it's not at the top of the page, but they did include a short proof-of-concept video of the Simpsons at the bottom of the page. https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.html https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.h...
- WrtCdEvrydy 6y ago
  That is broken on my computer :(
  - Manfred 6y ago
    I suspect that they only focused on particular versions of Chrome during development because it's a proof of concept.
    - WrtCdEvrydy 6y ago
      Yeah, but it is Windows 10 on the latest chrome... pictures other people have linked look cool though.
      - beernutz 6y ago
        Seems to work fine on Win 10 and latest chrome for me. I wonder if it might be related to turning on/off the GPU rendering in chrome maybe?
    - akdor1154 6y ago
      works on Firefox here for me
    - 6y ago

villgax 6y ago

Even their job postings on LinkedIn look like somebody anger-wrote it. For an Image Processing role, they've written no ML/DL engineers which is obvious but still, as if Computer Vision isn't any way linked to whatever it is they are trying to do with compression.

bkm 6y ago

One overlooked usecase is RDP and VNC-like concepts. Right now Appetize (App streaming service) uses raster graphics. Vector would allow high-FPS streaming of Apps, making running Apps in the cloud a realistic option.

kevingadd 6y ago
Most modern application UI can't really be encoded as vectors, there's lots of raster stuff in there plus bitmaps. Many mobile apps right now are a big pile of PNGs and/or raster filters (like drop shadows, etc). So you end up actually needing to ship scene graphs along with bitmaps. You can certainly leverage that, but it also increases the resource demands (and thus power demands) on the client.
GordonS 6y ago
I thought RDP was already a bit more intelligent than "only" sending raster frames, sending things like window dimensions and such to be rendered on the client?
- bob1029 6y ago
  RDP is probably more intelligent than most people are aware. Try remoting into a machine over RDP and playing a youtube video occupying a large portion of the screen. You may be surprised to find that it plays nearly perfectly, even over non-ideal network conditions (wifi). Try this same exercise with VNC and you will experience a frame maybe once every other second. I am not sure exactly the heuristics involved, but RDP is certainly switching between modes of operation based on what kind of visual information is on the screen.
  - ttsda 6y ago
    RDP also works more reliably over mobile connections than VNC, in my experience
IshKebab 6y ago
Nah, raster encoders can do a pretty great job on vector-style data, whereas vector encoders can't do anything with raster-style data. In other words what happens when you play a video or view a photograph? Are you going to auto-detect that and switch modes? You'll end up just recreating a modern video codec. The best thing to do is just use a modern video codec and make sure it works well with text and sharp edges.

nautical 6y ago

If I understand this correctly, will it also allow to send updates to video on the fly ? Example : Change video from X to Y time to new vectors [V1 ..] ?

black_puppydog 6y ago

This reminds me of a paper from 2005 by Daniel Sýkora et al. [1] which tries something very similar, with the specific use case of animation video. The authors describe it best in the abstract I think: > Video Codec for Classical Cartoon Animations with Hardware Accelerated Playback > We introduce a novel approach to video compression which is suitable for traditional outline-based cartoon animations. In this case the dynamic foreground consists of several homogeneous regions and the background is static textural image. For this drawing style we show how to recover hybrid representation where the background is stored as a single bitmap and the foreground as a sequence of vector images. The idea of using prior knowledge about the nature of the content to decide on an encoding scheme makes intuitive sense to me, though I'm not a codec person so I don't know how feasible it would be to make these ideas into the hardware-accelerated codecs we know from other methods. Of course, these methods would make the most sense when used directly by the animation studios during export, not as an afterthought. But I'll take what we can get. By the way, the corpus of Sýkora's works [2] is really really impressive in my opinion. He gave a talk in my institute while I was researching methods around neural style, and his take on parametric models, paired with the quality (and speed!) of his results, really left a mark, if not to say they made me seriously question wtf I was doing there. His work is strictly tailored to a professional animation / video production setting, so it seems extremely applicable compared to the toy-like nature of neural style methods. That is not to say he doesn't know about those. His team's recent papers actually fruitfully combine the two. [1]: https://link.springer.com/chapter/10.1007/11595755_6 https://link.springer.com/chapter/10.1007/11595755_6 [2]: https://dcgi.fel.cvut.cz/home/sykorad/ https://dcgi.fel.cvut.cz/home/sykorad/

dharma1 6y ago
Daniel's video style transfer tech was used in this - it's really cool. https://ebsynth.com/ https://ebsynth.com/

karteum 6y ago

In terms of stream format, it seems that BIFS / MPEG 4 part 11 originaly aimed at the same purpose (probably in a more efficient manner than textual SVG), isn't it ? https://en.wikipedia.org/wiki/MPEG-4_Part_11 https://en.wikipedia.org/wiki/MPEG-4_Part_11

okaleniuk 6y ago

It's awesome, but it also brings back memories of the flash-animation.

cypressious 6y ago
Exactly. The funny thing is that Flash animations where the more straight-forward way. You create vector animations and export them as such. Obviously, this is more universal. You can take any input as opposed to something that was vectorized to begin with.

sneak 6y ago

Patenting algorithms is supposed to be impossible. Companies that ignore this and patent the “system and method” for implementing algorithms are being jerks.

villgax 6y ago
They're using this to get VC clout & footing in job posts since they don't have any revenue the only way of convincing people for them is to say we have a 'patent'. Almost as if a shark-tank episode wrote their pitches.

rhn_mk1 6y ago

Maybe this finally brings back the crispy scaling quality of Flash videos. They seem to be released in lossy raster formats these days.

imtringued 6y ago

Can this technique be used to generate a CAD sketch from a photograph?

detritus 6y ago
Technically, probably, although there are exisitng and likely better vector conversions for none-animated purposes. The problem with CAD is that whilst a conversion to a rough sketch will be possible, 'CAD' implies very high accuracy or exactness, and the amount of work required to render a converted sketch accurate will likely trump that by just doing it manually in the first place. I suppose in future some ML-aided setup that takes a drawing in AND focuses on dimensional accuracy might fare better than this video-centric solution.

wodenokoto 6y ago

As a kid, I used to trace frames from the Simpsons in Macromedia Flash (later Adobe, later discontinued) as a way of creating high-res images, so reading this article really hit home for me! While the output of this algorithm (just like my traces) isn't as faithful to the source material as, say, H.264 is, the result looks great and has an amazing style. This might be a great target for mobile-first webtoons.

fefe23 6y ago

AFAIK MPEG-4 experimented with encoding 3d objects but it never took off. As usual for MPEG they did not specify how to get the 3d data from a scene but how to encode them, actually how to decode them, so that innovation could happen on the encoding side. The idea is so obvious that I would be astounded if this company gets anywhere. I'd wager many research teams already attempted this and were never heard from again. Also note that video compression is pretty impressive these days. A typical 2 hour 1080p movie compresses down to a handful of GiB. Compare that to a typical 1080p action game which is easily ten times that big, because storing all the meshes and textures takes a lot of space, it turns out.

bob1029 6y ago
I think the fundamentals are already in place to support direct encoding of 3 spatial dimensions (or more) using essentially the same ideas as in JPEG/MPEG: https://en.wikipedia.org/wiki/Discrete_cosine_transform#3-D_DCT-II_VR_DIF https://en.wikipedia.org/wiki/Discrete_cosine_transform#3-D_... Not sure what the pros/cons this kind of approach might bring, or if it's already being used in some areas. I find it really hard to believe that 3d modeling software ecosystem has not experimented with the idea of lossy compression. Does the loss of high frequency information cause more adverse outcomes in 3d models than in 2d models? Seems like this could be useful in games or other creative areas where mathematical precision of the final compressed artifact is not essential.

fredley 6y ago

I actually love the slightly 'off' aesthetic on the simpsons video, and I think there's some interesting creative space where this algorithm is deliberately de-tuned for interesting results.

dharma1 6y ago

This why Flash was such a nice format for some things (not going into the pitfalls of Flash - but in some ways it was fantastic). I guess these days we have animated SVG, and https://lottiefiles.com/ https://lottiefiles.com/ is getting some traction - but these require you to export in a specific format of course, you can't just convert/trace a bitmap movie with these. And SVG or Lottie aren't designed for longer/streaming vector animations, and they don't carry synchronised or streaming audio - Flash did all of those things. Vectorisation of bitmap images does have some artifacts, as is evident in the Simpsons demo on this website - when possible, you should export in a vector movie format directly from the vector animation software. It is kind of depressing that we don't have an open standard for vector movies (with sound), over a decade after Flash was killed. Sometimes it feels like technology stops or moves backwards.

choeger 6y ago
Actually I wonder why we should have a vector video format in the first place. Would it not be much simpler to render a series of vector images and their transformations directly to a standard video code? I mean all the modern video codecs have vector elements afaik, and most have motion prediction. One could just leave out the whole messy inference and directly tell the decoder "here is a rectangular shape, moving with that speed in this direction", no?
- nitrogen 6y ago
  Flash vector animations are infinitely scalable from a very tiny file size. The vectors in modern codecs are motion vectors. They don't generally have any concept of drawing primitives like splines, polygons, fills, gradients, etc., and they are raster formats so they have a fixed native pixel size. The idea of a vector codec isn't new though, and prior attempts have been made, though I don't remember enough details to find a reference. Just for anyone who might not be familiar, "vector" in this context refers to the graphics you would generate in a tool like Inkscape, where you can define complex geometric shapes with just a few datapoints. "Raster" graphics are what Gimp works with, where each pixel is specified (effectively) individually.
- 6y ago

vslira 6y ago

That's really interesting, excited to follow the project and see more!

iworkfromhome 6y ago

This is really cool. So the future video inspiration that will be built using only the code, without shooting again. Because everything can only be made with code. Cool!

laserpistus 6y ago

We are using a similar concept in production. Rendered videos for online education within programming. It is a limited usecase of course, but brings the same benefits: 1/100 bandwidth; crisp rendered text/content instead of rasterized; Easy editing; and the content is searchable / indexable. Using html gives us some additional benefits in that we can combine rendered with rasterized content. We also get access to a lot more advanced functionality through the browser context.

bufferoverflow 6y ago
But how do you convert a video to text? Or do you record straight to text?
- laserpistus 6y ago
  We provide the recording tool, so it is recorded straight to rich data.

gruez 6y ago

>Our first vectorized proof of concept for animations is a 17 second clip of the Simpsons located here. Keep in mind, our technology is still at a very early stage, and this is much optimization work left to be done. >https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.html https://files.vectorly.io/demo/v0-2-simpsons-250kbps/index.h... There isn't a raster version to compare to, but that looks noticeably worse than what I'd expect from a raster version. There's a lot of artifacting when there's motion, and the linework looks.... off. >https://files.vectorly.io/demo/khan-20kbps/index.html https://files.vectorly.io/demo/khan-20kbps/index.html The khan academy one looks much better, although there's still some minor artifacting, eg. when the mouse comes close to the "O" in "O_2" changes a bit.

flgr 6y ago
Hm, unless I miscalculated the bit rate they use for the Simpsons demo is about equivalent to YouTube's 240p rendition (which uses 257 kbps; they use 250 kbps). Looking at https://www.youtube.com/watch?v=a1nmZq1KEHk https://www.youtube.com/watch?v=a1nmZq1KEHk at 240p I think their Simpsons rendition looks better. But that's probably because of the low resolution. Makes me wonder how well H264 would perform with the same bit rate but a higher resolution. I think it'd be best to have a 1 to 1 comparison between the two right on the vectorly site. I'm thinking that they'd have done that if they believed they were already ready to outperform, but that's a bit of speculation.
- xhkkffbf 6y ago
  I think their demos are pretty good. But it would be nice to offer some comparison of the bandwidth savings. That being said, they're just starting and I'm sure they could find some fat to squeeze out of the data stream.
pixelhorse 6y ago
Most hand drawn animation wont't actually vectorize well, especially the ones from the hand drawn era, where every cel is a real painting: https://caseantiques.com/item/lot-738-the-simpsons-animation-cel-homer-bart-2/ https://caseantiques.com/item/lot-738-the-simpsons-animation... Look at all that line detail. It creates subtle gradients in the final film, which don't readily transfer to vector graphics. In this case, I think with some better pixel-based filtering before vectorization, they could've gotten a better result. However, the much bigger problem is that the vast majority of animation uses paintings for backgrounds that aren't just solid lines and colors.

cphoover 6y ago

Why use computer vision when likely most of these animations come from some software that can/could output a vectorized video format? You wouldn't have any conversion artifacts that way.

mattigames 6y ago
They do not, the vast amount are PNG files where the creator of the video themselves didn't have access to the vectors. It also is completely unrealistic to convince everyone to use a new video format, not to mention the gigantic amount of videos from the past (e.g. cartoon tv shows from the 90s) that no longer have any source material available and all that remains are digitalized copies.

zcw100 6y ago

Sounds like https://github.com/fogleman/primitive https://github.com/fogleman/primitive but for video

gardaani 6y ago

Domain specific video formats could be the future. They can reduce bandwidth requirements and offer features not available in pixel videos. One good example is https://asciinema.org https://asciinema.org , which plays back terminal sessions. The text in the "video" is selectable!

neoncontrails 6y ago
Wow. Could you embed these files in a GitHub README.md, for instance, or does the playback require a specific decoder? I've long wondered how developers manage to capture snazzy terminal demos of their projects, as the Linux tools I've tried using for this purpose generate bafflingly huge files (~750mb for roughly a minute of content).
- Kuinox 6y ago
  Sadly no, people include a preview image that link to asciinema
- sksavant 6y ago
  Apparently, you can after rendering the asciicast file to SVG: https://github.com/marionebl/svg-term-cli https://github.com/marionebl/svg-term-cli

tabtab 6y ago

If the vectors also relatively smoothly morph in time, then monitor-side interpolation (including motion smoothing) wouldn't be needed and directors would have more control over how interpolation is done. They've complained about monitors trying to do too much. It seems pixels are becoming obsolete for video and movies.

jcims 6y ago

Curious how ML would train on vectorized video instead of rasterized.

bane 6y ago

I imagine is a posterization preprocessing step would make this simpler and we could have very low bandwidth "video". If this could be done in real time, it would dramatically lower the bandwidth required for two-way video chat.