6 ms·
NeuralSVG: An Implicit Representation for Text-to-Vector Generation
- toisanji 2y agoThis is a group applying vector generation to animations: https://www.youtube.com/@studyturtlehq https://www.youtube.com/@studyturtlehq The graphic fidelity has been slowly improving over time.
- gcr 2y agocan you say more? all of these videos have less than 5 views and i can't find any explanation of their process
- andy_ppp 2y agoI’m looking forward to seeing what this makes of Simon Willison’s LLM SVG generation test prompt: “Generate an SVG of a pelican riding a bicycle”. It’s quite amazing the progress we are seeing in AI and it will keep getting better which is somewhat terrifying.
- nojvek 2y agoI asked both Claude and ChatGPT o3 to "generate svg of mainland USA with black outline". Tried various models and they got it hopelessly wrong. Claude does an okay job at "Generate an SVG of a pelican riding a bicycle"
- piombisallow 2y agoThis is much more useful for actual design jobs.
- CyberDildonics 2y agoIf you can generate an image you can flatten it and if you can flatten it you can cluster it, and if you can cluster the flat sections you can draw vectors around them.
- strangecasts 2y agoThis posterization-vectorization approach is what the Flash "Trace Bitmap" tool implemented (I'm not sure if Animate still has it?), but if your image isn't originally clipart/vector art, it gives the resulting vector art a very early 2000s look...
- intalentive 2y agoI’ve always thought that generation of intermediate representations was the way to go. Instead of generating concrete syntax, generate AST. Instead of generating PNG, generate SVG. Instead of generating a succession of images for animation, generate wire frame or rigging plus script. Once you have your IR, modify and render. Once you have your render, apply a final coat of AI pixie dust. Maybe generative models will get so powerful that fine-grained control can be achieved through natural language. But until then, this method would have the advantages of controllability, interoperability with existing tools (like Intellisense, image editors), and probably smaller, cheaper models that don’t have to accommodate high dimensional pixel space.
- IncreasePosts 2y agoShouldn't the girl with the pearl earring have an earring?
- mcraiha 2y agoNo, because it is not a pearl earring. https://www.theartnewspaper.com/2023/02/08/the-girl-with-a-glass-bauble-earring-why-vermeers-painting-probably-did-not-depict-a-real-pearl https://www.theartnewspaper.com/2023/02/08/the-girl-with-a-g...
- IncreasePosts 2y agoOkay, but shouldn't she at least have a glass teardrop-shaped bauble?
- niemandhier 2y agoIt looks as if this is not autoregressive. It would be interesting to see a similar approach that incrementally works from simpler ( fewer curves ) to more complex representations. That way one could probably apply RLHF along the trajectory too.
- nbzso 2y agoSo designers, artist, musicians we are done, right? Who's next, I wonder?
- thomasfl 2y agoFinally something that can benefit artists as a sketching tool.
- nikolayasdf123 2y agovery nice. had this idea for awhile, but never had time to implement it. glad someone actually did it! great work!
- Jean-Papoulos 2y agoThis is the kind of image generation I've been waiting for. No more messing around in Inkscape (or at least, less of it) when I need a specific icon.
- cyp0633 2y agoClaude has been doing a good job generating SVGs compared to its rivals, happy to see new models bringing image generation even further
- chestervonwinch 2y agoI wonder if you can use an existing svg as a starting point. I would love to use the sketch approach and generate frame-by-frame animations to plot with my pen plotter.
- shahzaibmushtaq 2y agoI am really impressed with how it generates rough sketches because everything in the design world begins that way.
- theckel 2y agoDoes anyone know how this compares to: https://github.com/ximinng/SVGDreamer https://github.com/ximinng/SVGDreamer ?
- pizza 2y agoPrompting Claude to make SVGs then dropping them into Inkscape and getting the last ~20% of it to match the picture in my head has been a phenomenal user experience for me. This, too, piques my curiosity..!
- xvfLJfx9 2y agoClaide doesn't work at all for me when generating SVGs
- iambateman 2y agoIt depends for me on if there is an existing SVG that exists in its training set. “Make an SVG of a clock icon” is likely to work. “Make an SVG of a playground swingset with the sun setting” is not.
- varunneal 2y agoYour prompt verbatim turned out quite well, single-shot. https://claude.site/artifacts/0f696bf8-399d-42c3-93c0-296493e8774e https://claude.site/artifacts/0f696bf8-399d-42c3-93c0-296493...
- deleted 2y ago[deleted]
- jgalt212 2y agoI poked around with NeoSVG a few months back. I was not happy with the results, the computation time, or the cost. That being said, I do hope they've made big progress lately because SVGs work real nice when you have an LLM and human working in tandem (as per the comment above). https://neosvg.com/generations https://neosvg.com/generations
- deleted 2y ago[deleted]
- kelseyfrog 2y agoWhy does the fourth example show a hamburger but is labeled as a dragon?
- jsheard 2y agoAmerican cultural bias in the training data led it to infer that dragons would be turned into burgers if they were real.
- airstrike 2y agoMost likely just a clerical error, since the dragon is two examples to the left with the same caption.
- deleted 2y ago[deleted]
- dekhn 2y agobecause hamburgers aren't made from chopped ham.
- murtio 2y agoThis is really cool! I have been using Claude to animate SVG, and it has been great.
- NiloCK 2y agoI'd be interested to see examples and hear about process here, if you're willing to share.
- zellyn 2y agoThe sketch generation is wild… and apparently comes for free.
- airstrike 2y agoThis opens up lots of opportunities for document authoring tools. Really cool stuff, can't wait to try out the code once it's available.
- lewisjoe 2y agoCurious how this can augment document authoring! Can you toss some ideas?
- airstrike 2y agoI just think about how often professionals need placeholder images or doodles in their documents, but cliparts are generally terrible and actually making a nice looking drawing for those purposes is out of scope for business users and immensely time consuming... so this fills a nice gap. I'm obviously biased as a former "business user" writing a document authoring software!
- lbj 2y ago"Code coming soon" - I hope someone reposts this when there's more to dig into
- deleted 2y ago[deleted]
- dheera 2y ago[flagged]
- deleted 2y ago[deleted]
- janalsncm 2y agoI am a huge fan of this type of incremental generative approach. Language isn’t precise enough to describe a final product, so generating intermediate steps is very powerful. I’d also like to see this in music generation. Tools like Suno are cool but I would much rather have something that generates MIDIs and instrument configurations instead. Maybe this is a good lesson for generative tools. It’s possible to generate something that’s a good starting point. But what people actually want is long tail, so including the capability of precision modification is the difference between a canned demo and a powerful tool. > Code coming soon The examples are quite nice but I have no idea how reproducible they are.
- chaosprint 2y agogood point. few days ago I was thinking about restarting this project with Glicol https://github.com/chaosprint/RaveForce https://github.com/chaosprint/RaveForce RaveForce - An OpenAI Gym style toolkit for music generation experiments. Love suno but eventually I need midi or xml or some lossless samples to work with
- kadushka 2y agoI’d also like to see this in music generation. Tools like Suno are cool but I would much rather have something that generates MIDIs and instrument configurations instead. Sounds like you're looking for something like https://www.aiva.ai https://www.aiva.ai
- janalsncm 2y agoHonestly that site feels like they have a database of midis tagged by genre and pick them out randomly. It’s totally different from their demo song. I guess I’m hoping for something better. It’s also closed source, the web ui doesn’t have editing functionality, and the output is pretty disjointed. Maybe if I messed around with it enough the result would be decent.
- kadushka 2y agoFair enough. Still, for what you’ve described, Aiva is the best tool available.
- jonathaneunice 2y agoNice! Looking forward to similar textual generation of diagrams. (The Pic/Pikchr for the LLM age.)
- _1 2y agoI've had some success with converting SQL to Mermaid Markdown diagrams.
- da_rob 2y agoIt’s not PIC and not really suitable for complex diagrams, yet, but you can use Vizzlo’s Chart Vizzard to create a subset of the supported chart types (let’s say a Gantt) and then continue editing it using the chart editor: https://vizzlo.com/ai https://vizzlo.com/ai
- fosterbuster 2y agoIts a wasted opportunity not using SVG to show the examples.
- deleted 2y ago[deleted]
- TeMPOraL 2y agoAvailable in ComfyUI when? :). Seriously though, this is amazing, I'm glad to see this tackled directly. Also, I just learned from this thread that Claude is apparently usable for generating SVGs (unlike e.g. GPT-4 when I tested for it some months ago), so I'll play with that while waiting for NeuralSVG to become available.
- vipshek 2y agoThis is excellent! I think the utility of generating vectors is far, far greater than all the raster generation that's been a big focus thus far (DALL-E, Midjourney, etc). Those efforts have been incredibly impressive, of course, but raster outputs are so much more difficult to work with. You're forced to "upscale" or "inpaint" the rasters using subsequent generative AI calls to actually iterate towards something useful. By contrast, generated vectors are inherently scalable and easy to edit. These outputs in particular seem to be low-complexity, with each shape composed of as few points as possible. This is a boon for "human-in-the-loop" editing experiences. When it comes to generative visuals, creating simplified representations is much harder (and, IMO, more valuable) than creating highly intricate, messy representations.
- tasuki 2y agoAh, we should be friends! I'm not sure what else to add, except that these are exactly the thoughts I think, and it used to feel lonely ;)
- Lerc 2y agoThere is also the possibility for using these images as guidance for rasterization models. Generate easily manipulatable and composible images as a first stage then add detail once the image composition is satisfactory.
- datadrivenangel 2y agoTrivially possible with controlnets!
- SillyUsername 2y agoMy little project for the highly intricate, messy representation ;) https://github.com/KodeMunkie/shapesnap https://github.com/KodeMunkie/shapesnap (it stands on the backs of giants, original was not mine). It's also available on npm.
- gwern 2y agoHave you looked at https://www.recraft.ai/ https://www.recraft.ai/ recently? The image quality of their vector outputs seems to have gotten quite good, although you obviously still wouldn't want to try to generate densely textured or photographic-like images like Midjourney excels at. (For https://gwern.net/dropcap https://gwern.net/dropcap last year or before, we had to settle for Midjourney and create a somewhat convoluted workflow through Recraft; but if I were making dropcaps now, I think the latest Recraft model would probably suffice.)
- 1970-01-01 2y agoAside: I've been having a very hard time prompting ChatGPT to spit out ASCII art. It really seems to not be able to do it. Here is an ASCII art representation of a hopping rabbit: ``` (\(\ ( -.-) o_(")(") ``` This is a simple representation of a rabbit with its ears up and in a hopping stance. Let me know if you'd like me to adjust it!
- deleted 2y ago[deleted]
- Kiro 2y agoPretty good if you ask me. What would a proper hopping rabbit ASCII art look like?
- jansan 2y agoNot sure, but that is a sitting rabbit.
- jsheard 2y agoIt seems to have just pulled an ASCII rabbit from the training data verbatim https://old.reddit.com/r/identifythisfont/comments/ytd25m/whats_the_font_used_for_this_little_ascii_bunny/ https://old.reddit.com/r/identifythisfont/comments/ytd25m/wh...
- goeiedaggoeie 2y agoThis is very nice. I has to convert a bitmask to svg and was wishing to skip the intermediatary step so looked around for papers about segmentation models outputting svg and found this one https://arxiv.org/abs/2311.05276 https://arxiv.org/abs/2311.05276
- knoxg 2y ago[flagged]
- deleted 2y ago[deleted]
- scosman 2y agoI’ve been impressed with even applying sonnet to SVGs for animations. This looks like it could be a lot more powerful. Fun example: https://gist.github.com/scosman/701275e737331aaab6a2acf74a523830 https://gist.github.com/scosman/701275e737331aaab6a2acf74a52...
- astrodude 2y agooh, wow. this actually works. I didn't know :) thanks!