4 ms·
What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made
by lukev 2mo ago
What if they’re not pelicanmaxxing, but svgmaxxxing in general?
Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.
Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
- charcircuit 2mo agoI agree, other formats, both textual and binary should be tested.
- netsec_burn 2mo agoAddressed in the article, in case you're curious.
- lukev 2mo agoWell, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.) That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)
- Dylan16807 2mo agoAddressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.
- qq66 2mo agoBut that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
- ryukoposting 2mo agoHow useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good. AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
- Kamshak 2mo agoGreat for charts, illustrations etc, can do much more with SVG than pretty pictures. Good spatial understanding in an SVG can be really nice for model to have
- fiddlerwoaroof 2mo agoAI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).
- zahlman 2mo agoEven if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start. As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.
- amarant 2mo agoVibecoding a SVG based metroidvania as we speak! This is gonna be lit!
- derektank 2mo agoJust today I asked Claude Fable to take an SVG map and add a circle with a 150 mile radius around a specific city, cutting off the circle at the edge of certain boundaries. It came back a minute later with more or less exactly what I wanted, saving me maybe 5 to 10 minutes of photoshop time. It’s almost certainly not perfect, but I didn’t need it to be perfect, just a legible representation for a group of 20ish people.
- somenameforme 2mo agoOTOH I had Fable laying out a 3d broadcast room, and as part of the assets it sourced, it grabbed a camera. Makes sense. There were a lot of problems with the scene - backwards facing props and clipping everywhere. But the true coup de grace was a mansized, more like a giant sized, 35mm camera pointing at everything. I was seriously confused for a minute until I realized what had happened.
- sysguest 2mo agowell that holds IF svgmaxxing is 100% "code-writing-maxxing" ...which.. hmm I dunno if they are same or not
- wasabi991011 2mo agoI don't see why that's true. LLMs don't have to only be good at code-writing.
- tsimionescu 2mo agoNo, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding). That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.
- runarberg 2mo agoIf people need to generate good-ish SVG why not simply use a specialized model for a much better result and for far cheaper and quicker? Why do LLMs need to be able to do this as well, but worse, slower and more expensive?
- exhaze 2mo agoAgree directionally - even back in sonnet 3.5 days, I was helping some friends by showing them how to create intermediate representations for SVG building blocks mapping to parametrizable functions that can be used to make interactive SVG-rendered visualizations for various medical needs.
- yorwba 2mo agoYeah, if you want to generate an aesthetically pleasing SVG, you'd be better off asking a pixel-based image-generation model for "vector art" and then deconstructing it into an equivalent SVG with something like LayerPeeler. https://layerpeeler.github.io/ https://layerpeeler.github.io/
- Balgair 2mo agohttps://www.youtube.com/watch?v=jgYYOUC10aM https://www.youtube.com/watch?v=jgYYOUC10aM reminds me of this Key and Peele skit
- ralusek 2mo agoAlternatively Mitchell and Webb https://www.youtube.com/watch?v=_pDTiFkXgEE https://www.youtube.com/watch?v=_pDTiFkXgEE
- OutOfHere 2mo agoThe fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.
- zarzavat 2mo agoIf a model can improve at drawing "X doing Y" and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text.
- OutOfHere 2mo agoThe improvement would be limited to the template "X doing Y". It would not be a general improvement unless hundreds or thousands of diverse templates were used.
- nl 2mo agoI use LLMs for 3D CAD design in OpenSCAD. There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD. Anecdote I know, but there does seem to be generalization going on here.
- somenameforme 2mo agoWhich models have you found good for working with 3d stuff?
- nl 2mo agoChatGPT 5.5 and Sol 5.6, Fable are good. I haven't really tested Opus 4.8, but 4.7 wasn't nearly as good as ChatGPT 5.5.
- somenameforme 2mo agoInteresting! The reason I asked is I've had poor results with Fable and 3d stuff. Its spatial awareness seems poor - doing things like rotating left and then right back, and then pitching nonsensically, just to try to capture a segment of a scene for a verification pass. And its placement often results in clipping, misrotations, and so on. It could well be that the exact domain matters more than the bigger picture concepts, like 3d. One of the ever fewer reminders that this tech is still just fundamentally a token prediction algorithm.
- sandcat_ 2mo agoYes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops. Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones). Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles. (Haiku 4.5: https://imgur.com/a/N112Nxo https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)
- TomatoCo 2mo agoI've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.
- sandcat_ 2mo agoFair, I don’t know if this helps but it was Opus 4.8 (high effort) via Claude Desktop, but through my org’s LLM gateway so hard to compare really. But it’s usually pretty quick. (don’t tell my boss.)
- qq66 2mo agoI do have a private benchmark I use for that. I have a repository with an in-progress codebase and a very large planning document, I tell the new models to simply finish everything in the planning document, and I never push those changes. All of the recent models have essentially gotten it perfect which means that the models are all equivalently good for my level of needs.
- londons_explore 2mo ago
- throw10920 2mo ago> benchmarkmaxxing on a weightlifting competition If you stick to the benchpress, it's just "benchmaxxing".
- isoprophlex 2mo agoI'm about ready for maxxmaxxing, where there's just maximally more of everything all the time
- Sharlin 2mo agoI’m waiting for the inevitable counterreaction to all this maxxing business, dubbed "minmaxxing"
- Paul_Clayton 2mo agoIn D&D, min-maxing (hyphen and one x) is already a used term for character optimization, minimizing undesirables and maximizing desirables (I guess). Minimizing peaks is probably not a good strategy generally. Even low peaks may have some benefit ("if you know your problem is in this domain and performance is critical, this tool offers a 3% advantage"). Minimizing troughs may be more attractive. People generally seem to react more strongly to negatives and competitors can devise benchmarks which emphasize one's troughs. While a universal expert would be convenient and broad knowledge aids some forms of creativity, specialization has substantial advantages. Trading max performance (primary metric of concern) for some improvement in a secondary metric often makes sense and that could be an interpretation of "minmaxxing", reducing the over-emphasis of a single metric which would otherwise be maximized.
- Sharlin 2mo agoYes, it was a subtle referencing to rpg minmaxing, and also the minimax algorithm, beyond the more surface-level idea that "maximalist" trends tend to be followed by "minimalist" ones as a natural counterreaction.
- seebeen 2mo agoI love this comment.
- tempfile 2mo agoUgh, they're just goodmaxxing.
- kaliqt 2mo agoFunnily enough, not that niche, because I have tried many times to do it as part of a wider project.
- brikym 2mo agoThen it's great. A year ago I couldn't get any AI to draw a simple company logo in SVG or even convert from raster. No doubt the Pelicans put pressure on the labs to fix the awful SVG situation. Now I can even make a decent Peli in 3D.
- kelvinjps10 2mo agoSVG are really useful you can create images that don't have the AI look
- timClicks 2mo agoIf they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".
- knollimar 2mo agoPlease it's almost my primary bench for drafting understanding. I'm almost about to post that xkcd 810
- 21asdffdsa12 2mo agoAnd SVG is fantastic to draw easter eggs or hidden messages. Just a insult to the top of the 4 in 404.
- simonw 2mo agoGemini have absolutely been SVGmaxxing. They've openly talked about it.
- exhaze 2mo agoNot an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting: https://arxiv.org/abs/2210.05359 https://arxiv.org/abs/2210.05359
- 6thbit 2mo agoI think svg is a balanced test because of the level of indirection and the required 'conceptualization' of physical elements then expressed through code.
- delichon 2mo ago> What if they’re not pelicanmaxxing, but svgmaxxxing in general? Mission fucking accomplished. https://xkcd.com/810/ https://xkcd.com/810/
- nbardy 2mo agoThen they’re legitimately getting better at svg which is a valuable tool.
- Eggpants 2mo agoI've been pretty happy with LLM svg based data plots I've asked, including log scaled axises and histograms. Definitely a first world problem of course.
- mgrandl 2mo agoBut did they create those plots “by hand”, or did they use one of million open source libraries that create SVG plots? In my experience even SOTA models suck at generating SVG graphics like logos.
- Melatonic 2mo agoProbably the case but not a bad one to "max"
- salomonk_mur 2mo agoGenerating images not by pixels but by code and instructions is niche to you?
- meowface 2mo agoThey arguably might be, but I kind of want them to be!
- rtpg 2mo ago> Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own. I don't understand the truth to this point, and there's a clear difficulty in establishing structure, a thing that you don't need to actually do with raster image gen, right? Like if you say "please give me 9 circles" you expect circles in the image and not just a bunch of pixel that are vaguely circle-like, right?
- Theodores 2mo agoAs a human SVG-maxxer, I doubt there is the training data out there for true SVG maxxing. SVG has its neat SVGO tool that optimises SVG code, this is very clever but it won't spot easy wins such as when a group of elements can be mirrored, or an element re-used with a transform. Neither will it use the full smorgasbord of features. SVGO is not AI, however, it does a fab job of taking bloated files from Adobe Illustrator and getting something good to work with. The bar for SVG is really low, mostly just paths, compressed and not human readable. SVG should be a human readable format, so 'circle radius 10' rather than two hundred points at six decimal places to draw the same circle. You have to RTFM to do cool things with SVG and there is a lack of appreciation of the format amongst developers and designers.
- hahahaa 2mo agosvgmaxxing, please do it! https://xkcd.com/810/ https://xkcd.com/810/