13 ms·
Automatic Generation of Visualizations and Infographics with LLMs
- dontupvoteme 3y agoIronically I've found GPT to be pretty terrible with plotting libraries like plotly/dash or even matplotlib compared to just about anything else in python.
- pplonski86 3y agoI wrote a simple wrapper around Matplotlib and ChatGPT-3.5-turbo. The LLM response is a Python code that is executed to get charts. It is working very nice. Here is a repo https://github.com/mljar/plotai https://github.com/mljar/plotai - you will find two videos in the readme. Maybe you should work on your prompts?
- dontupvoteme 3y agoHuh, neat. It never really bothered me enough/was important enough to spend specifically time on it since i was able to just hit a button to send them back for fixing but that's good to see the extra passes aren't explicitly needed.
- monkeydust 3y agoWas playing with the library this morning, the interesting part to me was the 'goal explorer' which generates the questions to asks of the data. Keen to see more research into this part specially making the questions more specific to the dataset in question and overlaying real-world situations.
- btbuildem 3y agoHonestly, that's the more interesting and more difficult part. Anyone with basic training can be coerced to slice and dice schemas and configs until pretty graphs are produced. LLMs might not even be the best for that. But knowing _what_ to look for in the data given a problem statement - that's valuable, and hard to teach. LLMs have such a broad base of "knowledge", they can be reasonably good at this in just about any domain.
- monkeydust 3y agoRight, isn't knowing what to look for a must have on the path to AGI?
- btbuildem 3y agoI would agree -- that's why (to me at least) the recent wave of LLMs is such a big deal. They make semantic contexts accessible for interaction with code logic.
- w-m 3y agoLast week I helped someone organizing and analyzing their data in Excel. As I'm using Excel only once every couple of years, I had to rewatch the wonderful "You Suck at Excel with Joel Spolsky" to be productive again. Now seeing this announcement page, I was immediately reminded of the mini-rant towards the end of the video [0]: > On average, once every three months, there's a startup that makes a thing that they say is going to be amazing, and it's just PivotTables. They're like, "It works with Excel, and it does this amazing consolidation, and slicing and dicing of all your data, and it's amazing, and we're going to make a startup. I'm going to sell this for four hundred ninety-five dollars." And that happens at least once every three months. The trouble is, the VCs usually know about PivotTables. Of course this product goes a little further, making suggestions what columns to analyze and chart with an LLM. But it's quite funny to me that this Microsoft Research product is reinventing the PivotTable (+PivotChart) part with Python and Pandas. [0]: https://youtu.be/0nbkaYsR94c?si=kkfFHZ_fyGmG3Lnj&t=2988 https://youtu.be/0nbkaYsR94c?si=kkfFHZ_fyGmG3Lnj&t=2988
- Incolotopo 3y agoTo be nitpicking here: Its not reinventing if its new. And the focus of this research was probably not to invent PivotTables but the Interface for these through LLMs
- beebmam 3y agoIt's impossible to use excel for big data, the application has soft limits due to responsiveness.
- dylan604 3y ago
- dcastm 3y agoSuper cool. Here the viz-related prompts (generation, editing, etc), for those interested: https://github.com/microsoft/lida/tree/main/lida/components/viz https://github.com/microsoft/lida/tree/main/lida/components/... I built a tool that lets you use GPT to analyze data and build interactive graphs on the browser (https://deepsheet.dylancastillo.co/ https://deepsheet.dylancastillo.co/). I may try to adapt it to use LIDA or a similar approach.
- nologic01 3y agoExcellent technical work but subject to the same major questionmarks around the morality and legality of LLM business models. From the discussion section: > Low Resource Grammars: ... LIDA depends on the underlying LLM s having some knowledge of visualization grammars as represented in text and code in its training dataset (e.g., examples of Altair, Vega, Vega-Lite, GGPLot, Matplotlib, represented in Github, Stackoverflow, etc.). For visualization grammars not well represented in these datasets (e.g., tools like Tableau, PowerBI, etc., that have graphical user interfaces as opposed to code representations) ), the performance of LIDA may be limited without additional model fine-tuning or translation. In other words, open source programmatic visualizations are required to feed the LLM, which then can, e.g., be licensed to corporates to accelerate various internal exploratory data analyses. A win-win for corporates and LLM providers. Spot the loser.
- Incolotopo 3y agoAnd what in particular is now novel or unique to the general 'issue' you mention? Most companies use OpenSource in one way or the other. Nonetheless, a company like MS has probably already build visualizers purely commercially (see excel) or/and is absolutly able to write it themselfs.
- nologic01 3y agoI am not sure what you are talking about. If I release a novel visualization library on github under some open source license I want it to be attributed to me. I don't want some specialized LLM to be lifting and offering the same visualization concepts to unnamed corporates for a hefty fee without me ever even knowing about it and these corporates pretending they don't know where that concept is coming from. It is you choice whether you think that is a problem and how "novel" it is. Theft after all has a very long history.
- cooperaustinj 3y agoEverything you described been possible since the dawn of intellectual property. Just replace LLM with "person". Furthermore, it isn't theft to learn from others' work and reproduce similar qualities.
- itronitron 3y agoThey should get rid of the infographer module as it really undermines the rest of the work.
- processing 3y agoWeird landing page - I’m expecting to see infographic examples but the images I see are of people looking at screens and smiling
- f6v 3y agoUncanny fingers.
- jgalt212 3y agoI sort of know what I'm doing with data, so I don't want LLMs building any models for me, but I do like the concept of making my lame visualizations look more professional and slicker.
- RH123 3y agoI find the quality of the code really questionable: system_prompt = """ You are a an experienced data analyst that can annotate datasets. Your instructions are as follows: i) ALWAYS generate the name of the dataset and the dataset_description ii) ALWAYS generate a field description. iv.) ALWAYS generate a semantic_type (a single word) for each field given its values e.g. company, city, number, supplier, location, gender, longitude, latitude, url, ip address, zip code, email, etc You return an updated JSON dictionary without any preamble or explanation. """ Some spelling errors, and where is number 3?
- ukuina 3y agoAll you need is attention?
- ilovefood 3y agoI took this a step further, turning charts into infographics using stable diffusion (SDXL) https://karimjedda.com/beautiful-data-visualizations-powered-by-generative-ai/ https://karimjedda.com/beautiful-data-visualizations-powered...
- w-m 3y agoThat's a really nice idea. Have you thought about making it a product?
- ilovefood 3y agoThank you for the kind words. I wouldn't even know where to start. There's a lot I can code and do, but making something into a product is something I have no experience with. What's your recommended approach? Even if open-source, curious to learn.
- aliyeysides 3y agoContinue as an open source project by building features and refining the product. Once you have enough users you can start offering premium features and support to organizations. Maybe even apply to an accelerator? Good Luck!
- ilovefood 3y agoThanks for the positivity, I'll give it a shot.
- thundergolfer 3y agoIf the product got good enough you’d have a shot at selling to Canva.
- ukuina 3y agoI would pay $5 for the occasional, chart-heavy month to upload my graphs without proprietary detail and get a few images to reroll through.
- 2devnull 3y agoThis would really be something if you could just give it voice commands. Typing is absurd amidst all this wonderful automation!
- javajosh 3y agoNo, absolutely not. How can you trust the output from such a black box system? Who is to say that the LLM won't add or remove data points to make the chart "look good"? Heaven help us if decision makers start taking this output seriously. But of course they will, because the charts will look professional and plausible, because that's what the prompt requires.
- vimesy 3y ago"You are a helpful assistant highly skilled in writing PERFECT code for visualizations. Given some code template, you complete the template to generate a visualization given the dataset and the goal described. The code you write MUST FOLLOW VISUALIZATION BEST PRACTICES ie. meet the specified goal, apply the right transformation, use the right visualization type, use the right data encoding, and use the right aesthetics (e.g., ensure axis are legible). The transformations you apply MUST be correct and the fields you use MUST be correct. The visualization CODE MUST BE CORRECT and MUST NOT CONTAIN ANY SYNTAX OR LOGIC ERRORS. You MUST first generate a brief plan for how you would solve the task e.g. what transformations you would apply e.g. if you need to construct a new column, what fields you would use, what visualization type you would use, what aesthetics you would use, etc. YOU MUST ALWAYS return code using the provided code template. DO NOT add notes or explanations." (https://github.com/microsoft/lida/blob/main/lida/components/viz/vizgenerator.py https://github.com/microsoft/lida/blob/main/lida/components/...) They prompted that things MUST be correct in their prompts and it reports any transformations it does to your data, it might give you some insight into its logic to test yourself against the data.
- computerex 3y agoTelling the LLM that it must do something is not a guarantee that it'll follow through.
- vykthur 3y agoTrue. This is an open area of research. Tools like guidance (or other implementations of constrained decoding with llms [1,2]) will likely help improve this problem. [1] A guidance language for controlling large language models. https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance [2] Knowledge Infused Decoding https://arxiv.org/abs/2204.03084 https://arxiv.org/abs/2204.03084
- mdorazio 3y agoAm I missing something here? From the video and examples this looks like it's helping you make Excel charts with less suck (slightly stylized), not really building what I would consider "infographics" in the traditional marketing sense. I guess it counts as visualizations, but not what I was expecting.
- deleted 3y ago[deleted]
- mnky9800n 3y agoIn the horsepower versus mpg example, every generated result creates new data on the plot that isn't there in the original plot. this is terrible.