7 ms·

I personally don't like packages that combine aggregation or other data processing with display. There is clearly a community that likes this paradigm (plotly i

I personally don't like packages that combine aggregation or other data processing with display. There is clearly a community that likes this paradigm (plotly is similar) but I favor doing the analysis and getting the view I want as a data structure, and then plotting as is. I would love to hear the advantages of combining aggregation and plotting the way Vega does.

tibbetts 6y ago
Generally being able to interactively adjust the granularity of the data, drill down, and also to interactively filter the data and see it properly re-aggregated.
breck 6y ago
I think the Vega team would agree with you since it seems to me like they are doing exactly that with Arquero (https://github.com/uwdata/arquero https://github.com/uwdata/arquero). From my experience it's the right architecture. You cannot have a great data visualization library without a great data transformation library, but the data transformation functions should be at lower level; not provided by the visualization layer.
- kanitw 6y ago
  Vega author here. We actually disagree with the last part. For visual analysis tools (not just UI charting library), having the ability to quickly summarize data is very useful for analyses (esp exploratory ones). We are not alone with this design choice. ggplot2 in R and GUIs like Tableau also includes data aggregation as first class citizen in their tools.
  - jules 6y ago
    I've only used ggplot2 and found it fantastic compared to standard plotting tools. What has been improved about the design of these type of visualization tools since then?
  - breck 6y ago
    I think I misread the OP that they were talking about separating the implementation code at the vis and data layer (rereading I think they were talking about the user's experience). I disagree and think for a user it's great to have your visualizations and transformations work seamlessly. My memory of Vega was that it mixed data transform code with vis code. So I didn't have like a dplyr + ggplot2 combo. I thought design wise that was a mistake, because nailing the transforms from an edge case and performance perspective is hard and doing it 2x doesn't make sense to me. So decoupling the vis code from the data engine I think is better. But I took a fresh look at the Vega repo and it is indeed nicely decoupled, and the transforms look usable standalone. So maybe it sort of already has the Dplyr+GGPlot2 style decoupled architecture that I thought Arquero would bring. I had thought of Vega as "a monolith for datavis", but now looks like there's lots of smaller usable packages in there.
klysm 6y ago
I think the problem is aggregations frequently become part of the display of the data, especially if they are even a little bit interactive.
- viraptor 6y ago
  That's 99% of my use cases. If I'm plotting data, I'm doing exploration and want to change aggregation / display quickly.
domoritz 6y ago
Vega-Lite author here. The reason why we integrated the two is that we wanted to provide support for interactions with a fully reactive runtime system (https://idl.cs.washington.edu/papers/reactive-vega-architecture/ https://idl.cs.washington.edu/papers/reactive-vega-architect...). It's difficult to do that while being agnostic to different implementations of these transformations.
jryb 6y ago
I would actually turn this question around: why would you want to implement your own aggregation functionality? It's super convenient to be able to just make a histogram. And I'm sure my hexbin aggregation function would have errors in it the first time I wrote it. But these are all opt-in - if I need to make a histogram of 10 trillion datapoints and performance is critical, sure, I'll do the aggregation myself and just call the barplot function instead of the histogram function. What did bundling a histogram function take away?
nicolaskruchten 6y ago
I agree that some libraries do this but in general Plotly does not: we mostly visualize the data given, and lots of users wish we would onboard more transformation/aggregation/processing :)

tibbetts 6y ago

Generally being able to interactively adjust the granularity of the data, drill down, and also to interactively filter the data and see it properly re-aggregated.

breck 6y ago

I think the Vega team would agree with you since it seems to me like they are doing exactly that with Arquero (https://github.com/uwdata/arquero https://github.com/uwdata/arquero). From my experience it's the right architecture. You cannot have a great data visualization library without a great data transformation library, but the data transformation functions should be at lower level; not provided by the visualization layer.

kanitw 6y ago
Vega author here. We actually disagree with the last part. For visual analysis tools (not just UI charting library), having the ability to quickly summarize data is very useful for analyses (esp exploratory ones). We are not alone with this design choice. ggplot2 in R and GUIs like Tableau also includes data aggregation as first class citizen in their tools.
- jules 6y ago
  I've only used ggplot2 and found it fantastic compared to standard plotting tools. What has been improved about the design of these type of visualization tools since then?
- breck 6y ago
  I think I misread the OP that they were talking about separating the implementation code at the vis and data layer (rereading I think they were talking about the user's experience). I disagree and think for a user it's great to have your visualizations and transformations work seamlessly. My memory of Vega was that it mixed data transform code with vis code. So I didn't have like a dplyr + ggplot2 combo. I thought design wise that was a mistake, because nailing the transforms from an edge case and performance perspective is hard and doing it 2x doesn't make sense to me. So decoupling the vis code from the data engine I think is better. But I took a fresh look at the Vega repo and it is indeed nicely decoupled, and the transforms look usable standalone. So maybe it sort of already has the Dplyr+GGPlot2 style decoupled architecture that I thought Arquero would bring. I had thought of Vega as "a monolith for datavis", but now looks like there's lots of smaller usable packages in there.

klysm 6y ago

I think the problem is aggregations frequently become part of the display of the data, especially if they are even a little bit interactive.

viraptor 6y ago
That's 99% of my use cases. If I'm plotting data, I'm doing exploration and want to change aggregation / display quickly.

domoritz 6y ago

Vega-Lite author here. The reason why we integrated the two is that we wanted to provide support for interactions with a fully reactive runtime system (https://idl.cs.washington.edu/papers/reactive-vega-architecture/ https://idl.cs.washington.edu/papers/reactive-vega-architect...). It's difficult to do that while being agnostic to different implementations of these transformations.

jryb 6y ago

I would actually turn this question around: why would you want to implement your own aggregation functionality? It's super convenient to be able to just make a histogram. And I'm sure my hexbin aggregation function would have errors in it the first time I wrote it. But these are all opt-in - if I need to make a histogram of 10 trillion datapoints and performance is critical, sure, I'll do the aggregation myself and just call the barplot function instead of the histogram function. What did bundling a histogram function take away?

nicolaskruchten 6y ago

I agree that some libraries do this but in general Plotly does not: we mostly visualize the data given, and lots of users wish we would onboard more transformation/aggregation/processing :)