4 ms·

We had plans to move DataModel (manages all data ops) to serverside. We even have a half baked DataModel in Scala which we thought we would do it once we unders

by yonl 8y ago

We had plans to move DataModel (manages all data ops) to serverside. We even have a half baked DataModel in Scala which we thought we would do it once we understand some usecase. But currently we have put it on hold. We would love to know your - use case - number of data points - ops on data on serverside You can mail us to eng@charts.com

deleted 8y ago
[deleted]
adilmoujahid 8y ago
One use case could be a data visualization similar to what I built in [1] To build the visualization in [1], I used 3 datasets in csv format from a kaggle competition [2], and I implemented the charts using dc.js and Leaflet.js. The charts were interactive and I could managed to filter the data even in the map. The largest dataset was 284 MB, which was still ok and didn't crash my browser. There were 2 drawbacks to my approach: 1- All the data was in the browser. If my data was bigger (~1GB), then it would crash my browser. 2- If I deploy the visualization to a server (for example AWS), then it would make the rendering extremely slowly as it has to download all the data to the browser... [1] http://adilmoujahid.com/posts/2016/08/interactive-data-visualization-geospatial-d3-dc-leaflet-python/ http://adilmoujahid.com/posts/2016/08/interactive-data-visua... [2] https://www.kaggle.com/c/talkingdata-mobile-user-demographics/data https://www.kaggle.com/c/talkingdata-mobile-user-demographic...
- yonl 8y ago
  Makes sense. Will keep you posted on the plans.
- dbetteridge 8y ago
  Just had to say thanks [1] , was one of my first reads when learning dc.js and very helpful!
greggyb 8y ago
It seems to me like you could leverage any number of analytical engines that expose relational interfaces, rather than go to the trouble of building your own relational model. What are the goals in building first, rather than integrating?
- yonl 8y ago
  I'm glad you asked this. So here is the thing with our DataModel. Every time you perform an ops on DataModel it create another instance. Now performing multiple such actions create a DAG where each node is an instance of DataModel and each edge is an operation. We have auto interactivity, which propagates data (dimensions) pulse along the network. Any node which is attached to visualiztion receives those pulses and changes the visual. So far I have not found any relational interface which exposes this DAG graph and api to user. Hence we though of building this. Having said that, we might use some established relational interface and do the propagation ourself.
  - greggyb 8y ago
    The implementation you are discussing sounds pretty elegant. I am most familiar with Power BI from a data viz perspective, but have used most of the enterprise viz tools out there. The thing that always struck me about Power BI (and also Qlik) is that it is very much a model-first tool. Visualization is secondary to the model, to the extent that much of the friction I see in new users has been treating it as a reporting/layout/visualization tool when, in fact, it is a data modeling tool with a visualization engine strapped on. One of the big drawbacks with Power BI is that it has a terribly inefficient implementation for propagating filter contexts for visual interactions (this is their translation of your "auto interactivity, which propagates data (dimensions) pulse along the network"). I do not know the internal implementation, but I am relatively certain that visual interactions are ~O(N!) in the number of visual elements on a report page, based on my experience of performance scaling across a wide range of reports. Regardless, one of the best practices is to limit a Power BI report page to a small number of visualizations (recommendations of the cutoff value vary, and types of visuals can also impact this). If I understand you correctly, you are calculating the minimum set of recalculations/re-renderings necessary, based on the data element that a user has interacted with. This should be something much closer to O(N) in the number of visuals to propagate user selections to other visuals. I am making an assumption that most visuals should interact, as typically the scope of a single report should have a high degree of intersection of dimensionality across all report elements. I do not know of any analytics engine that exposes the sort of DAG and associated API you are discussing, either. The reason for my initial question was simply because that sort of engine is a product in and of itself. There are plenty of columnstore databases (and following other paradigms, but optimized for OLAP workloads) out there. It seems like biting off a lot to tackle both the data engine and the visualization tier at the same time. The big reason that I ask is that this sort of approach to visualization seems to me to benefit greatly from a data model that supports transaction-level detail. The type of interactivity that you expose is extremely powerful. I have seen interactive tools hamstrung by data models that do not allow sufficient interaction. As soon as you put interactivity in front of users, in my experience, they want to do more with the data. If you are limited to datasets that can live comfortably in the browser, that seems a showstopper to me, as it will require pre-aggregation to fit most of the datasets I've seen; pre-aggregation negates many benefits of interactive data exploration. I'll be taking a much further dive into your product either this weekend or next. I'm very interested.

by yonl 8y ago

deleted 8y ago

[deleted]

adilmoujahid 8y ago

One use case could be a data visualization similar to what I built in [1] To build the visualization in [1], I used 3 datasets in csv format from a kaggle competition [2], and I implemented the charts using dc.js and Leaflet.js. The charts were interactive and I could managed to filter the data even in the map. The largest dataset was 284 MB, which was still ok and didn't crash my browser. There were 2 drawbacks to my approach: 1- All the data was in the browser. If my data was bigger (~1GB), then it would crash my browser. 2- If I deploy the visualization to a server (for example AWS), then it would make the rendering extremely slowly as it has to download all the data to the browser... [1] http://adilmoujahid.com/posts/2016/08/interactive-data-visualization-geospatial-d3-dc-leaflet-python/ http://adilmoujahid.com/posts/2016/08/interactive-data-visua... [2] https://www.kaggle.com/c/talkingdata-mobile-user-demographics/data https://www.kaggle.com/c/talkingdata-mobile-user-demographic...

yonl 8y ago
Makes sense. Will keep you posted on the plans.
dbetteridge 8y ago
Just had to say thanks [1] , was one of my first reads when learning dc.js and very helpful!

greggyb 8y ago

It seems to me like you could leverage any number of analytical engines that expose relational interfaces, rather than go to the trouble of building your own relational model. What are the goals in building first, rather than integrating?

yonl 8y ago
I'm glad you asked this. So here is the thing with our DataModel. Every time you perform an ops on DataModel it create another instance. Now performing multiple such actions create a DAG where each node is an instance of DataModel and each edge is an operation. We have auto interactivity, which propagates data (dimensions) pulse along the network. Any node which is attached to visualiztion receives those pulses and changes the visual. So far I have not found any relational interface which exposes this DAG graph and api to user. Hence we though of building this. Having said that, we might use some established relational interface and do the propagation ourself.
- greggyb 8y ago
  The implementation you are discussing sounds pretty elegant. I am most familiar with Power BI from a data viz perspective, but have used most of the enterprise viz tools out there. The thing that always struck me about Power BI (and also Qlik) is that it is very much a model-first tool. Visualization is secondary to the model, to the extent that much of the friction I see in new users has been treating it as a reporting/layout/visualization tool when, in fact, it is a data modeling tool with a visualization engine strapped on. One of the big drawbacks with Power BI is that it has a terribly inefficient implementation for propagating filter contexts for visual interactions (this is their translation of your "auto interactivity, which propagates data (dimensions) pulse along the network"). I do not know the internal implementation, but I am relatively certain that visual interactions are ~O(N!) in the number of visual elements on a report page, based on my experience of performance scaling across a wide range of reports. Regardless, one of the best practices is to limit a Power BI report page to a small number of visualizations (recommendations of the cutoff value vary, and types of visuals can also impact this). If I understand you correctly, you are calculating the minimum set of recalculations/re-renderings necessary, based on the data element that a user has interacted with. This should be something much closer to O(N) in the number of visuals to propagate user selections to other visuals. I am making an assumption that most visuals should interact, as typically the scope of a single report should have a high degree of intersection of dimensionality across all report elements. I do not know of any analytics engine that exposes the sort of DAG and associated API you are discussing, either. The reason for my initial question was simply because that sort of engine is a product in and of itself. There are plenty of columnstore databases (and following other paradigms, but optimized for OLAP workloads) out there. It seems like biting off a lot to tackle both the data engine and the visualization tier at the same time. The big reason that I ask is that this sort of approach to visualization seems to me to benefit greatly from a data model that supports transaction-level detail. The type of interactivity that you expose is extremely powerful. I have seen interactive tools hamstrung by data models that do not allow sufficient interaction. As soon as you put interactivity in front of users, in my experience, they want to do more with the data. If you are limited to datasets that can live comfortably in the browser, that seems a showstopper to me, as it will require pre-aggregation to fit most of the datasets I've seen; pre-aggregation negates many benefits of interactive data exploration. I'll be taking a much further dive into your product either this weekend or next. I'm very interested.