4 ms·
One approach is - don't. The front end visualization of data is scale dependent and you can often build systems that are order of magnitude faster by aggregatin
by perrygeo 4y ago
One approach is - don't. The front end visualization of data is scale dependent and you can often build systems that are order of magnitude faster by aggregating/filtering the data server-side to a resolution that's appropriate for the viewing window.
Two examples:
A 400px wide line chart with time on the X axis. At the absolute maximum, you can fit 400 values, likely less. Your dataset could be GB of time-series data but you can bin it into time buckets and return the mean.
An map of the world based on Open Street Map data. If you're showing things at the continent scale, you can filter out 99+% of the map features server-side and return only relevant things (major highways and country borders for example). OSM is a 100GB dataset but maps that use this technique can be fetched and rendered in ms.
Bonus points for aggregating to a fixed grid, which allows you to aggressively cache the results.
If you're committed to handling the raw data on the client side, there's not much you can do other than write optimized web workers in wasm (effectively reinventing the aggregates and indexing provided by a database) and set realistic limits.
- loa_observer 4y agoThe target is to handle common size csv/excel files on the client-side. Supporting to 1GB may cover most cases in daily work. For larger datasets, I agree to handle them on server-side.