3 ms·

This is great, you’ve substantially added better detail to my question thank you. State caching & two-way sync is exactly the challenge I’m targeting. Backend

This is great, you’ve substantially added better detail to my question thank you. State caching & two-way sync is exactly the challenge I’m targeting. Backend optimization is straight-forward at this stage of dev (appreciate the suggestions though). What I’m seeking to do is provide dense, tabular data with rapid refreshing as users make changes on the fly. Any suggestions on how I might learn about managing the state model better? Vue 3 + Pinia is my current front runner.

solardev 2y ago
Sorry, this is a bit out of my area of expertise (especially in non-React frameworks), but until someone more knowledgeable comes along, maybe I can at least offer some links for getting started? A writeup: https://blog.logrocket.com/rendering-large-datasets-vue-js/ https://blog.logrocket.com/rendering-large-datasets-vue-js/ As for implementation, this is probably a situation where I'd reach for Someone Else's Library™ instead, at least to learn from. Maybe something like https://www.ag-grid.com/example/ https://www.ag-grid.com/example/ (try the 100k row example) with maybe a serverside data model: https://www.ag-grid.com/vue-data-grid/server-side-model-updating-refresh/ https://www.ag-grid.com/vue-data-grid/server-side-model-upda... That's a ready-made open-core implementation (paid additional features), but you can see an example of how to make an performant implementation. Often, big tables like that can use some sort of virtualization (https://www.ag-grid.com/vue-data-grid/dom-virtualisation/ https://www.ag-grid.com/vue-data-grid/dom-virtualisation/), or here's another lib that does something similar: https://tanstack.com/virtual/latest https://tanstack.com/virtual/latest There is also a JS framework benchmark at https://github.com/krausest/js-framework-benchmark https://github.com/krausest/js-framework-benchmark and a vue-pinia implementation at https://github.com/krausest/js-framework-benchmark/tree/master/frameworks/keyed/vue-pinia https://github.com/krausest/js-framework-benchmark/tree/mast... Lastly, I would say it might be worth considering how the whole dataset works, holistically, as in whether the client really needs to handle all that data at once or if it can be streamed in chunks to & from the server, maybe with a bit of buffer in every direction. It's the same windowing concept, just moved to the network. And of the data on the client, how much of it is bottlenecked by rendering vs other in-memory operations (sorting, filtering, pagination, etc.) -- can you speed that part up with any better algorithms or data structures? But it sounds like your main bottleneck right now is rendering large datasets into the DOM, so hopefully some of those above will help. At some scale, it might also be worth considering moving out of the DOM altogether and into a canvas, like https://canvas-datagrid.js.org/ https://canvas-datagrid.js.org/ or https://github.com/xwinstone/canvastable?tab=readme-ov-file https://github.com/xwinstone/canvastable?tab=readme-ov-file. The DOM isn't really made for something like that, and it's pretty amazing what the libs can do, but the canvas can be much much faster. You do lose all the built-in DOM functionality though so you end up having to recreate a lot of functionality (like search, selection, drag & drop, etc.) ------ Good luck! I hope someone comes along with more experience and wisdom to share.

solardev 2y ago

Sorry, this is a bit out of my area of expertise (especially in non-React frameworks), but until someone more knowledgeable comes along, maybe I can at least offer some links for getting started? A writeup: https://blog.logrocket.com/rendering-large-datasets-vue-js/ https://blog.logrocket.com/rendering-large-datasets-vue-js/ As for implementation, this is probably a situation where I'd reach for Someone Else's Library™ instead, at least to learn from. Maybe something like https://www.ag-grid.com/example/ https://www.ag-grid.com/example/ (try the 100k row example) with maybe a serverside data model: https://www.ag-grid.com/vue-data-grid/server-side-model-updating-refresh/ https://www.ag-grid.com/vue-data-grid/server-side-model-upda... That's a ready-made open-core implementation (paid additional features), but you can see an example of how to make an performant implementation. Often, big tables like that can use some sort of virtualization (https://www.ag-grid.com/vue-data-grid/dom-virtualisation/ https://www.ag-grid.com/vue-data-grid/dom-virtualisation/), or here's another lib that does something similar: https://tanstack.com/virtual/latest https://tanstack.com/virtual/latest There is also a JS framework benchmark at https://github.com/krausest/js-framework-benchmark https://github.com/krausest/js-framework-benchmark and a vue-pinia implementation at https://github.com/krausest/js-framework-benchmark/tree/master/frameworks/keyed/vue-pinia https://github.com/krausest/js-framework-benchmark/tree/mast... Lastly, I would say it might be worth considering how the whole dataset works, holistically, as in whether the client really needs to handle all that data at once or if it can be streamed in chunks to & from the server, maybe with a bit of buffer in every direction. It's the same windowing concept, just moved to the network. And of the data on the client, how much of it is bottlenecked by rendering vs other in-memory operations (sorting, filtering, pagination, etc.) -- can you speed that part up with any better algorithms or data structures? But it sounds like your main bottleneck right now is rendering large datasets into the DOM, so hopefully some of those above will help. At some scale, it might also be worth considering moving out of the DOM altogether and into a canvas, like https://canvas-datagrid.js.org/ https://canvas-datagrid.js.org/ or https://github.com/xwinstone/canvastable?tab=readme-ov-file https://github.com/xwinstone/canvastable?tab=readme-ov-file. The DOM isn't really made for something like that, and it's pretty amazing what the libs can do, but the canvas can be much much faster. You do lose all the built-in DOM functionality though so you end up having to recreate a lot of functionality (like search, selection, drag & drop, etc.) ------ Good luck! I hope someone comes along with more experience and wisdom to share.