4 ms·

Show HN: A work in progress large table virtualization component

4 points by graderjs 5y ago

Hello Humans and Huwomans of HN Main demo link ----------------- https://i5ik.github.io/_____/infinite/ Background ------------- I'm creating this large table virtualization component. It's not easy. The challenge is motivated by creating a basic spreadsheet component, as part of running the 7GUIs 'UI benchmark' against my fledgling web component framework, and discovering I really liked the idea of having a reusable spreadsheet / grid data component. I went looking and saw some, but it looked like a good challenge and I wanted one with the following qualities (which I didn't see anywhere else): 1. uses semantic HTML markup (so table, thead, tr, th, td, colgroup, col elements) not divs to build a table 2. could handle JavaScript scripting of cells (rather than an Excel-like DSL) 3. could drag resize columns and rows 4. could handle very large data 5. ideally, but not absolutely, uses web components, or vanilla JS instead of another framework 6. provides effortless natural scrolling (not synthetic, aka Google Sheets) on desktop and mobile 7. includes the CSS Scroll Snap API 8. make the row and column headers work by using CSS sticky positioning I built a reasonably good table component[bt] to take care of 1-3 and 5-8, but requirement 4 still eluded me. The table took ages to load (order of seconds for A-Z x 1-99) and drag resizing rows and columns was awful UX (very slow, and janky even on high spec phones). Large performant tables were not a problem unique only to me, even Palantir has had issues with it in their Blueprint UI kit[bp]. It was then that I decided I needed to create an "infinite" table (table capable of showing a portal (with a moveable window) into a arbitrary-sized grid of data. I knew this would be a challenge so procrastinated for a bit. The best thing I saw was this Sproutcore demo of a "Big Data List"[sc]. I loved how it kept a pool of unused rows, moved the rows around using absolute positioning, and printed the correct data inside them based on the current scroll position. This was my template for addressing the problem in 2 dimensions. The closest thing to what I was trying to achieve was "Fattable"[ft]. I loved how you could just keep scrolling, but disliked how it uses divs instead of table markup, uses 'transform: translate' to move the cells around (versus absolute positioning, although transform probably has the advantage of being 'GPU-able' perhaps), and how the scrolling was synthetic (ie, there was not a giant element behind the portal that you're actually scrolling over, it was just calculated by the number of 'scroll triggering' events). Basically, I wanted something that pushed as much of the cross-cutting logic (like scrolling) onto the browser, and utilized HTML/JavaScript/CSS APIs in as "natural" a way as possible (rather than trying to recreate already provided functionalities, or hack them to do something unintended). This is my general approach to UI challenges, "use the qualities of the existing materials"--it's a kind of aesthetic, but it guides how I do my work. I also loved the canvas data grid type tables [ct]. I really love the idea of trying to build a GUI in the browser using canvas (such as like C# Blazor does), but I don't want to do that right now. Other notable data grid and table implementations are include in [other] below.

graderjs 5y ago
Approach ----------- Basically the approach is like that used by the Sproutcore Big Data List demo, being: 1. construct a moving foreground window onto a giant, scrollable background element (giant, as in arbitrarily wide and high, like 200000em) 2. move rows from the 'scroll wake region' to a pool when they move offscreen because of scroll, 3. move rows from the pool to the 'scroll bowshock region' when we need new rows because of scroll, 4. don't change the position of any existing rows in the window (instead allowing them to scroll naturally), but set new rows (and remove pool rows) using absolute positioning. Except as we need to scroll across both rows and columns do all of the above in 2 dimensions, for columns as well. And also the way I detect when we need to "add or remove" rows or columns is via Intersection Observers[mi]. Major issues overcome so far --------------------------------- - It's possible to scroll it now infinitely (and performantly) for the duration of the very large background element. - Major problem was I initially was listening for intersection of cells individually on each row, but if a row had already travelled off screen due to a vertical scroll, it would not be possible for its cells to 'intersect' with the viewport element (the window, in Intersection Observer API parlance, the "root"), and so no intersection for that cell would occur, and so cells in that row would not be updated. And if they were not updated, they would not be onscreen when the row was in future removed from the pool and positioned back on screen, and so its cells could no longer be updated, as there would be no cells within that scroll viewport to intersect anymore. The solution to this was to create a logical grouping of cells into columns, and handle any intersection observations for any cell in a column, at once, and apply them to all cells in a column. So the whole column would be updated. Before I fixed this, repeated scrolling (particularly in opposite diagonal directions) would erode the cells present, gradually decreasing their number, and throwing many out of their column alignments. - Also performance was another issue that's now mostly resolved tho I think I can do better. Basically too much work was happening on handling each Intersection record and we were not ignoring records that triggered the same work to be redone (such as intersection events from each cell in a column). These redundant records are now ignored (but mostly in a haphazard manner that could cause browser dependent bugs). Next steps ------------- - Add all of the outstanding features (scroll snap, row / col size dragging) from the table component. - Style it to make it look nice - Fix the outstanding border 'off by 1' bugs. - Sync up the 'view portal' with a 'data portal' (movable portal over a very large set of data), and print the correct data corresponding to the current scroll position in the table - Add the row and column headers using CSS sticky. This is the only comment I'll make in this thread, thanks for reading, and hit me up with questions at my email if you want: cris@dosycorp.com [bt]: https://i5ik.github.io/_____/7guis/ https://i5ik.github.io/_____/7guis/ (scroll down to the last 7GUIs component) [bp]: https://github.com/palantir/blueprint/issues/390 https://github.com/palantir/blueprint/issues/390 [ft]: https://fulmicoton.com/fattable/index2.html https://fulmicoton.com/fattable/index2.html [sc]: https://showcase.sproutcore.com/#demos/Big%20Data%20Lists https://showcase.sproutcore.com/#demos/Big%20Data%20Lists [ct]: https://canvas-datagrid.js.org/demo.html https://canvas-datagrid.js.org/demo.html [mi]: https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API https://developer.mozilla.org/en-US/docs/Web/API/Intersectio... [other]: https://js.devexpress.com/Demos/Widgetsgallery/Demo/DataGrid/InfiniteScrolling/Mvc/Light/ https://js.devexpress.com/Demos/Widgetsgallery/Demo/DataGrid... https://codepen.io/enigmatic/pen/JGvwEv https://codepen.io/enigmatic/pen/JGvwEv https://www.htmlelements.com/demos/grid/overview/ https://www.htmlelements.com/demos/grid/overview/

4 points by graderjs 5y ago

graderjs 5y ago

Approach ----------- Basically the approach is like that used by the Sproutcore Big Data List demo, being: 1. construct a moving foreground window onto a giant, scrollable background element (giant, as in arbitrarily wide and high, like 200000em) 2. move rows from the 'scroll wake region' to a pool when they move offscreen because of scroll, 3. move rows from the pool to the 'scroll bowshock region' when we need new rows because of scroll, 4. don't change the position of any existing rows in the window (instead allowing them to scroll naturally), but set new rows (and remove pool rows) using absolute positioning. Except as we need to scroll across both rows and columns do all of the above in 2 dimensions, for columns as well. And also the way I detect when we need to "add or remove" rows or columns is via Intersection Observers[mi]. Major issues overcome so far --------------------------------- - It's possible to scroll it now infinitely (and performantly) for the duration of the very large background element. - Major problem was I initially was listening for intersection of cells individually on each row, but if a row had already travelled off screen due to a vertical scroll, it would not be possible for its cells to 'intersect' with the viewport element (the window, in Intersection Observer API parlance, the "root"), and so no intersection for that cell would occur, and so cells in that row would not be updated. And if they were not updated, they would not be onscreen when the row was in future removed from the pool and positioned back on screen, and so its cells could no longer be updated, as there would be no cells within that scroll viewport to intersect anymore. The solution to this was to create a logical grouping of cells into columns, and handle any intersection observations for any cell in a column, at once, and apply them to all cells in a column. So the whole column would be updated. Before I fixed this, repeated scrolling (particularly in opposite diagonal directions) would erode the cells present, gradually decreasing their number, and throwing many out of their column alignments. - Also performance was another issue that's now mostly resolved tho I think I can do better. Basically too much work was happening on handling each Intersection record and we were not ignoring records that triggered the same work to be redone (such as intersection events from each cell in a column). These redundant records are now ignored (but mostly in a haphazard manner that could cause browser dependent bugs). Next steps ------------- - Add all of the outstanding features (scroll snap, row / col size dragging) from the table component. - Style it to make it look nice - Fix the outstanding border 'off by 1' bugs. - Sync up the 'view portal' with a 'data portal' (movable portal over a very large set of data), and print the correct data corresponding to the current scroll position in the table - Add the row and column headers using CSS sticky. This is the only comment I'll make in this thread, thanks for reading, and hit me up with questions at my email if you want: cris@dosycorp.com [bt]: https://i5ik.github.io/_____/7guis/ https://i5ik.github.io/_____/7guis/ (scroll down to the last 7GUIs component) [bp]: https://github.com/palantir/blueprint/issues/390 https://github.com/palantir/blueprint/issues/390 [ft]: https://fulmicoton.com/fattable/index2.html https://fulmicoton.com/fattable/index2.html [sc]: https://showcase.sproutcore.com/#demos/Big%20Data%20Lists https://showcase.sproutcore.com/#demos/Big%20Data%20Lists [ct]: https://canvas-datagrid.js.org/demo.html https://canvas-datagrid.js.org/demo.html [mi]: https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API https://developer.mozilla.org/en-US/docs/Web/API/Intersectio... [other]: https://js.devexpress.com/Demos/Widgetsgallery/Demo/DataGrid/InfiniteScrolling/Mvc/Light/ https://js.devexpress.com/Demos/Widgetsgallery/Demo/DataGrid... https://codepen.io/enigmatic/pen/JGvwEv https://codepen.io/enigmatic/pen/JGvwEv https://www.htmlelements.com/demos/grid/overview/ https://www.htmlelements.com/demos/grid/overview/