7 ms·

How we make 5000 row tables fast in Streak

139 points by OmarIsmail 14y ago

ax 14y ago
Is there a way to view one of these tables without having to log in?
- OmarIsmail 14y ago
  No, but I'll make a video right now. edit: here's the video http://www.youtube.com/watch?v=o1O7HkBj74c http://www.youtube.com/watch?v=o1O7HkBj74c
deleted 14y ago
[deleted]
- OmarIsmail 14y ago
  This isn't a database table, this is a rendered HTML table.
- johns 14y ago
  The first sentence of the article clears up the title. It's not talking about DB tables, it's talking about HTML tables.
neves 14y ago
Open source it! Please.
pixelbath 14y ago
FYI, the right-side menu makes the scrollbar impossible to grab at the top of the page. However, I can still scroll with the wheel or bottom of the scrollbar.
gklitt 14y ago
Ironically, when I move the scrollbar on this blog post (in Chrome), the scrolling is noticeably not "buttery", probably because of the JS-heavy interface...
- morsch 14y ago
  Something is very strange with scrolling using a mouse wheel, too. It seems to scroll about a third to half of the distance it usual does for each "tick" of the mouse wheel (Ubuntu, Firefox). I guess it doesn't recognize the OS setting.
  - alttab 14y ago
    The whole page lags like a bitch on ipad3. Irony, for sure.
danielweber 14y ago
I love articles about web design that are completely blank if you don't have JavaScript activated.
- benihana 14y ago
  I love how there are people who have to point out that they have JavaScript disabled in 2012.
  - Dylan16807 14y ago
    How about a phone browser where it loads but freaks out and makes a glitchy mess that can't be scrolled. And on my desktop Javascript uses most of the browser's memory. No thanks.
  - Kliment 14y ago
    I use requestpolicy with fairly strict settings, and I have since grown to hate websites that have the content and styles loaded from two or more different domains. The vastly faster load times and vastly less junk on the page definitely makes it worth it.
    - rimantas 14y ago
      Funny, because loading assets from different domain actually can help improve page performance: for one it allows to work around maximum-parallel-request-per-domain restriction, for other you may serve resources from cookie-less domain shaving a few more milliseconds.
      - Kliment 14y ago
        It absolutely can. Unfortunately the stuff usually loaded is a crapload of antisocial share buttons and tracking crap that more that destroy any gain that the techniques you mention bring.
- dclowd9901 14y ago
  Probably feels that his article is for peers, not gotchas.
dunham 14y ago
We do something similar to render large tables, but we use divs to layout our data. We establish the height of the scrollable area with a large div, and then absolutely position the rows within it. We have just enough real physical rows to fill the screen, plus a buffer of a few rows. The virtual rows are assigned to physical rows via modular arithmetic, so only a couple of rows change their data on scroll. (And are repositioned accordingly.) This technique (particularly calculating the positions of the rows) requires fixed height rows. (There is a solution for variable height rows, but it's a bit more complicated.) I originally got the idea by looking at google books years ago. Here is a quick and dirty example as a self-contained HTML file: https://gist.github.com/3086578 (It constructs a list of 10,000 objects with title/description fields, and then displays that data in a table.)
- kellysutton 14y ago
  This sounds pretty similar to how UITableView on the iPhone keeps its performance up. If I recall correctly, iOS actually recycles a small number of discrete cells for use in the table and shuffles in the necessary data.
  - georgemcbay 14y ago
    I'm always surprised (though it happens fairly frequently) when I run into a UI toolkit with any kind of fixed-size item list view that doesn't recycle the child widgets and handle the actual item list as a sliding-window data provider. Not to take anything away from any of the mentioned projects, but this pattern is not new or novel.
    - ralfn 14y ago
      No, its not. Its even more relevant for Javascript, because creating a dom node is way more expensive, thsn just altering it.
- OmarIsmail 14y ago
  This is awesome. I like your technique a lot better actually!
kellysutton 14y ago
I recently jumped through a lot of similar hoops in making an Activity Feed for my company. We use a few finesses to cut down on the number of repaints the browser likes to do when dealing with huge tables like these. More on that here: http://layervault.tumblr.com/post/26117253199/making-a-new-type-of-activity-feed http://layervault.tumblr.com/post/26117253199/making-a-new-t...
bazzargh 14y ago
That slow multisort doesn't do multisort at all, generally. If it did, it would be relying on (1) the sort algorithm being stable and (2) the properties being in the reverse order from what you used in the second multisort algorithm. Eg with properties a,b it sorts on a, then sorts the list again based on b. The list is now in b-order (not a-order) and b-equal blocks are only in a-order if the sort is stable. Meanwhile, the 'fast' algortithm is in a-order with b-ordered a-equal blocks.
zenocon 14y ago
GWT had a PagingScrollTable - I used it 3-4 years ago. It has been a while since I built a GWT project, so I'm sure it has improved dramatically, but I was able to get lightning fast tables using PagingScrollTable. Here's a demo http://zenoconsulting.biz/com.example.Application/Application.html http://zenoconsulting.biz/com.example.Application/Applicatio... -- try 5,000 rows. It is pretty speedy.
- tterrace 14y ago
  One of their requirements was no paging - everything on one page.
nodata 14y ago
5000 rows or 5000 columns? 5000 rows isn't a lot. Edit: the article says 5000 rows, 10 columns. Hm.
jawinn 14y ago
It's great to read some of these advanced optimization techniques, especially for us web devs that aren't working on these larger projects. I'm interested to know more about how and why you are using this "Levinshtein distance" (Fast Levinshtein) algorithm. I looked at the Wikipedia and the explanation makes sense; it's just not sinking in as to where I might think about making use of it in my own code.
robocat 14y ago
Some other gotchas to beware of: 1. position:relative; or absolute on the ccell or within a cell causes same slowdown as overflow:hidden. 2. On iOS, overflow:hidden and position:relative on cells can cause serious slowdowns later when an absolute div later dynamically over document. 3. Beware of tables in IE8 - changing className of div in cell or other changes can cause complete table reflow calc - found out using commercial profiler that gives information about reflow/redraw times.

139 points by OmarIsmail 14y ago

ax 14y ago

Is there a way to view one of these tables without having to log in?

OmarIsmail 14y ago
No, but I'll make a video right now. edit: here's the video http://www.youtube.com/watch?v=o1O7HkBj74c http://www.youtube.com/watch?v=o1O7HkBj74c

deleted 14y ago

[deleted]

OmarIsmail 14y ago
This isn't a database table, this is a rendered HTML table.
johns 14y ago
The first sentence of the article clears up the title. It's not talking about DB tables, it's talking about HTML tables.

neves 14y ago

Open source it! Please.

pixelbath 14y ago

FYI, the right-side menu makes the scrollbar impossible to grab at the top of the page. However, I can still scroll with the wheel or bottom of the scrollbar.

gklitt 14y ago

Ironically, when I move the scrollbar on this blog post (in Chrome), the scrolling is noticeably not "buttery", probably because of the JS-heavy interface...

morsch 14y ago
Something is very strange with scrolling using a mouse wheel, too. It seems to scroll about a third to half of the distance it usual does for each "tick" of the mouse wheel (Ubuntu, Firefox). I guess it doesn't recognize the OS setting.
- alttab 14y ago
  The whole page lags like a bitch on ipad3. Irony, for sure.

danielweber 14y ago

I love articles about web design that are completely blank if you don't have JavaScript activated.

benihana 14y ago
I love how there are people who have to point out that they have JavaScript disabled in 2012.
- Dylan16807 14y ago
  How about a phone browser where it loads but freaks out and makes a glitchy mess that can't be scrolled. And on my desktop Javascript uses most of the browser's memory. No thanks.
- Kliment 14y ago
  I use requestpolicy with fairly strict settings, and I have since grown to hate websites that have the content and styles loaded from two or more different domains. The vastly faster load times and vastly less junk on the page definitely makes it worth it.
  - rimantas 14y ago
    Funny, because loading assets from different domain actually can help improve page performance: for one it allows to work around maximum-parallel-request-per-domain restriction, for other you may serve resources from cookie-less domain shaving a few more milliseconds.
    - Kliment 14y ago
      It absolutely can. Unfortunately the stuff usually loaded is a crapload of antisocial share buttons and tracking crap that more that destroy any gain that the techniques you mention bring.
dclowd9901 14y ago
Probably feels that his article is for peers, not gotchas.

dunham 14y ago

We do something similar to render large tables, but we use divs to layout our data. We establish the height of the scrollable area with a large div, and then absolutely position the rows within it. We have just enough real physical rows to fill the screen, plus a buffer of a few rows. The virtual rows are assigned to physical rows via modular arithmetic, so only a couple of rows change their data on scroll. (And are repositioned accordingly.) This technique (particularly calculating the positions of the rows) requires fixed height rows. (There is a solution for variable height rows, but it's a bit more complicated.) I originally got the idea by looking at google books years ago. Here is a quick and dirty example as a self-contained HTML file: https://gist.github.com/3086578 (It constructs a list of 10,000 objects with title/description fields, and then displays that data in a table.)

kellysutton 14y ago
This sounds pretty similar to how UITableView on the iPhone keeps its performance up. If I recall correctly, iOS actually recycles a small number of discrete cells for use in the table and shuffles in the necessary data.
- georgemcbay 14y ago
  I'm always surprised (though it happens fairly frequently) when I run into a UI toolkit with any kind of fixed-size item list view that doesn't recycle the child widgets and handle the actual item list as a sliding-window data provider. Not to take anything away from any of the mentioned projects, but this pattern is not new or novel.
  - ralfn 14y ago
    No, its not. Its even more relevant for Javascript, because creating a dom node is way more expensive, thsn just altering it.
OmarIsmail 14y ago
This is awesome. I like your technique a lot better actually!

kellysutton 14y ago

I recently jumped through a lot of similar hoops in making an Activity Feed for my company. We use a few finesses to cut down on the number of repaints the browser likes to do when dealing with huge tables like these. More on that here: http://layervault.tumblr.com/post/26117253199/making-a-new-type-of-activity-feed http://layervault.tumblr.com/post/26117253199/making-a-new-t...

bazzargh 14y ago

That slow multisort doesn't do multisort at all, generally. If it did, it would be relying on (1) the sort algorithm being stable and (2) the properties being in the reverse order from what you used in the second multisort algorithm. Eg with properties a,b it sorts on a, then sorts the list again based on b. The list is now in b-order (not a-order) and b-equal blocks are only in a-order if the sort is stable. Meanwhile, the 'fast' algortithm is in a-order with b-ordered a-equal blocks.

zenocon 14y ago

GWT had a PagingScrollTable - I used it 3-4 years ago. It has been a while since I built a GWT project, so I'm sure it has improved dramatically, but I was able to get lightning fast tables using PagingScrollTable. Here's a demo http://zenoconsulting.biz/com.example.Application/Application.html http://zenoconsulting.biz/com.example.Application/Applicatio... -- try 5,000 rows. It is pretty speedy.

tterrace 14y ago
One of their requirements was no paging - everything on one page.

nodata 14y ago

5000 rows or 5000 columns? 5000 rows isn't a lot. Edit: the article says 5000 rows, 10 columns. Hm.

jawinn 14y ago

It's great to read some of these advanced optimization techniques, especially for us web devs that aren't working on these larger projects. I'm interested to know more about how and why you are using this "Levinshtein distance" (Fast Levinshtein) algorithm. I looked at the Wikipedia and the explanation makes sense; it's just not sinking in as to where I might think about making use of it in my own code.

robocat 14y ago

Some other gotchas to beware of: 1. position:relative; or absolute on the ccell or within a cell causes same slowdown as overflow:hidden. 2. On iOS, overflow:hidden and position:relative on cells can cause serious slowdowns later when an absolute div later dynamically over document. 3. Beware of tables in IE8 - changing className of div in cell or other changes can cause complete table reflow calc - found out using commercial profiler that gives information about reflow/redraw times.