5 ms·

Congrats! Been running a startup off only liveview for about a year as a solo dev and it's been wonderful. Appreciate the work you all do. Selfish demo idea: B

Congrats! Been running a startup off only liveview for about a year as a solo dev and it's been wonderful. Appreciate the work you all do. Selfish demo idea: Bi-directional cursor based infinite pagination with largish datasets with state managed in the url and streaming updates that change the order of the results. Like some kind of soft realtime leaderboard. With long render times (morphdom bench on large sets isn't great as far as I can tell) it's hard to escape the jank. Particularly on slow connections.

andrewmutz 2y ago
How much state is it safe to store in the URL these days?
- ljoshua 2y ago
  Still not much, realistically 4096 bytes or less. Browsers aren’t as much the issue as they’ve been in the past, but I’ve hit snags with proxies, old servers, etc.
  - ethbr1 2y ago
    How does pagination in urls work nowadays? You'd need ~3 bytes to index a reasonable number of pages naively, no? But curious what current art is re: performance optimizations between frontend and backend. Or is it simply page indices?
    - out_of_protocol 2y ago
      If you do limit/offset on database side, page number is enough. Though this doesn't work well for bigger page numbers. There's other ways to do pagination, e.g. with "cursors", where cursor is simply id of last record on previous page. SQL query is very efficient, but jumping to page X is impossible. In this scenario storing cursors for past pages is needed
      - cess11 2y ago
        There are libraries for Ecto that help with this. https://github.com/duffelhq/paginator https://github.com/duffelhq/paginator
        sph 2y ago
        Careful that this library last I used it (2020 or so) used a particularly insecure encoding of the cursor that basically allows remote execution. Not sure if they ever addressed it. Here's the fork I created at the time to work around some of these issues: https://github.com/1player/paginator https://github.com/1player/paginator
        cess11 2y ago
        Thanks for pointing that out.
        ethbr1 2y ago
        Thanks to thread for the informative responses! (And the useful README on your fork/the upstream) I try to assume someone's thought of better than the best I could, or at least learned the hard way what edge cases need to be handled.
        gregors 2y ago
        This looks like it was fixed * https://github.com/duffelhq/paginator/blob/main/lib/paginator/cursor.ex#L9 https://github.com/duffelhq/paginator/blob/main/lib/paginato... * https://github.com/duffelhq/paginator/blob/main/test/paginator_test.exs#L830 https://github.com/duffelhq/paginator/blob/main/test/paginat...
chrismccord 2y ago
Will keep this in mind! My ElixirConfEU keynote just used limit/offset pagination, but shows how easy it is to do bidirectional infinite pagination. Streams allows you to keep a virtualized infinite list in the DOM without keeping the giant collection around on either client or server, example: https://www.youtube.com/watch?v=FADQAnq0RpA&t=2322s https://www.youtube.com/watch?v=FADQAnq0RpA&t=2322s This demo doesn't update the URL to keep state (but push_patch would make that trivial). I'll think about more complex examples for cursor pagination, but the solution to massive collections (thousands of entries) is going to be stream + limit
- goosejuice 2y ago
  Thanks! That is the approach I took, though I believe reordering rows requires a reset. So in that demo, I'm curious how smooth the scroll would be when adding latency via the debug tools in the browser console. Even with some padding and fiddling with limi ts I've had trouble getting it to be an enjoyable experience. Just bouncy if that makes sense. More frustrating on mobile than desktop. With localhost latency it's great. I'm thinking it may just need a hook to preload rows or something of that nature to provide space for the render. Though my CSS could very well be wrong.
- kuatroka 2y ago
  flop_phoenix is great. If it lacks anything maybe it'd be great to just enhance it instead of building anything new. The author also has cursor based functionality
davydog187 2y ago
I highly recommend checking out [Flop](https://hexdocs.pm/flop/readme.html https://hexdocs.pm/flop/readme.html) and its Phoenix.Component [Flop Phoenix](https://hexdocs.pm/flop_phoenix/readme.html https://hexdocs.pm/flop_phoenix/readme.html). Its extremely easy to customize and build a reusable paginated table component
- kuatroka 2y ago
  Second that The flop is a good idea. Love the flop

andrewmutz 2y ago

How much state is it safe to store in the URL these days?

ljoshua 2y ago
Still not much, realistically 4096 bytes or less. Browsers aren’t as much the issue as they’ve been in the past, but I’ve hit snags with proxies, old servers, etc.
- ethbr1 2y ago
  How does pagination in urls work nowadays? You'd need ~3 bytes to index a reasonable number of pages naively, no? But curious what current art is re: performance optimizations between frontend and backend. Or is it simply page indices?
  - out_of_protocol 2y ago
    If you do limit/offset on database side, page number is enough. Though this doesn't work well for bigger page numbers. There's other ways to do pagination, e.g. with "cursors", where cursor is simply id of last record on previous page. SQL query is very efficient, but jumping to page X is impossible. In this scenario storing cursors for past pages is needed
    - cess11 2y ago
      There are libraries for Ecto that help with this. https://github.com/duffelhq/paginator https://github.com/duffelhq/paginator
      - sph 2y ago
        Careful that this library last I used it (2020 or so) used a particularly insecure encoding of the cursor that basically allows remote execution. Not sure if they ever addressed it. Here's the fork I created at the time to work around some of these issues: https://github.com/1player/paginator https://github.com/1player/paginator
        cess11 2y ago
        Thanks for pointing that out.
        ethbr1 2y ago
        Thanks to thread for the informative responses! (And the useful README on your fork/the upstream) I try to assume someone's thought of better than the best I could, or at least learned the hard way what edge cases need to be handled.
        gregors 2y ago
        This looks like it was fixed * https://github.com/duffelhq/paginator/blob/main/lib/paginator/cursor.ex#L9 https://github.com/duffelhq/paginator/blob/main/lib/paginato... * https://github.com/duffelhq/paginator/blob/main/test/paginator_test.exs#L830 https://github.com/duffelhq/paginator/blob/main/test/paginat...

chrismccord 2y ago

Will keep this in mind! My ElixirConfEU keynote just used limit/offset pagination, but shows how easy it is to do bidirectional infinite pagination. Streams allows you to keep a virtualized infinite list in the DOM without keeping the giant collection around on either client or server, example: https://www.youtube.com/watch?v=FADQAnq0RpA&t=2322s https://www.youtube.com/watch?v=FADQAnq0RpA&t=2322s This demo doesn't update the URL to keep state (but push_patch would make that trivial). I'll think about more complex examples for cursor pagination, but the solution to massive collections (thousands of entries) is going to be stream + limit

goosejuice 2y ago
Thanks! That is the approach I took, though I believe reordering rows requires a reset. So in that demo, I'm curious how smooth the scroll would be when adding latency via the debug tools in the browser console. Even with some padding and fiddling with limi ts I've had trouble getting it to be an enjoyable experience. Just bouncy if that makes sense. More frustrating on mobile than desktop. With localhost latency it's great. I'm thinking it may just need a hook to preload rows or something of that nature to provide space for the render. Though my CSS could very well be wrong.
kuatroka 2y ago
flop_phoenix is great. If it lacks anything maybe it'd be great to just enhance it instead of building anything new. The author also has cursor based functionality

davydog187 2y ago

I highly recommend checking out [Flop](https://hexdocs.pm/flop/readme.html https://hexdocs.pm/flop/readme.html) and its Phoenix.Component [Flop Phoenix](https://hexdocs.pm/flop_phoenix/readme.html https://hexdocs.pm/flop_phoenix/readme.html). Its extremely easy to customize and build a reusable paginated table component

kuatroka 2y ago
Second that The flop is a good idea. Love the flop