4 ms·

Show HN: 1 Million Rows

24 points by ankitchhatbar 1y ago

captn3m0 1y ago
Firefox/iOS - attempting to scroll the demo after zooming in a bit, just refreshes the page.
xnx 1y ago
"lightening fast"? Probably meant "lightning fast".
forshaper 1y ago
what can I do with these really big tables?
stanmancan 1y ago
Not 100% sure what I'm looking at here? Am I missing something or is it just table of data with an "infinite scroll" that loads 200 records at a time?
- deleted 1y ago
  [deleted]
- vlan121 1y ago
  Looks like this, like someone is rebranding pagination.
- leftnode 1y ago
  The company names look like Amazon merchants.
- ankitchhatbar 1y ago
  Loading more than a few thousand rows on a web page will make unusably slow. Especially when you add a lot more features to it. This is such that only what's seen or about to be seen is put on the page. The rest is kept ready on the server on local memory depending on what the user is doing. This allows for a scalable solution that allows you to view thousands of records and interact with them
  - mzajc 1y ago
    > This is such that only what's seen or about to be seen is put on the page This reads like it's a lazy loading library, but then the roadmap has features like > Assign Items to Users > Kanban View > Collaborative Editing Which read like something you'd have in a project management solution. How do these two concepts form a cohesive product and who is the target audience? I've seen my fair share of Jira and Trello hellscapes, but I doubt Kanban boards with more entries than memory can handle are very common.
  - runlaszlorun 1y ago
    I'm not so sure that loading "more than a few thousand rows" is as bad as it used to be. I did some quick benchmarks a couple years back. It's been a while but I want to say that Chrome was drawing 10k rows of a decent size each (10 columns of real world data and about 500b/row iirc) in about 300ms on a 10 year old MBP. I'll do a little benchmarking later today if I get a chance.
    - zdragnar 1y ago
      10 columns is pretty tame. Add in another 5 or 10 to introduce horizontal scrolling, add in some fixed position columns, html inputs like a checkbox in each row, apply styling or custom rendering to individual cells, and it can starts to show a little more quickly.
  - motoxpro 1y ago
    Here are some things that are already built that do what you are building: https://github.com/bvaughn/react-virtualized https://github.com/bvaughn/react-virtualized https://github.com/bvaughn/react-window https://github.com/bvaughn/react-window https://tanstack.com/virtual/latest https://tanstack.com/virtual/latest
  - stanmancan 1y ago
    I know why you're doing it; I'm just not sure what I'm looking at. I'm not sure if this is supposed to be a product, but right now it's literally just a paginated HTML table?
  - 1y ago
- burkaman 1y ago
  In Phase 3 there will be AI Features though.
_1tem 1y ago
oh dear, another Airtable clone. Also see Baserow.
notachatbot123 1y ago
Congrats on your vibe-coded website!
turblety 1y ago
Nice one! Coincidentally I worked on a large table renderer too this weekend: https://github.com/markwylde/react-massive-table https://github.com/markwylde/react-massive-table I noticed you didn't quite get to a million rows. For me, it cut off at 671088. Same thing happened when I built my one. I came across the same thing. In the end I just manual made the rows appear at their absolute position. Seemed to work well.
- ankitchhatbar 1y ago
  What browser are you using? Some browsers cut off early due to scroll limitations. I could get Firefox to about 300,000
wwdx 1y ago
Good idea but it flashes/blinks when I scroll which doesns't feel very smooth.
spicybright 1y ago
What is it though? Management tool is so vague.
whalesalad 1y ago
I really wish 1M rows was impressive.
zoba 1y ago
Is this... a database? A set of react components? An app? Should be much more immediately understandable.
bram2w 1y ago
When I started working on Baserow (this seems similar based on the roadmap), a couple of years ago, I thought it would be a big challenge to quickly render a million rows in the browser. Introducing a system that fetches a page of rows based on the scroll offset, and with a small debounce did the trick. We only had a couple of field types, and it was all incredibly fast The thing that make performance complicated for a no-code database is when you have 30 interconnected tables, some tables with 200 fields, containing many formulas or other computed fields like lookups or rollups. Updating a single cell, can result in thousands of other rows that must be updated across different tables. If there are 30 users making constant changes, locking PostgreSQL rows under the hood while the formulas are recalculated, and then a couple of n8n workflows making a many API requests to those tables, that's when things get interesting. Especially in combination with features like webhooks, real-time updates, 100+ filters, grouping, 26 field types, date dependencies, aggregations, importing/exporting whole databases. When implementing a new feature, I've heard users say that's not complicated because it's just adding a checkbox. Making to run it at scale and keeping things performant is what's making it complicated.
betageek 1y ago
By "Lightning fast management tool" you mean "virtualised table"?
jasonjmcghee 1y ago
I tried the live demo with synthetic data. To whom it may concern: I scrolled a bit with the scrollbar on iOS and the page immediately crashed.
- OsrsNeedsf2P 1y ago
  On Android the rows just didn't load
  - pmontra 1y ago
    On Android Firefox they do load. I'm on my tablet. It fills the first half of the page with rows. The bottom half is empty.
hn111 1y ago
Having the column widths jump around while scrolling and the absent page search doesn’t seem like ‘rock-solid reliability’ to me. It seems this is just a minimum implementation of a ‘virtual list’.
bestest 1y ago
This is terrible and not worthy of HN front. Terrible from the front-end side of implementation: - performs worse than your average arbitrary-amount-of-rows-that-won't-fit-on-the-screen library (it should perform the same no matter if its 1k, 1m or 1mm rows) - is seemingly buggy - is pointless on its own, because THIS demo is a client-side demo, and no one loads that much data on the client-side. Revisit this when this demo is performant AND data is loaded from the backend. Ignoring that, every front-end JS developer should explore these kinds of libs and also try to implement them themselves, because they're basically front-end 101.
mockingloris 1y ago
I'd trim the columns down to a few and show the user a filter with the displaying columns turned on and let them know that they can toggle more on. The column text length too should be trimmed to a uniform max-length except when clicked on. You could make it pup out on the page with CSS. A better color scheme too won't hurt. Had the same idea when I saw https://github.com/rowyio/rowy https://github.com/rowyio/rowy. Stumbled on an idea while reading a HN entry a few days back and now I will merge them into a niche product idea. └── Dey well
oulipo 1y ago
Nobody wants to "view" 1M row... people want to have analytics about what subset of the rows they should look at for a given task
merelysounds 1y ago
Possible bug report: on Safari mobile, when I grab the scroll bar and move it down a bit, the website reloads; if I do it again, I get an error message (“a problem repeatedly occurred…”).
timacles 1y ago
One of the rare times HN can come together and agree on something. Which is: that this is a loose weekend project thrown together quickly that doesnt solve any problem in particular
NoboruWataya 1y ago
I had hoped it might be like one of those "1 million pixels" websites, where anyone can run queries to update the rows.
pdyc 1y ago
i dabbled into this last year and wrote about the challenge here https://newbeelearn.com/blog/million-rows-csv-debug-story/ https://newbeelearn.com/blog/million-rows-csv-debug-story/
djfobbz 1y ago
You basically reinvented something Elixir Streams already nails out of the box. Streams in Elixir are lazy, chunked, and backpressure-friendly, so you can process any size dataset without loading it all into memory...whether it's a million or a trillion rows. The trick is you never try to render them all in the browser (that's where virtualization comes in). So yeah...neat work, but battle-tested versions of this have been around for a while.
- dustingetz 1y ago
  above 50k and the UI needs to change because 1) you can’t count the collection, and 2) the scrollheight becomes too unwieldy to use the mouse, slight adjustments to the handle will skip forward many pages. And if you try at some point you exceed the pixel height browser scroll bars can support, needing a custom non-native scroll bar. anyway well before that point, roughly at 50k records you need to switch to “search” UX (much smaller result sets) because there is no way to actually access page 99910 of your million record collection. tldr: “show me the demo”
huqedato 1y ago
What the heck is this app doing?
ychnlt 1y ago
[dead]
oksurewhynot 1y ago
I can only see 670k rows
arjonagelhout 1y ago
I get “A problem repeatedly occurred on <url>” on iOS Safari.
mrcsharp 1y ago
Doesn't even work. On a Galaxy Fold 6 in the unfolded position using Firefox and quickly scrolling past row #150 broke the rendering completely. So much for fast performance as advertised.

24 points by ankitchhatbar 1y ago

captn3m0 1y ago

Firefox/iOS - attempting to scroll the demo after zooming in a bit, just refreshes the page.

xnx 1y ago

"lightening fast"? Probably meant "lightning fast".

forshaper 1y ago

what can I do with these really big tables?

stanmancan 1y ago

Not 100% sure what I'm looking at here? Am I missing something or is it just table of data with an "infinite scroll" that loads 200 records at a time?

deleted 1y ago
[deleted]
vlan121 1y ago
Looks like this, like someone is rebranding pagination.
leftnode 1y ago
The company names look like Amazon merchants.
ankitchhatbar 1y ago
Loading more than a few thousand rows on a web page will make unusably slow. Especially when you add a lot more features to it. This is such that only what's seen or about to be seen is put on the page. The rest is kept ready on the server on local memory depending on what the user is doing. This allows for a scalable solution that allows you to view thousands of records and interact with them
- mzajc 1y ago
  > This is such that only what's seen or about to be seen is put on the page This reads like it's a lazy loading library, but then the roadmap has features like > Assign Items to Users > Kanban View > Collaborative Editing Which read like something you'd have in a project management solution. How do these two concepts form a cohesive product and who is the target audience? I've seen my fair share of Jira and Trello hellscapes, but I doubt Kanban boards with more entries than memory can handle are very common.
- runlaszlorun 1y ago
  I'm not so sure that loading "more than a few thousand rows" is as bad as it used to be. I did some quick benchmarks a couple years back. It's been a while but I want to say that Chrome was drawing 10k rows of a decent size each (10 columns of real world data and about 500b/row iirc) in about 300ms on a 10 year old MBP. I'll do a little benchmarking later today if I get a chance.
  - zdragnar 1y ago
    10 columns is pretty tame. Add in another 5 or 10 to introduce horizontal scrolling, add in some fixed position columns, html inputs like a checkbox in each row, apply styling or custom rendering to individual cells, and it can starts to show a little more quickly.
- motoxpro 1y ago
  Here are some things that are already built that do what you are building: https://github.com/bvaughn/react-virtualized https://github.com/bvaughn/react-virtualized https://github.com/bvaughn/react-window https://github.com/bvaughn/react-window https://tanstack.com/virtual/latest https://tanstack.com/virtual/latest
- stanmancan 1y ago
  I know why you're doing it; I'm just not sure what I'm looking at. I'm not sure if this is supposed to be a product, but right now it's literally just a paginated HTML table?
- 1y ago
burkaman 1y ago
In Phase 3 there will be AI Features though.

_1tem 1y ago

oh dear, another Airtable clone. Also see Baserow.

notachatbot123 1y ago

Congrats on your vibe-coded website!

turblety 1y ago

Nice one! Coincidentally I worked on a large table renderer too this weekend: https://github.com/markwylde/react-massive-table https://github.com/markwylde/react-massive-table I noticed you didn't quite get to a million rows. For me, it cut off at 671088. Same thing happened when I built my one. I came across the same thing. In the end I just manual made the rows appear at their absolute position. Seemed to work well.

ankitchhatbar 1y ago
What browser are you using? Some browsers cut off early due to scroll limitations. I could get Firefox to about 300,000

wwdx 1y ago

Good idea but it flashes/blinks when I scroll which doesns't feel very smooth.

spicybright 1y ago

What is it though? Management tool is so vague.

whalesalad 1y ago

I really wish 1M rows was impressive.

zoba 1y ago

Is this... a database? A set of react components? An app? Should be much more immediately understandable.

bram2w 1y ago

When I started working on Baserow (this seems similar based on the roadmap), a couple of years ago, I thought it would be a big challenge to quickly render a million rows in the browser. Introducing a system that fetches a page of rows based on the scroll offset, and with a small debounce did the trick. We only had a couple of field types, and it was all incredibly fast The thing that make performance complicated for a no-code database is when you have 30 interconnected tables, some tables with 200 fields, containing many formulas or other computed fields like lookups or rollups. Updating a single cell, can result in thousands of other rows that must be updated across different tables. If there are 30 users making constant changes, locking PostgreSQL rows under the hood while the formulas are recalculated, and then a couple of n8n workflows making a many API requests to those tables, that's when things get interesting. Especially in combination with features like webhooks, real-time updates, 100+ filters, grouping, 26 field types, date dependencies, aggregations, importing/exporting whole databases. When implementing a new feature, I've heard users say that's not complicated because it's just adding a checkbox. Making to run it at scale and keeping things performant is what's making it complicated.

betageek 1y ago

By "Lightning fast management tool" you mean "virtualised table"?

jasonjmcghee 1y ago

I tried the live demo with synthetic data. To whom it may concern: I scrolled a bit with the scrollbar on iOS and the page immediately crashed.

OsrsNeedsf2P 1y ago
On Android the rows just didn't load
- pmontra 1y ago
  On Android Firefox they do load. I'm on my tablet. It fills the first half of the page with rows. The bottom half is empty.

hn111 1y ago

Having the column widths jump around while scrolling and the absent page search doesn’t seem like ‘rock-solid reliability’ to me. It seems this is just a minimum implementation of a ‘virtual list’.

bestest 1y ago

This is terrible and not worthy of HN front. Terrible from the front-end side of implementation: - performs worse than your average arbitrary-amount-of-rows-that-won't-fit-on-the-screen library (it should perform the same no matter if its 1k, 1m or 1mm rows) - is seemingly buggy - is pointless on its own, because THIS demo is a client-side demo, and no one loads that much data on the client-side. Revisit this when this demo is performant AND data is loaded from the backend. Ignoring that, every front-end JS developer should explore these kinds of libs and also try to implement them themselves, because they're basically front-end 101.

mockingloris 1y ago

I'd trim the columns down to a few and show the user a filter with the displaying columns turned on and let them know that they can toggle more on. The column text length too should be trimmed to a uniform max-length except when clicked on. You could make it pup out on the page with CSS. A better color scheme too won't hurt. Had the same idea when I saw https://github.com/rowyio/rowy https://github.com/rowyio/rowy. Stumbled on an idea while reading a HN entry a few days back and now I will merge them into a niche product idea. └── Dey well

oulipo 1y ago

Nobody wants to "view" 1M row... people want to have analytics about what subset of the rows they should look at for a given task

merelysounds 1y ago

Possible bug report: on Safari mobile, when I grab the scroll bar and move it down a bit, the website reloads; if I do it again, I get an error message (“a problem repeatedly occurred…”).

timacles 1y ago

One of the rare times HN can come together and agree on something. Which is: that this is a loose weekend project thrown together quickly that doesnt solve any problem in particular

NoboruWataya 1y ago

I had hoped it might be like one of those "1 million pixels" websites, where anyone can run queries to update the rows.

pdyc 1y ago

i dabbled into this last year and wrote about the challenge here https://newbeelearn.com/blog/million-rows-csv-debug-story/ https://newbeelearn.com/blog/million-rows-csv-debug-story/

djfobbz 1y ago

You basically reinvented something Elixir Streams already nails out of the box. Streams in Elixir are lazy, chunked, and backpressure-friendly, so you can process any size dataset without loading it all into memory...whether it's a million or a trillion rows. The trick is you never try to render them all in the browser (that's where virtualization comes in). So yeah...neat work, but battle-tested versions of this have been around for a while.

dustingetz 1y ago
above 50k and the UI needs to change because 1) you can’t count the collection, and 2) the scrollheight becomes too unwieldy to use the mouse, slight adjustments to the handle will skip forward many pages. And if you try at some point you exceed the pixel height browser scroll bars can support, needing a custom non-native scroll bar. anyway well before that point, roughly at 50k records you need to switch to “search” UX (much smaller result sets) because there is no way to actually access page 99910 of your million record collection. tldr: “show me the demo”

huqedato 1y ago

What the heck is this app doing?

ychnlt 1y ago

[dead]

oksurewhynot 1y ago

I can only see 670k rows

arjonagelhout 1y ago

I get “A problem repeatedly occurred on <url>” on iOS Safari.

mrcsharp 1y ago

Doesn't even work. On a Galaxy Fold 6 in the unfolded position using Firefox and quickly scrolling past row #150 broke the rendering completely. So much for fast performance as advertised.