5 ms·

can it handle a million rows? 10M? believe it or not this is my biggest problem with Sheets, and while Excel does better it's still capped at 1M

idiotsecant 4y ago
Once you hit a million rows wouldn't it be easier to just make it a proper database? Maybe there are some use cases where this is not true, but I can't imagine what they are.
tomxor 4y ago
Since manually scrolling through 10M rows is not likely... at that point isn't it better to just use an SQL DB rather than a spreadsheet? SQlite could be enough, and it's comparable to a spreadsheet in that it is well suited to a single-file single-user local-DB scenario, yet can still handle large quantities of data.
- slotrans 4y ago
  No one loves SQL more than me but the effort jump between "paste into a spreadsheet" and "load into SQLite" is at least 100x Computers are fast, I have gigabytes of RAM, I should be able to look at 10M rows
dosinga 4y ago
Currently it cannot. However we find that many millions of row use cases don't actually need those millions of rows to be in the spreadsheet - you just want to be able to process that amount of data. So connect to a SQL database from the python side of things, read in 10 million rows, aggregate them into something succinct and write that to the spreadsheet is to sort of thing we're seeing.
- slotrans 4y ago
  I (mainly) don't want to aggregate them, I want to look at them. Don't forget "Filter". It's very easy in a spreadsheet to take a huge number of rows and cut them down -- interactively! -- to a subset that's of interest. Based on what I see I may want to scribble formulas in the margins. I love love love SQL but it is not a spreadsheet. SQL is great at aggregating down a column but very awkward at aggregating across a row. My SQL prompt doesn't have scatter plots built-in. I can't arbitrarily color rows/columns/cells.

idiotsecant 4y ago

Once you hit a million rows wouldn't it be easier to just make it a proper database? Maybe there are some use cases where this is not true, but I can't imagine what they are.

tomxor 4y ago

Since manually scrolling through 10M rows is not likely... at that point isn't it better to just use an SQL DB rather than a spreadsheet? SQlite could be enough, and it's comparable to a spreadsheet in that it is well suited to a single-file single-user local-DB scenario, yet can still handle large quantities of data.

slotrans 4y ago
No one loves SQL more than me but the effort jump between "paste into a spreadsheet" and "load into SQLite" is at least 100x Computers are fast, I have gigabytes of RAM, I should be able to look at 10M rows

dosinga 4y ago

Currently it cannot. However we find that many millions of row use cases don't actually need those millions of rows to be in the spreadsheet - you just want to be able to process that amount of data. So connect to a SQL database from the python side of things, read in 10 million rows, aggregate them into something succinct and write that to the spreadsheet is to sort of thing we're seeing.

slotrans 4y ago
I (mainly) don't want to aggregate them, I want to look at them. Don't forget "Filter". It's very easy in a spreadsheet to take a huge number of rows and cut them down -- interactively! -- to a subset that's of interest. Based on what I see I may want to scribble formulas in the margins. I love love love SQL but it is not a spreadsheet. SQL is great at aggregating down a column but very awkward at aggregating across a row. My SQL prompt doesn't have scatter plots built-in. I can't arbitrarily color rows/columns/cells.