3 ms·

It's been common knowledge since... Forever really. I doubt there was a specific person. It just spread across industries due to the known limitations in perfor

It's been common knowledge since... Forever really. I doubt there was a specific person. It just spread across industries due to the known limitations in performance. Excel was well known to not be able to handle large datasets back in 2006 when I started in data analytics field. I'd guess Excels limitations go way back to when it was originally released. Excel is very good btw. A marvel in many ways. It just becomes painful with big data around the 100-300k mark.

SunlightEdge 5y ago
The million row mark for excel while widely known is never taken seriously by anyone who actually uses excel often. It's performances collapses around the 100-300k mark.
- tyingq 5y ago
  I would guess it depends on the columns as well. I just opened a million-row, 10 column spreadsheet and added a few formulas like =average(a1:a999999) and it's responsive...no hangs or other issues. "Save as" ran for a 5 seconds or so. On an older i5 laptop with 8GB.
  - SunlightEdge 5y ago
    It's difficult to comment without seeing your workbook. But broadly if you're developing any kind of useful analytical model you will find the performance fall off a cliff very quickly if using 1 million rows. e.g. waiting around 25 minutes to save the file (or being corrupted), the software being suddenly very slow and weird. And this includes tricks like turning off automatic calculations, hard copy pasting in the results of calculations (rather than keeping all cells as formulas), only running calculations on specific cells etc. There are things you can do to try and manage big data in excel. But it becomes a chore and ultimately a big bloated monster. And why bother with the pain when you can use Python (e.g pandas) or R or SAS. Power excel / an SQL backend (with VBA) is also a common solution. Or was. I mostly work in Python/R these days.
  - CryptoBanker 5y ago
    Try adding a recursive formula or two in there and see what happens
- LegitShady 5y ago
  My guess is that even if it is factorable it would be made into its own separate product or attached to another program like power query or their bi offerings.

SunlightEdge 5y ago

The million row mark for excel while widely known is never taken seriously by anyone who actually uses excel often. It's performances collapses around the 100-300k mark.

tyingq 5y ago
I would guess it depends on the columns as well. I just opened a million-row, 10 column spreadsheet and added a few formulas like =average(a1:a999999) and it's responsive...no hangs or other issues. "Save as" ran for a 5 seconds or so. On an older i5 laptop with 8GB.
- SunlightEdge 5y ago
  It's difficult to comment without seeing your workbook. But broadly if you're developing any kind of useful analytical model you will find the performance fall off a cliff very quickly if using 1 million rows. e.g. waiting around 25 minutes to save the file (or being corrupted), the software being suddenly very slow and weird. And this includes tricks like turning off automatic calculations, hard copy pasting in the results of calculations (rather than keeping all cells as formulas), only running calculations on specific cells etc. There are things you can do to try and manage big data in excel. But it becomes a chore and ultimately a big bloated monster. And why bother with the pain when you can use Python (e.g pandas) or R or SAS. Power excel / an SQL backend (with VBA) is also a common solution. Or was. I mostly work in Python/R these days.
- CryptoBanker 5y ago
  Try adding a recursive formula or two in there and see what happens
LegitShady 5y ago
My guess is that even if it is factorable it would be made into its own separate product or attached to another program like power query or their bi offerings.