4 ms·

Does anyone know why scientists aren’t just using Excel? Nothing against Python/Julia but Excel seems like it would be my first choice (paired with some CSV fil

Does anyone know why scientists aren’t just using Excel? Nothing against Python/Julia but Excel seems like it would be my first choice (paired with some CSV files).

ur-whale 6y ago
Excel is far too slow for large scale datasets.
- jimbob45 6y ago
  Is that true? Python is pretty slow too. I can’t imagine many scientists would really mind leaving their work running overnight.
  - proverbialbunny 6y ago
    It's true. Fun and relevant fact: The job title data scientist was invented at LinkedIn when they saw data analysts that needed to know R or Python skills. These skills seemed different enough from the traditional Excel and SAS data analyst jobs at the time to warrant a new job title. So did Excel not cut it, or was it a preference to use Python? Back then Excel supported up to 65,535 rows, and would typically crash with over 8,000 rows. I worked in that era in Excel. One model once was split across three spreadsheets. Only one spreadsheet could be open at a time, they would take 20 minutes each to load, and there was about a 50% chance they would crash while loading. So what do you do if you need over 65,000 instances of labeled data? For a neural network it's nice to have a million instances, yes a million. R and Python have in them what is called a dataframe. It's a spreadsheet, but in another programming language. We tend to load our data into those, which is just like Excel, but without the hardware limitations. So in many ways, today it's just like it once was, but we get to choose which programming language to use while working in a spreadsheet, and let's be fair, the Excel programming language isn't exactly great.
  - mattkrause 6y ago
    I think you are...dramatically underestimating the size of many "modern" datasets and what most scientists want to do with them. We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution). An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller. All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.
  - ur-whale 6y ago
    Python itself is rather slow indeed, but more often than not, it doesn't matter: for many data science projects, python is only used as a glue to control the processing and/or pass data between low-level C++ implementation that can chew through data at great speed, (e.g using parallelism). Examples that immediately come to mind include : numpy, tensorflow, etc ... Even heavy-duty I/O can be made to crank with python if you do it properly.
  - deleted 6y ago
    [deleted]
adenadel 6y ago
Excel doesn't scale, isn't reproducible, and has behaviors that silently destroy data https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1044-7 https://genomebiology.biomedcentral.com/articles/10.1186/s13... https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
atty 6y ago
I don’t know what people think scientists are doing, but we are not just spreadsheet number crunching. In my PhD I was analyzing petabytes of structured data from particle detectors that required relatively complex algorithms just to reconstruct, before any physics analysis is ever started, and the analyses were just as complex. My experiment’s total code base was at least 100kloc of C++ and Fortran if not more, and used a farm with over 20k cores for massive distributed processing, single jobs regularly taking hundreds or thousands of cores and terabytes of memory. I’m currently working on distributed GPU accelerated hydrodynamic simulations where a single input is in the ~1-10Gb range of structured mesh data and the calculation requires ~200+ Gb of GPU memory. I really don’t see how excel would be a viable option for basically anything I have worked on in my scientific career, beyond my undergraduate toy analyses.

Does anyone know why scientists aren’t just using Excel? Nothing against Python/Julia but Excel seems like it would be my first choice (paired with some CSV files).

ur-whale 6y ago

Excel is far too slow for large scale datasets.

jimbob45 6y ago
Is that true? Python is pretty slow too. I can’t imagine many scientists would really mind leaving their work running overnight.
- proverbialbunny 6y ago
  It's true. Fun and relevant fact: The job title data scientist was invented at LinkedIn when they saw data analysts that needed to know R or Python skills. These skills seemed different enough from the traditional Excel and SAS data analyst jobs at the time to warrant a new job title. So did Excel not cut it, or was it a preference to use Python? Back then Excel supported up to 65,535 rows, and would typically crash with over 8,000 rows. I worked in that era in Excel. One model once was split across three spreadsheets. Only one spreadsheet could be open at a time, they would take 20 minutes each to load, and there was about a 50% chance they would crash while loading. So what do you do if you need over 65,000 instances of labeled data? For a neural network it's nice to have a million instances, yes a million. R and Python have in them what is called a dataframe. It's a spreadsheet, but in another programming language. We tend to load our data into those, which is just like Excel, but without the hardware limitations. So in many ways, today it's just like it once was, but we get to choose which programming language to use while working in a spreadsheet, and let's be fair, the Excel programming language isn't exactly great.
- mattkrause 6y ago
  I think you are...dramatically underestimating the size of many "modern" datasets and what most scientists want to do with them. We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution). An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller. All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.
- ur-whale 6y ago
  Python itself is rather slow indeed, but more often than not, it doesn't matter: for many data science projects, python is only used as a glue to control the processing and/or pass data between low-level C++ implementation that can chew through data at great speed, (e.g using parallelism). Examples that immediately come to mind include : numpy, tensorflow, etc ... Even heavy-duty I/O can be made to crank with python if you do it properly.
- deleted 6y ago
  [deleted]

adenadel 6y ago

Excel doesn't scale, isn't reproducible, and has behaviors that silently destroy data https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1044-7 https://genomebiology.biomedcentral.com/articles/10.1186/s13... https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

atty 6y ago

I don’t know what people think scientists are doing, but we are not just spreadsheet number crunching. In my PhD I was analyzing petabytes of structured data from particle detectors that required relatively complex algorithms just to reconstruct, before any physics analysis is ever started, and the analyses were just as complex. My experiment’s total code base was at least 100kloc of C++ and Fortran if not more, and used a farm with over 20k cores for massive distributed processing, single jobs regularly taking hundreds or thousands of cores and terabytes of memory. I’m currently working on distributed GPU accelerated hydrodynamic simulations where a single input is in the ~1-10Gb range of structured mesh data and the calculation requires ~200+ Gb of GPU memory. I really don’t see how excel would be a viable option for basically anything I have worked on in my scientific career, beyond my undergraduate toy analyses.