3 ms·

Is that true? Python is pretty slow too. I can’t imagine many scientists would really mind leaving their work running overnight.

proverbialbunny 6y ago
It's true. Fun and relevant fact: The job title data scientist was invented at LinkedIn when they saw data analysts that needed to know R or Python skills. These skills seemed different enough from the traditional Excel and SAS data analyst jobs at the time to warrant a new job title. So did Excel not cut it, or was it a preference to use Python? Back then Excel supported up to 65,535 rows, and would typically crash with over 8,000 rows. I worked in that era in Excel. One model once was split across three spreadsheets. Only one spreadsheet could be open at a time, they would take 20 minutes each to load, and there was about a 50% chance they would crash while loading. So what do you do if you need over 65,000 instances of labeled data? For a neural network it's nice to have a million instances, yes a million. R and Python have in them what is called a dataframe. It's a spreadsheet, but in another programming language. We tend to load our data into those, which is just like Excel, but without the hardware limitations. So in many ways, today it's just like it once was, but we get to choose which programming language to use while working in a spreadsheet, and let's be fair, the Excel programming language isn't exactly great.
mattkrause 6y ago
I think you are...dramatically underestimating the size of many "modern" datasets and what most scientists want to do with them. We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution). An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller. All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.
ur-whale 6y ago
Python itself is rather slow indeed, but more often than not, it doesn't matter: for many data science projects, python is only used as a glue to control the processing and/or pass data between low-level C++ implementation that can chew through data at great speed, (e.g using parallelism). Examples that immediately come to mind include : numpy, tensorflow, etc ... Even heavy-duty I/O can be made to crank with python if you do it properly.
deleted 6y ago
[deleted]

proverbialbunny 6y ago

It's true. Fun and relevant fact: The job title data scientist was invented at LinkedIn when they saw data analysts that needed to know R or Python skills. These skills seemed different enough from the traditional Excel and SAS data analyst jobs at the time to warrant a new job title. So did Excel not cut it, or was it a preference to use Python? Back then Excel supported up to 65,535 rows, and would typically crash with over 8,000 rows. I worked in that era in Excel. One model once was split across three spreadsheets. Only one spreadsheet could be open at a time, they would take 20 minutes each to load, and there was about a 50% chance they would crash while loading. So what do you do if you need over 65,000 instances of labeled data? For a neural network it's nice to have a million instances, yes a million. R and Python have in them what is called a dataframe. It's a spreadsheet, but in another programming language. We tend to load our data into those, which is just like Excel, but without the hardware limitations. So in many ways, today it's just like it once was, but we get to choose which programming language to use while working in a spreadsheet, and let's be fair, the Excel programming language isn't exactly great.

mattkrause 6y ago

I think you are...dramatically underestimating the size of many "modern" datasets and what most scientists want to do with them. We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution). An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller. All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.

ur-whale 6y ago

Python itself is rather slow indeed, but more often than not, it doesn't matter: for many data science projects, python is only used as a glue to control the processing and/or pass data between low-level C++ implementation that can chew through data at great speed, (e.g using parallelism). Examples that immediately come to mind include : numpy, tensorflow, etc ... Even heavy-duty I/O can be made to crank with python if you do it properly.

deleted 6y ago

[deleted]