3 ms·

I don't think scripting languages, of themselves, are the real barrier, after all, excel formulas are not too far off from what one would use in a language like

by pyoung 13y ago

I don't think scripting languages, of themselves, are the real barrier, after all, excel formulas are not too far off from what one would use in a language like R. I think the barrier is the data import/manipulation phase. Scripting languages loose out to the ease of copy/paste and data inspection that spreadsheets offer. It's not that using scripting languages for these tasks is very difficult, it's just that spreadsheets are, comparatively, much easier to use. I worked in a place that used both, and the programmers were successful in preaching the benefits and converting some of the excel users, where it made sense. However, I imagine a large majority of analysts using excel in the corporate world do not have that exposure and don't realize there are other alternatives out there, or if they do know about the alternatives, they get quickly turned off from them when they try to import/manipulate the data. I don't think any tool, by itself, is going to push individuals to change. People need to be exposed to the benefits of approaching data analysis as a small software/programming project over the long term. It is short term thinking that pushes people to use excel in the first place.

gruseom 13y ago
I respectfully but strongly disagree. There is a chasm between Excel formulas and general-purpose programming languages. Very many (even very sophisticated) spreadsheet users have no interest in crossing it. These are the users I was talking about. They're not programmers-in-waiting. They think in spreadsheets. The difference between the two paradigms is much deeper than data import/export, and exposing most spreadsheet users to scripting languages (even with good import/export) won't convert them, it will just turn them off. I know this from hard experience, having built a domain-specific scripting language on a previous project that was trying to make software for petroleum economic modelers (i.e., sophisticated but non-programmer business experts). Every feature request we got boiled down to, "make it be like Excel, complete with an interactive grid UI". It isn't short-term thinking that drives this kind of user to build complex models in Excel. It's that Excel fits how their minds work. Partly that's because spreadsheets are intuitive for many people, and partly it's because spreadsheets been mass-adopted for so long that they're utterly entrenched. Trying to get spreadsheet users to think more like programmers is a sure way to keep them stuck on Excel forever. Here's an example of how different the thinking is. In a programming language one first builds a computation abstractly (code), then instantiates it on concrete data (input) and looks at the results (output). Spreadsheet users don't do that. They start with a concrete example, build the formulas they need to get that particular example working, and then (if they need a similar example) copy what they did and tweak it. In other words, a spreadsheet user is never looking at a pure abstraction; they are always looking at a concrete example, even when the computations they're building up are complex. If you showed them all the formulas at once, they would recoil. I think this low tolerance for abstraction is one reason why spreadsheets fit the minds of non-programmers better, and also why it is so difficult for those of us who are comfortable with abstraction (and thus with programming languages) to appreciate how different spreadsheets are.
- pyoung 13y ago
  I agree about your point on abstraction, but I also think there are different methods to deal with it, and that those methods often get overlooked. My experience with SAS and R suggests it is possible to somewhat re-create the typical excel workflow. The SAS IDE allows you to open the data sets created in your code, which means you can write code one line (well, more accurately, one data step or procedure, as it is called in SAS) at a time and inspect the intermediate results at each point. Similarly, RStudio allows you to inspect the values of data frames and variables, once again allowing the user to step through the code writing process and get relatively immediate feedback. Regardless, there are going to be a large portion of excel users that won't or can't switch, and there are also a number of use cases where excel makes perfect sense (smaller data, simpler summaries), but I have personally 'converted' a few analysts, and think that, with more exposure, a not insignificant portion of excel users could be convinced to make the leap.

by pyoung 13y ago

gruseom 13y ago

I respectfully but strongly disagree. There is a chasm between Excel formulas and general-purpose programming languages. Very many (even very sophisticated) spreadsheet users have no interest in crossing it. These are the users I was talking about. They're not programmers-in-waiting. They think in spreadsheets. The difference between the two paradigms is much deeper than data import/export, and exposing most spreadsheet users to scripting languages (even with good import/export) won't convert them, it will just turn them off. I know this from hard experience, having built a domain-specific scripting language on a previous project that was trying to make software for petroleum economic modelers (i.e., sophisticated but non-programmer business experts). Every feature request we got boiled down to, "make it be like Excel, complete with an interactive grid UI". It isn't short-term thinking that drives this kind of user to build complex models in Excel. It's that Excel fits how their minds work. Partly that's because spreadsheets are intuitive for many people, and partly it's because spreadsheets been mass-adopted for so long that they're utterly entrenched. Trying to get spreadsheet users to think more like programmers is a sure way to keep them stuck on Excel forever. Here's an example of how different the thinking is. In a programming language one first builds a computation abstractly (code), then instantiates it on concrete data (input) and looks at the results (output). Spreadsheet users don't do that. They start with a concrete example, build the formulas they need to get that particular example working, and then (if they need a similar example) copy what they did and tweak it. In other words, a spreadsheet user is never looking at a pure abstraction; they are always looking at a concrete example, even when the computations they're building up are complex. If you showed them all the formulas at once, they would recoil. I think this low tolerance for abstraction is one reason why spreadsheets fit the minds of non-programmers better, and also why it is so difficult for those of us who are comfortable with abstraction (and thus with programming languages) to appreciate how different spreadsheets are.

pyoung 13y ago
I agree about your point on abstraction, but I also think there are different methods to deal with it, and that those methods often get overlooked. My experience with SAS and R suggests it is possible to somewhat re-create the typical excel workflow. The SAS IDE allows you to open the data sets created in your code, which means you can write code one line (well, more accurately, one data step or procedure, as it is called in SAS) at a time and inspect the intermediate results at each point. Similarly, RStudio allows you to inspect the values of data frames and variables, once again allowing the user to step through the code writing process and get relatively immediate feedback. Regardless, there are going to be a large portion of excel users that won't or can't switch, and there are also a number of use cases where excel makes perfect sense (smaller data, simpler summaries), but I have personally 'converted' a few analysts, and think that, with more exposure, a not insignificant portion of excel users could be convinced to make the leap.