3 ms·

I work with an army of very smart people who need to build incredibly complex models to drive our business. Like most companies, we turn to Excel. It's the only

by roin 13y ago

I work with an army of very smart people who need to build incredibly complex models to drive our business. Like most companies, we turn to Excel. It's the only widely used application with a low enough barrier to entry that "non-technical" people can build legitimately complex (if not always robust) models. But the problems with Excel and general spreadsheet modeling limitations offer such a great opportunity for improvement that would impact just about every business out there. I wish I saw more people tackling this, or I had a great idea to do it myself. The problems I see are around barrier to entry, in that it must be usable in just a couple minutes, simple enough for non-programmers to learn, and equally or more efficient to get a basic model functioning. Otherwise I just don't see wide adoption despite possible maintenance, accuracy, and reliability benefits. James Kwak has written some smart stuff about this (http://baselinescenario.com/2013/04/18/more-bad-excel/ http://baselinescenario.com/2013/04/18/more-bad-excel/), and I'm interested in what Data Nitro (datanitro.com) is doing.

peteforde 13y ago
I think Data Nitro is really cool, and as a data geek the idea of having a real language available inside the Excel environment gets me very excited. Unfortunately, by increasing the power and capability of what a careless user can do, while expanding the learning curve and moving Python code to a different place than any cell-level functions or global VBA functions are stored... all of this just makes the potential for issues of human failure far greater.
- BrianEatWorld 13y ago
  Is there a reason you don't use one of the current data analysis languages like R, Stata, SAS, etc?
  - peteforde 13y ago
    I use those languages frequently, when they are the right tool for the job. Well, R and Stata, at least — but also Processing and D3 and Google Maps... Not sure how this is relevant to my concern, though.
elirubel 13y ago
Interesting and agreed. What if excel was very basic in functionality, and then as soon as you get into complex modeling or advanced functions you pull from a user generated (either private or public) template library. The trick would be in making the implementation of templating simple and intuitive. The goal behind templating would be to prevent complex formulas from being entered in tiny cells, and to leverage a userbase that has "been there, done that." Do you think that would shorten new hire ramp up in using existing complex models etc?
gruseom 13y ago
I worked in a similar environment and came to similar conclusions. In my view, your three requirements (usable without a big learning curve, intuitive to non-programmers, equally or more efficient than Excel) point directly back to the spreadsheet UI. Spreadsheet users get spreadsheets in a way they don't get anything else. It's their mental model and they love it and there's no way they're going to give it up. (In particular, there's no way they're going to migrate to even the simplest of general-purpose scripting languages.) Therefore the only thing that will get them off Excel is a new kind of spreadsheet. The question is, what would that look like? If it's too different from Excel, it won't meet your three requirements. But if it's too like Excel, there won't be any reason to use it, nor will it address Excel's limitations which are the subject of the OP. I've been thinking about and working on this ever since—it's a deeper problem than it appears to be! Edit: by the way, you're right (in my opinion) to emphasize barrier-to-entry as a great danger in this area. If you look at the history of attempts to innovate in the spreadsheet space (most famously Lotus Improv), that is the reef on which they tend to run aground.
- pyoung 13y ago
  I don't think scripting languages, of themselves, are the real barrier, after all, excel formulas are not too far off from what one would use in a language like R. I think the barrier is the data import/manipulation phase. Scripting languages loose out to the ease of copy/paste and data inspection that spreadsheets offer. It's not that using scripting languages for these tasks is very difficult, it's just that spreadsheets are, comparatively, much easier to use. I worked in a place that used both, and the programmers were successful in preaching the benefits and converting some of the excel users, where it made sense. However, I imagine a large majority of analysts using excel in the corporate world do not have that exposure and don't realize there are other alternatives out there, or if they do know about the alternatives, they get quickly turned off from them when they try to import/manipulate the data. I don't think any tool, by itself, is going to push individuals to change. People need to be exposed to the benefits of approaching data analysis as a small software/programming project over the long term. It is short term thinking that pushes people to use excel in the first place.
  - 13y ago

by roin 13y ago

peteforde 13y ago

I think Data Nitro is really cool, and as a data geek the idea of having a real language available inside the Excel environment gets me very excited. Unfortunately, by increasing the power and capability of what a careless user can do, while expanding the learning curve and moving Python code to a different place than any cell-level functions or global VBA functions are stored... all of this just makes the potential for issues of human failure far greater.

BrianEatWorld 13y ago
Is there a reason you don't use one of the current data analysis languages like R, Stata, SAS, etc?
- peteforde 13y ago
  I use those languages frequently, when they are the right tool for the job. Well, R and Stata, at least — but also Processing and D3 and Google Maps... Not sure how this is relevant to my concern, though.

elirubel 13y ago

Interesting and agreed. What if excel was very basic in functionality, and then as soon as you get into complex modeling or advanced functions you pull from a user generated (either private or public) template library. The trick would be in making the implementation of templating simple and intuitive. The goal behind templating would be to prevent complex formulas from being entered in tiny cells, and to leverage a userbase that has "been there, done that." Do you think that would shorten new hire ramp up in using existing complex models etc?

gruseom 13y ago

I worked in a similar environment and came to similar conclusions. In my view, your three requirements (usable without a big learning curve, intuitive to non-programmers, equally or more efficient than Excel) point directly back to the spreadsheet UI. Spreadsheet users get spreadsheets in a way they don't get anything else. It's their mental model and they love it and there's no way they're going to give it up. (In particular, there's no way they're going to migrate to even the simplest of general-purpose scripting languages.) Therefore the only thing that will get them off Excel is a new kind of spreadsheet. The question is, what would that look like? If it's too different from Excel, it won't meet your three requirements. But if it's too like Excel, there won't be any reason to use it, nor will it address Excel's limitations which are the subject of the OP. I've been thinking about and working on this ever since—it's a deeper problem than it appears to be! Edit: by the way, you're right (in my opinion) to emphasize barrier-to-entry as a great danger in this area. If you look at the history of attempts to innovate in the spreadsheet space (most famously Lotus Improv), that is the reef on which they tend to run aground.

pyoung 13y ago
I don't think scripting languages, of themselves, are the real barrier, after all, excel formulas are not too far off from what one would use in a language like R. I think the barrier is the data import/manipulation phase. Scripting languages loose out to the ease of copy/paste and data inspection that spreadsheets offer. It's not that using scripting languages for these tasks is very difficult, it's just that spreadsheets are, comparatively, much easier to use. I worked in a place that used both, and the programmers were successful in preaching the benefits and converting some of the excel users, where it made sense. However, I imagine a large majority of analysts using excel in the corporate world do not have that exposure and don't realize there are other alternatives out there, or if they do know about the alternatives, they get quickly turned off from them when they try to import/manipulate the data. I don't think any tool, by itself, is going to push individuals to change. People need to be exposed to the benefits of approaching data analysis as a small software/programming project over the long term. It is short term thinking that pushes people to use excel in the first place.
- 13y ago