8 ms·

The reason for this is simple. Excel is 1. An incredibly powerful tool. 2. No bar of entry (cost aside, true in corporate environment). 3. Very gradual learn

The reason for this is simple. Excel is 1. An incredibly powerful tool. 2. No bar of entry (cost aside, true in corporate environment). 3. Very gradual learning curve. 4. The efficiency gain vs time invested is exponential. Power Excel users, much like their VIM/Emacs counter parts don't use a mouse. It is just keyboard short cuts [1][2]. This makes them insanely productive. [1] https://youtu.be/jFSf5YhYQbw https://youtu.be/jFSf5YhYQbw [2] https://youtu.be/0nbkaYsR94c https://youtu.be/0nbkaYsR94c

walshemj 10y ago
There are also many downsides
- gravypod 10y ago
  You can say that about every technology but the question is if the good out ways the bad in your specific usecase.
- valarauca1 10y ago
  Yes and No. If all the data you receiving is also coming to you as an Excel format (csv, xls, xlsx), but with major differences in formatting, or wholly inconsistent formatting. Now you have a multi-month long project just to have a consistent import script. Replacing a 1 second task done 2-3's times a day with a 4month project has an ROI on the scale of decades. Not worth it. Then you add visualization. What is 3-4 keystrokes in Excel is a lot of back of forth, learning a new library, ensuring it works on your system. Vetting the visualizing, dealing with that weird bug on the triple line double axis line chart. Then you have to validate integer handling and mathematics to ensure your newly written Python, Julia, etc. handles the same as your well vetted Excel Spread Sheet. Replacing that one slow bloated spread sheet is now nearly a year long project which requires a new employee who will have comparable pay to the person who ALREADY operates excel.
  - Swizec 10y ago
    > spread sheet is now nearly a year long project which requires a new employee who will have comparable pay to the person who ALREADY operates excel. And now you have a scalable system. You can go from something one employee takes all day to look at 2x/day, to something anyone in the company can see in real time on a dashboard of some sort. Is that worth it? Depends
- digikata 10y ago
  Don't know if this is the best study, but it contains an overview of many studies that show that excel calcs tend to have huge error rates. http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm Of course that raises the question would any other software environment have a lower error rate?
gravypod 10y ago
I can vouch for this. I thought spreadsheets where stupid and useless until I was forced to use them by my boss at the CS department here for grading. I didn't realize how easy having your computation and data being in the same place. It makes it slow, but usually for these sorts of things you want your data to be available on the scale of days, not nanoseconds so it works out. We have a complicated grading sheet here that manages all of the students information all one the same sheet in a google drive. Grades, attendance, recitation attendance, and at the end of the class homework. It also verifies the test answers against the correct ones to make sure we score exams correctly. Excel is great for fixed sets of data that need simple map/reductions & input verification because that's all we really use it for. After that move to something like Python&Numpy/R/Julia/Matlab.
- amyjess 10y ago
  I used to work at an NLP company, and we made extensive use of Google Sheets for doing P/R/F calculations on the results of various tests. It was so useful.
- wuschel 10y ago
  The nice thing about Excel is that it does resemble a functional programming tool. Would love to work with a replacement, even if it is some sort of Pandas/Python/Matplotlibb derivative - but it takes too long to set up things with these tools, and it seems not all operations are as trivial as I want them to be.
  - gravypod 10y ago
    Yea, for exel you can use "=(function)" and you're done! In Google Sheets you can also add in additional JavaScript code to run in the sheets.
flamedoge 10y ago
does it have regex search though?
- JoBrad 10y ago
  It's pretty easy to get RegEx exposed in Excel via VBA/extensions
  - valarauca1 10y ago
    This is the big key. A few short VBA macro's can give you regex's and cell swapping. Now you can conditionally swap the programs in other cells, via the contents of other cells. You have a Turing Complete spread sheet.
    - dsp1234 10y ago
      At one point, VBA in Microsoft Word was my only available programming outlet. I was able, easily, to access DirectDraw and create a faux screen saver. DLL imports are available in VBA, so the entire win32 API is available (in addition to the normal Office automation stuff like sending email, modifying spreadsheets, etc)
      - eon1 10y ago
        Yeah, in a similar situation right now and I find the DLL thing is relatively unknown and incredibly powerful if used well. Recently this 'robotics' evangelist keeps trying to rope us into spending a few mil on his automation and I keep showing him up by automating the same stuff right out of Excel for little to nothing.
gaur 10y ago
> 4. The efficiency gain vs learning curve is exponential. Does this actually mean anything?
- deleted 10y ago
  [deleted]
- DigitalJack 10y ago
  Yes. He is saying as you become more competent using excel, the efficiency increases exponentially. Probably true for smallish values of competency, but it must be logarithmic after that.
  - jackmott 10y ago
    probably a sigmoid function
  - gaur 10y ago
    > He is saying as you become more competent using excel, the efficiency increases exponentially. OK, does this mean anything? How have you quantified efficiency? How have you quantified "learning curve"? What data do you have supporting that the relationship is exponential?
    - DigitalJack 10y ago
      Being obtuse is not a desirable attribute, less so deliberately.
      - gaur 10y ago
        Asking people to say clearly and concretely what they mean is not obtuseness. Nobody here has been able to elaborate on the initial statement "The efficiency gain vs learning curve is exponential". People are just rewording the sentence slightly and passing that off as an explanation. That seems to indicate that nobody knows what the statement means because the statement is vacuous.
  - 10y ago
joelthelion 10y ago
Can you version/diff it though? Basically can you have a semi-sane software engineering process when it becomes big?
- moftz 10y ago
  You can track changes within Excel and with Office 2016, there is a comparison and merge tool included (that's been long overdue). But since the files are binary encoded, there is no external way to track changes unless someone wants to write up a parser for the XLSX format that can keep up with all the new features that MS adds every release.
  - bri3d 10y ago
    For what it's worth, XLSX files aren't binary. They're just XML in a ZIP. They're not particularly nasty to diff once they're extracted - actually, I think they might do pretty well in Git.
    - vsgxvhdxh 10y ago
      Ever tried editing the xml and rezipping? Also on examination you might notice the odd binary blob in the xml..
    - bjoerns 10y ago
      The diff is actually not really straightforward - a sheet is two-dimensional and you have to work out column/row inserts/deletes (which themselves are intertwined) before you can even start looking at cell changes. So it's quite non-standard stuff.
  - kofejnik 10y ago
    .docx and .xlsx are zipped xml, there are git plugins that produce reasonable diffs
VMG 10y ago
Is there any open source framework that has an excel-like GUI and good integration with standard coding tools?
- JBReefer 10y ago
  LibreOffice, but it's not as good as Excel for the upper-end stuff. Microsoft doesn't have an empire built on Office by accident.
- taserian 10y ago
  There used to be a couple of them for Python, but it seems they've died out. - Resolver One ( https://en.wikipedia.org/wiki/Resolver_One https://en.wikipedia.org/wiki/Resolver_One ) - Project Dirigible ( https://github.com/pythonanywhere/dirigible-spreadsheet https://github.com/pythonanywhere/dirigible-spreadsheet )
s_q_b 10y ago
This is digital Stockhold syndrome. Excel is something managers and executives can understand, so it became the default language for data analysis. Now technologists trapped using it have to create ex post facto justifications for why it's really "just misunderstood." Excel is massively slow, makes it easy for beginners to make massive mistakes, computes lots of types in very odd manners, performs floating point operations wrong, and leads to spaghetti code that is a rat's nest of incomprehensible cross-references. Worst of all, the lack of code path visibility usually leads to a bus factor of 1. Sure, one can learn to operate Excel for data analysis with a decent level of efficiency, in the same manner one can cross the Pacific in a canoe, but both are still terrible tools for the job.
- valarauca1 10y ago
  You are projecting so hard I could show PwerPoints off your forehead. I take it you are rather unhappy in your line of work? I'm just a lowly DBA re-posting and summarizing comments [1] for karma. [1] https://news.ycombinator.com/item?id=12448545 https://news.ycombinator.com/item?id=12448545
  - threatofrain 10y ago
    Do these videos have anything to do with beginners making mistakes, floating points and other type conversions, bus factor of 1, spaghetti code, etc? All I see is the same old Martin Shkreli video that has been floated around before, and all you see him do is 'Vim' around as he explains his thoughts -- not on Excel, but on company financials. Also, if you post a lopsided list of pros, it makes sense to the audience to see someone else post a lopsided list of cons. But then you reply with pettiness. Why?
- leshow 10y ago
  you mean Stockholm?
  - Smaug123 10y ago
    I thought it was mildly witty, given the subject matter.
inopinatus 10y ago
I know of a retail bank using a huge and complex Excel spreadsheet for their entire loan approvals process. Did it have bugs? No-one knew.
- walterstucco 10y ago
  could be the same Bank I know where they use excel for everything
  - ayuvar 10y ago
    I can personally guarantee you it is literally every bank. Even if they have a fancy tool, someone is using an Excel spreadsheet to figure out how to subvert it.
- bsg75 10y ago
  > Did it have bugs? No-one knew. And there is the reason Excel (and spreadsheets in general) are dangerous.
  - gokhan 10y ago
    I've seen software specifically made to tackle such complex processes and they were buggy as hell. And I've seen 100+ connected Excel spreadsheets managing $500M+ yearly transactions of the buying department of a white goods manufacturer, it was a work of art.
    - goatlover 10y ago
      Why would anyone do that? There are much better tools for handling complexity on that level. That's insane.
      - eon1 10y ago
        Two words: Corporate IT. Another two: Onboarding Process.
    - 10y ago
adakbar 10y ago
Couldn't agree more, in Indonesia Oil and Gas Upstream Special Task Force use Excel for almost everything, we try to replace Excel with many good app, but alas, when an app failed we just switch to good old Excel, someone even created Montecarlo simulation in Excel!
JackFr 10y ago
Indeed -- the most compelling reasons to move trading desks off Excel are non-technical. Model auditability and traceability are no longer 'nice-to-haves' but are compliance requirements.
deleted 10y ago
[deleted]
ska 10y ago
People really love it because they can do business analysis without hiring programmers or becoming them. And spreadsheets are a great model for a constrained set of problems. However, it's not all sweetness and light. Excel even gets some basic calculations wrong - and those ignorant of its quirks happily propagate those errors. More problematically, it can easily be pushed to the degree that your modeling is really beyond the tool or spreadsheet's design strengths without knowing it. And debugging is a pain in the ass. As a result lots of erroneous outputs get presented as meaninful.
bsg75 10y ago
All at the expense of reproducibility, testing, auditing. Quick and very, very dirty.
vegabook 10y ago
The efficiency gain vs time invested is not exponential. It's linear at best and plateaus after about 5k hours. It is this plateauing of the curve that is the biggest reason power Excel users move to R or Python. That was certainly my experience after eeking everything I could out of Excel in 15 years of trading floor fixed income. A visual 2d paradigm is excellent for quick productivity but is severely limiting as complexity and data size rises. Even with VB.
hyperbovine 10y ago
Lol, seriously? Martin Shkreli? I had been wondering what hole he crawled into.

walshemj 10y ago

There are also many downsides

gravypod 10y ago
You can say that about every technology but the question is if the good out ways the bad in your specific usecase.
valarauca1 10y ago
Yes and No. If all the data you receiving is also coming to you as an Excel format (csv, xls, xlsx), but with major differences in formatting, or wholly inconsistent formatting. Now you have a multi-month long project just to have a consistent import script. Replacing a 1 second task done 2-3's times a day with a 4month project has an ROI on the scale of decades. Not worth it. Then you add visualization. What is 3-4 keystrokes in Excel is a lot of back of forth, learning a new library, ensuring it works on your system. Vetting the visualizing, dealing with that weird bug on the triple line double axis line chart. Then you have to validate integer handling and mathematics to ensure your newly written Python, Julia, etc. handles the same as your well vetted Excel Spread Sheet. Replacing that one slow bloated spread sheet is now nearly a year long project which requires a new employee who will have comparable pay to the person who ALREADY operates excel.
- Swizec 10y ago
  > spread sheet is now nearly a year long project which requires a new employee who will have comparable pay to the person who ALREADY operates excel. And now you have a scalable system. You can go from something one employee takes all day to look at 2x/day, to something anyone in the company can see in real time on a dashboard of some sort. Is that worth it? Depends
digikata 10y ago
Don't know if this is the best study, but it contains an overview of many studies that show that excel calcs tend to have huge error rates. http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm Of course that raises the question would any other software environment have a lower error rate?

gravypod 10y ago

I can vouch for this. I thought spreadsheets where stupid and useless until I was forced to use them by my boss at the CS department here for grading. I didn't realize how easy having your computation and data being in the same place. It makes it slow, but usually for these sorts of things you want your data to be available on the scale of days, not nanoseconds so it works out. We have a complicated grading sheet here that manages all of the students information all one the same sheet in a google drive. Grades, attendance, recitation attendance, and at the end of the class homework. It also verifies the test answers against the correct ones to make sure we score exams correctly. Excel is great for fixed sets of data that need simple map/reductions & input verification because that's all we really use it for. After that move to something like Python&Numpy/R/Julia/Matlab.

amyjess 10y ago
I used to work at an NLP company, and we made extensive use of Google Sheets for doing P/R/F calculations on the results of various tests. It was so useful.
wuschel 10y ago
The nice thing about Excel is that it does resemble a functional programming tool. Would love to work with a replacement, even if it is some sort of Pandas/Python/Matplotlibb derivative - but it takes too long to set up things with these tools, and it seems not all operations are as trivial as I want them to be.
- gravypod 10y ago
  Yea, for exel you can use "=(function)" and you're done! In Google Sheets you can also add in additional JavaScript code to run in the sheets.

flamedoge 10y ago

does it have regex search though?

JoBrad 10y ago
It's pretty easy to get RegEx exposed in Excel via VBA/extensions
- valarauca1 10y ago
  This is the big key. A few short VBA macro's can give you regex's and cell swapping. Now you can conditionally swap the programs in other cells, via the contents of other cells. You have a Turing Complete spread sheet.
  - dsp1234 10y ago
    At one point, VBA in Microsoft Word was my only available programming outlet. I was able, easily, to access DirectDraw and create a faux screen saver. DLL imports are available in VBA, so the entire win32 API is available (in addition to the normal Office automation stuff like sending email, modifying spreadsheets, etc)
    - eon1 10y ago
      Yeah, in a similar situation right now and I find the DLL thing is relatively unknown and incredibly powerful if used well. Recently this 'robotics' evangelist keeps trying to rope us into spending a few mil on his automation and I keep showing him up by automating the same stuff right out of Excel for little to nothing.

gaur 10y ago

> 4. The efficiency gain vs learning curve is exponential. Does this actually mean anything?

deleted 10y ago
[deleted]
DigitalJack 10y ago
Yes. He is saying as you become more competent using excel, the efficiency increases exponentially. Probably true for smallish values of competency, but it must be logarithmic after that.
- jackmott 10y ago
  probably a sigmoid function
- gaur 10y ago
  > He is saying as you become more competent using excel, the efficiency increases exponentially. OK, does this mean anything? How have you quantified efficiency? How have you quantified "learning curve"? What data do you have supporting that the relationship is exponential?
  - DigitalJack 10y ago
    Being obtuse is not a desirable attribute, less so deliberately.
    - gaur 10y ago
      Asking people to say clearly and concretely what they mean is not obtuseness. Nobody here has been able to elaborate on the initial statement "The efficiency gain vs learning curve is exponential". People are just rewording the sentence slightly and passing that off as an explanation. That seems to indicate that nobody knows what the statement means because the statement is vacuous.
- 10y ago

joelthelion 10y ago

Can you version/diff it though? Basically can you have a semi-sane software engineering process when it becomes big?

moftz 10y ago
You can track changes within Excel and with Office 2016, there is a comparison and merge tool included (that's been long overdue). But since the files are binary encoded, there is no external way to track changes unless someone wants to write up a parser for the XLSX format that can keep up with all the new features that MS adds every release.
- bri3d 10y ago
  For what it's worth, XLSX files aren't binary. They're just XML in a ZIP. They're not particularly nasty to diff once they're extracted - actually, I think they might do pretty well in Git.
  - vsgxvhdxh 10y ago
    Ever tried editing the xml and rezipping? Also on examination you might notice the odd binary blob in the xml..
  - bjoerns 10y ago
    The diff is actually not really straightforward - a sheet is two-dimensional and you have to work out column/row inserts/deletes (which themselves are intertwined) before you can even start looking at cell changes. So it's quite non-standard stuff.
- kofejnik 10y ago
  .docx and .xlsx are zipped xml, there are git plugins that produce reasonable diffs

VMG 10y ago

Is there any open source framework that has an excel-like GUI and good integration with standard coding tools?

JBReefer 10y ago
LibreOffice, but it's not as good as Excel for the upper-end stuff. Microsoft doesn't have an empire built on Office by accident.
taserian 10y ago
There used to be a couple of them for Python, but it seems they've died out. - Resolver One ( https://en.wikipedia.org/wiki/Resolver_One https://en.wikipedia.org/wiki/Resolver_One ) - Project Dirigible ( https://github.com/pythonanywhere/dirigible-spreadsheet https://github.com/pythonanywhere/dirigible-spreadsheet )

s_q_b 10y ago

This is digital Stockhold syndrome. Excel is something managers and executives can understand, so it became the default language for data analysis. Now technologists trapped using it have to create ex post facto justifications for why it's really "just misunderstood." Excel is massively slow, makes it easy for beginners to make massive mistakes, computes lots of types in very odd manners, performs floating point operations wrong, and leads to spaghetti code that is a rat's nest of incomprehensible cross-references. Worst of all, the lack of code path visibility usually leads to a bus factor of 1. Sure, one can learn to operate Excel for data analysis with a decent level of efficiency, in the same manner one can cross the Pacific in a canoe, but both are still terrible tools for the job.

valarauca1 10y ago
You are projecting so hard I could show PwerPoints off your forehead. I take it you are rather unhappy in your line of work? I'm just a lowly DBA re-posting and summarizing comments [1] for karma. [1] https://news.ycombinator.com/item?id=12448545 https://news.ycombinator.com/item?id=12448545
- threatofrain 10y ago
  Do these videos have anything to do with beginners making mistakes, floating points and other type conversions, bus factor of 1, spaghetti code, etc? All I see is the same old Martin Shkreli video that has been floated around before, and all you see him do is 'Vim' around as he explains his thoughts -- not on Excel, but on company financials. Also, if you post a lopsided list of pros, it makes sense to the audience to see someone else post a lopsided list of cons. But then you reply with pettiness. Why?
leshow 10y ago
you mean Stockholm?
- Smaug123 10y ago
  I thought it was mildly witty, given the subject matter.

inopinatus 10y ago

I know of a retail bank using a huge and complex Excel spreadsheet for their entire loan approvals process. Did it have bugs? No-one knew.

walterstucco 10y ago
could be the same Bank I know where they use excel for everything
- ayuvar 10y ago
  I can personally guarantee you it is literally every bank. Even if they have a fancy tool, someone is using an Excel spreadsheet to figure out how to subvert it.
bsg75 10y ago
> Did it have bugs? No-one knew. And there is the reason Excel (and spreadsheets in general) are dangerous.
- gokhan 10y ago
  I've seen software specifically made to tackle such complex processes and they were buggy as hell. And I've seen 100+ connected Excel spreadsheets managing $500M+ yearly transactions of the buying department of a white goods manufacturer, it was a work of art.
  - goatlover 10y ago
    Why would anyone do that? There are much better tools for handling complexity on that level. That's insane.
    - eon1 10y ago
      Two words: Corporate IT. Another two: Onboarding Process.
  - 10y ago

adakbar 10y ago

Couldn't agree more, in Indonesia Oil and Gas Upstream Special Task Force use Excel for almost everything, we try to replace Excel with many good app, but alas, when an app failed we just switch to good old Excel, someone even created Montecarlo simulation in Excel!

JackFr 10y ago

Indeed -- the most compelling reasons to move trading desks off Excel are non-technical. Model auditability and traceability are no longer 'nice-to-haves' but are compliance requirements.

deleted 10y ago

[deleted]

ska 10y ago

People really love it because they can do business analysis without hiring programmers or becoming them. And spreadsheets are a great model for a constrained set of problems. However, it's not all sweetness and light. Excel even gets some basic calculations wrong - and those ignorant of its quirks happily propagate those errors. More problematically, it can easily be pushed to the degree that your modeling is really beyond the tool or spreadsheet's design strengths without knowing it. And debugging is a pain in the ass. As a result lots of erroneous outputs get presented as meaninful.

bsg75 10y ago

All at the expense of reproducibility, testing, auditing. Quick and very, very dirty.

vegabook 10y ago

The efficiency gain vs time invested is not exponential. It's linear at best and plateaus after about 5k hours. It is this plateauing of the curve that is the biggest reason power Excel users move to R or Python. That was certainly my experience after eeking everything I could out of Excel in 15 years of trading floor fixed income. A visual 2d paradigm is excellent for quick productivity but is severely limiting as complexity and data size rises. Even with VB.

hyperbovine 10y ago

Lol, seriously? Martin Shkreli? I had been wondering what hole he crawled into.