6 ms·
In defense of Excel
- golergka 13y agoI don't think that Excel was ever in need to be defended. It's one of the best analytic tools for tasks of proper scope. It was also the first tool that we had to learn and master in bioinformatics class before writing even a single line of code.
- cowls 13y agoAgreed, in my experience Excel is an excellent tool for beginners and advanced users. I don't think it has too much to worry about from "tableau"...
- dangoldin 13y agoMaybe I'm in the wrong circles but many of the "hardcore" quant folk will belittle it. That's why I decided to write something up. I don't think Excel's the right tool for every job but it's a lot more powerful than many people give it credit for.
- chollida1 13y agoInteresting, the two quants beside me and myself, all have excel open right now. I guess our fund isn't "hardcore":)
- dangoldin 13y agoHaha - when I was a quant I used it all the time too. We had Perl jobs that would generate Excel reports that would then be distributed to the various PMs so that they could look into the details. I think the finance industry runs on Excel.
- ardemue 13y agoAny industry runs on Excel I guess. I work in aeronautics and everyone who works on a computer maintains some Excel spreadsheets. It's everywhere but so few people know how to use formula, pivot tables, etc, that it has become a real competitive advantage to master them.
- dons 13y agoIf they don't have vim or emacs open, they're not hardcore.
- jasonpbecker 13y agoI belittle it. A lot. And rightfully so. It's fine if you want to keep a small budget or a quick bit of work with numbers, but it's awful for dealing even modest amounts of data and here's why: Nothing you do is captured and everything is destructive. Any time someone edits their spreadsheet they are irreparably altering the data without real documentation. Excel encourages moving, shaping, dragging, and futzing that may be easier, but ultimately can cause major problems because there's no audit trail. Ever try and replicate something modestly sophisticated that was done in Excel? Good freaking luck. There's nothing wrong with Excel for some basic graphs and calculations. The problem is when people start to use Excel as a data base or as a serious tool for analytics (like some of those business folks).
- smacktoward 13y agoWhat's needed is a "serious tool" that's as easy for people to get into and understand as Excel is. Excel is as ubiquitous as it is because the mental model of a spreadsheet is very simple and easy to wrap one's brain around. Here's a grid of cells, type stuff into them. Boom, you're an Excel user. (It can do a lot more than that, of course, but many many many people never realize that and still get lots of value out of it. Which is kind of remarkable, really.) Proper databases are much more powerful, but none of them present the user with an interface whose basic metaphors are as immediately obvious as Excel's. You have to learn about tables, data types, relations, etc. to get anything done in them. Most of that stuff comes up with Excel too, but you don't need to know it to get started. The Excel user's first baby steps are much smaller than the database user's are. Given a choice between a flawed tool that's immediately understandable and a perfect tool that requires learning, people will always choose the flawed-but-understandable tool. So they go with Excel.
- pyoung 13y agoI don't think we should necessarily aim to make "serious tools" easy for people to get into. I am not saying that the UI shouldn't be intuitive, but rather that making something 'too easy' results in low barriers to entry and you end up with the wrong types of people using that tool. Excel is very useful (I use it much in the same way the OP does), but it has it's limitations. Because it is so easy to get into, it's easy for individuals lacking the proper experience to over extend themselves. I think much of the explosion in data science jobs these last few years isn't really about "big data", but rather the fact that your typical corporate analyst does not have the experience to tackle mid range data analysis jobs because the only tool they have under their belts is Excel. What really is needed is a larger number of people with experience in statistics, scientific computing (Python, R etc..), and database systems. The current tools we have are fine, they just need to be used properly (well, we should always strive to improve our tools, but you get the idea).
- Sulfolobus 13y agoI've had success shifting my bioinformatics students that want something graphical but with more of a statistical framework to Rstudio or minitab. You did remind me of this amusing paper from a few years ago though: http://www.biomedcentral.com/1471-2105/5/80 http://www.biomedcentral.com/1471-2105/5/80
- sailfast 13y agoExcel can be an excellent tool - I think the problem is in creating something that buries key calculations or is heavily prone to human error so major mistakes are made. Some of these things (recently reported in the news) move markets and made / lost billions. You should definitely check your math! Tableau is great for visualizations but if you want to do anything on your own servers it is prohibitively expensive and there are other alternatives out there. Not sure it provides the same capability as Excel for analysis.
- chadgeidel 13y agoThis is my biggest gripe. There have been several well-publicised studies where calculations were in error. 88% Of excel spreadsheets have errors - I couldn't find the actual study after a quick googling. http://www.marketwatch.com/story/88-of-spreadsheets-have-errors-2013-04-17 http://www.marketwatch.com/story/88-of-spreadsheets-have-err... Recent national debt study found to contain erroneous calculations; http://www.nextnewdeal.net/rortybomb/researchers-finally-replicated-reinhart-rogoff-and-there-are-serious-problems http://www.nextnewdeal.net/rortybomb/researchers-finally-rep...
- simonh 13y agoThe finance industry definitely has love hate relationship with Excel. We're trying to encourage moving Excel style analysis into Python and have developed a library specifically to facilitate that, meanwhile we make sure our application platforms have first class Excel integration and export functionality. To my mind the key point in the article is lack of facilities for code review of Excel spreadsheets. Yes there are tools to help with that, but they're not widely used. If our developers routinely pushed their development code directly to production without proper review and testing, someone (probably several people) would get fired on the spot, yet important business decisions get made based on unreviewed Excel spreadsheets all the time.
- lucozade 13y agoIf 88% of spreadsheets have errors, what's the likelihood that the figure of 88% is correct?
- bgilroy26 13y agoSarbane Oxley will take a big bite out of MS Office licenses. The law is 10 years old, but there was a lot of lock in in 2003. The JP whale provides a great just-so story for regulators who want to take powerful, flexible tools like excel out of analysts hands. Give it another 10 years, Excel will not have the ubiquity it enjoys today in financial institutions.
- Pxtl 13y agoMy problem with excel is that it is not conducive to making well-documented systems. The formulas are hidden and there's no way to comment them. The ability to name your variables is clumsy and pushed away from the main UI so most users are unaware of it and tend to use anonymous variables. The default UI for creating formulas binds them to a fixed range of cells instead of the entire column, which is absurd. It's intensely powerful (my boss can do spectacular things with pivot tables and teaches all the business people to do this as a means to doing their own analysis of our reports) but the high-power features have a positively nasty UI compared to the corresponding SQL query. I'd love to see an Excel designed-from-scratch that scratches the same itches of approachability and user-friendliness but with an awareness of Excel's failings and without obsessing over the spreadsheet metaphor. Something in-between Excel and Access.
- jasonpbecker 13y agoExcel is super powerful, but I loathe that power. It leads to people doing a ton of amazing, great work in Excel that's super impressive... that no on else can replicate. If you're really smart about it, Excel is one of the most remarkable tools for analysis ever devised. The problem is virtually everyone who is smart about it can use different better tools that are more suited to their purpose.
- dragonwriter 13y ago> Excel is super powerful, but I loathe that power. It leads to people doing a ton of amazing, great work in Excel that's super impressive... that no on else can replicate. More than "replicate", I think the problem is no one else -- and, with enough time and complexity, not even the originator -- can maintain it, which results in Excel often being a vehicle for low-initiation-cost and rapidly-rising-maintenance-cost tools (which, in effect, often end up as straightjackets as no one wants to bear the cost of starting over, or the cost of updating, so business processes end up getting constrained by the tool rather than the tool adapting to processes.)
- dangoldin 13y agoInteresting idea. It's such a difficult usability problem that it's tough to solve it for everyone. I had a job where I was responsible for getting business folk the data they needed and kept on trying to come up with a way for them to extract data from the database without having them to write queries. I got this nugget from the CTO: "To get the expressiveness of SQL you will be writing SQL" Just need to figure out where the tradeoffs are and I think Excel made a pretty good one. I think Excel shouldn't be used for recurring jobs but for the one off it can't be beat.
- PaulHoule 13y agoI just computed summary statistics from something in my Hadoop cluster and I made the plots with Excel rather than R because it was a lot quicker. If I had to generate these plots every week then I could script it easily with R, but in a fast-paced environment excel has its place
- nonchalance 13y agoExcel needs no defense: it is a rock-solid product which gives non-programmers lots of power. People have tried for decades to build alternatives with little success. EDIT: My favorite example is http://carywalkin.ca/download-arena-xlsm/ http://carywalkin.ca/download-arena-xlsm/
- Someone 13y agoAlso: http://www.geocities.jp/nchikada/pac/ http://www.geocities.jp/nchikada/pac/ Pacman in Excel.
- DanBC 13y agoI agree, Excel (and other spreadsheet software) is easy and fun. But people are bad with numbers and spreadsheets give them the power to be bad with lots of numbers very quickly. Spreadsheets are tricky to audit, and often they're not audited because they're just a tool that some guy uses without anyone knowing. I've mention the EU SPreadsheet Risk Interest Group before, but here's the link for people who haven't seen it (http://www.eusprig.org/ http://www.eusprig.org/) Panko has a nice website with some information about human error and the things that people do with spreadsheets (http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm) If you like Excel and pivot tables you might like PowerPivot, a free addon from MS. Here's some information about it: (http://www.powerpivotpro.com/what-is-powerpivot/ http://www.powerpivotpro.com/what-is-powerpivot/) EDIT: I like some of the horror stories here (http://www.eusprig.org/horror-stories.htm http://www.eusprig.org/horror-stories.htm), eg > The London 2012 organising committee (Locog) confirmed on Wednesday that a decidedly unsynchronised error in its ticketing process had led to four synchronised swimming sessions being oversold by 10,000 tickets. [...] > Locog said the error occurred in the summer, between the first and second round of ticket sales, when a member of staff made a single keystroke mistake and entered ‘20,000’ into a spreadsheet rather than the correct figure of 10,000 remaining tickets. The error was discovered when Locog reconciled the number of tickets sold against the final layouts and seating configurations for venues, and began contacting ticket holders before Christmas.
- gruseom 13y agoIsn't it a bit silly to blame Excel for a user's typing 20,000 instead of 10,000? Computers can't magically come up with the right answer from wrong input. You could argue that there should have been error handling or data validation, but most custom software doesn't do that either.
- DanBC 13y agoI'm not singling out one spreadsheet package. I'm not even singling out spreadsheets - thus the link to Panko and "human errors". But fat-finger errors are very easy to make with spreadsheets, and very hard to find. This error, (20,000 instead of 10,000) was only found when LOCOG realised they didn't have enough seats for all the tickets they'd sold. Spreadsheets are particularly tricky because people scatter input data among all those cells; Sometimes data is manually copied and pasted (which can either update automatically with outher changes, or not, depending how it's cut and pasted); they then manipulate those cells with formulas (sometimes across multiple sheets) and then they spit out a number. Or sometimes not even a number but a bar on a chart. Few people bother looking at the input or at the formulas used, they just see the output and sometimes apply a brief sanity check. > You could argue that there should have been error handling or data validation, but most custom software doesn't do that either. Most people don't write custom software. Spreadsheets give people a very powerful tool. Most people don't have the skills to protect themselves from that power - they don't know about error handling or data validation and spreadsheets make it hard to provide those sanitation and checking tools.
- abvdasker 13y agoI recently had to develop a little reporting system for my company's sales team and I have to say, Excel has some excellent interfaces on Windows. The little app I built is a GUI that lets someone choose a spreadsheet tied to a remote MySQL database via ODBC, refresh the data in that spreadsheet and email it at scheduled intervals. It literally took like a few days. Also, coming from a Java background but doing all my recent work in Ruby (about which I feel ambivalent), coding in C# was a real pleasure. I'm more of a text-editor type, but I was also profoundly impressed with VisualStudio. Microsoft ain't all bad.
- James_Duval 13y agoI'm currently attempting to write a roguelike using nothing other than Excel formulae (no macros etc.). It's going surprisingly well so far, although the limitations are obviously frustrating - not least the inability to correctly lock-in random numbers, although I think that can be worked around by linking the data from another workbook. Obviously this is not a serious project, but it's kind of fun. I think this is where Excel gets dangerous. To a certain type of mind (perhaps like mine), the enjoyment from forcing a non-compliant system to do your bidding is greater than the enjoyment of using the proper tool to do the job quickly and effectively. I still don't use pivot tables. Those are too practical.
- doppenhe 13y agothis is cool. I am pretty well versed in excel myself if you need any help.
- DanBC 13y ago> I think this is where Excel gets dangerous. To a certain type of mind (perhaps like mine), the enjoyment from forcing a non-compliant system to do your bidding is greater than the enjoyment of using the proper tool to do the job quickly and effectively. You're hacking around with something for fun. Other people? Not so much. EG, the guy who used MS Works spreadsheet (which was not loadable by anything else) for everything, including word processing. He'd open a spreadsheet, and type text in, and get "nicely" laid out documents.
- kyllo 13y agoThe biggest problem with Excel is version control, because a spreadsheet is just a file (.xls are binary, .xlsx are XML but not human-readable), and people share it as an e-mail attachment. This is what makes it a horrible, unmaintainable mess. This is also the advantage that Google Docs Spreadsheets have over Excel. The data is cloud-hosted and automatically keeps a revision history, allowing non-technical users to collaboratively edit without having to learn about version control. But most companies don't allow their employees to upload company data into Google Docs, and with good reason, especially in light of the NSA spying scandal.
- doppenhe 13y agoSkyDrive and Office 365 provide this for the Microsoft suite.
- kyllo 13y agoRight, but with the disadvantage of not having a free version, and they still have the same downside as Google Docs, which is that your cloud-hosted data is accessible by the NSA, who may then share it with other government agencies. Or it could simply be compromised if there were a security breach in the application. On the other hand, as soon as you e-mail an Excel spreadsheet outside to anyone outside your corporate intranet, the NSA can see that data too, if they want.
- lucozade 13y agoIt's really not the NSA or GCHQ that most companies are concerned about, at least in this regard. They're concerned that confidential information will be become available to competitors, the public etc. Companies are generally much more afraid of Anonymous and Lulzsec than the NSA. The concern with the NSA, if they have one, is usually the perception that customer/client information that they hold internally being abused.
- kyllo 13y agoRight, anyway the point being that cloud-hosting the spreadsheets resolves the biggest problem with Excel, which is version control, but it introduces a new problem, which is data security on servers that you don't control.
- maxcan 13y agohttp://blog.docmunch.com/blog/2013/re-in-defense-of-excel http://blog.docmunch.com/blog/2013/re-in-defense-of-excel
- mistercow 13y agoIt would have been nice if this article had explained the shortcomings Google Spreadsheets have compared to Excel. As someone who has never actually run into a significant difference in practice, I would have found that interesting.
- leandrod 13y agoGnumeric is so much more pleasurable… and LibreOffice so much more compatible…