10 ms·
Big big reason: R doesn't alter data without being asked to, Excel does so, silently, sneakily. Anything important should not be done in Excel. https://www.the
by corty 5y ago
Big big reason: R doesn't alter data without being asked to, Excel does so, silently, sneakily. Anything important should not be done in Excel.
https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
https://stackoverflow.com/questions/165042/stop-excel-from-automatically-converting-certain-text-values-to-dates https://stackoverflow.com/questions/165042/stop-excel-from-a...
- bradleyjg 5y ago> Anything important should not be done in Excel. Hold on a second while I go shut down the global economy for two years so we can teach everyone finance person how to program.
- shakow 5y agoWould not necessarily be a bad idea. https://theconversation.com/economists-an-excel-error-and-the-misguided-push-for-austerity-13584 https://theconversation.com/economists-an-excel-error-and-th... Not that this is Excel's fault, but researchers should definitely either seriously learn how to use computer stuff, or just don't. A person can't drive on the highway without a license, the same rigor should be applied here, especially in academic circles.
- manishsharan 5y agoThis. Most finance people who live by Excel have ninja level skills with Excel. I have had to unpack logic from some excel sheets into webapps and they had so much sophisticated programming embedded into it. And it is so much easier intuitive to use Solver for linear equations in Excel than any web application. Regarding the graphing ability , R may have more power but plotting the graphs in Excel is so much more WYSIWYG.
- belter 5y agoDo not shutdown the global economy. Just maybe move to tools with less issues around process governance: "The 7 Biggest Excel Mistakes of All Time" https://www.teampay.co/insights/biggest-excel-mistakes-of-all-time/ https://www.teampay.co/insights/biggest-excel-mistakes-of-al... "The financial fails and business risks of spreadsheets" https://www.webexpenses.com/2020/10/financial-fails-business-risks-spreadsheets/ https://www.webexpenses.com/2020/10/financial-fails-business... "Nightmare on spreadsheet: take Excel use seriously" https://www.icaew.com/insights/viewpoints-on-the-news/2020/oct-2020/nightmare-on-spreadsheet-take-excel-use-seriously https://www.icaew.com/insights/viewpoints-on-the-news/2020/o... "Excel – The Dirty Secret" https://tax.thomsonreuters.co.uk/blog/excel-the-dirty-secret/ https://tax.thomsonreuters.co.uk/blog/excel-the-dirty-secret... "8 Challenges When Using Excel For Accounting" https://www.senacea.co.uk/post/excel-for-accounting-challenges https://www.senacea.co.uk/post/excel-for-accounting-challeng...
- eyeball 5y agoAs though mistakes wouldn’t happen with non-programmers working in R.
- corty 5y agoIn R, you can use accepted methods of software development and engineering, such as version control, automated testing, CI/CD to catch mistakes and prevent them from causing harm. In Excel? Not really.
- eyeball 5y agoDo you think the average finance department employee is going to do that?
- corty 5y agoEspecially those departments should do so and learn to do it. Financial departments are always the ones complaining about the cost of mistakes, compliance risks (of the monetary kind) and doing due diligence. Preventing the production of wrong figures in financial excel sheets ticks all those boxes. They should bloody well be doing it, but I'm also not holding my breath ;)
- deleted 5y ago[deleted]
- kenjackson 5y agoAnd there are massive bugs in programs that use these tools like the Linux kernel still.
- tomlue 5y agofinance people who learn to program will outperform those who don't.
- bradleyjg 5y agoI believe that’s true. There still aren’t enough to go around and won’t be for a very long time. Excel is the closest we’ve come as an industry to building a tool that enables non-programmers to program. If it disappeared tomorrow, as some arrogant posters apparently wish it would, tremendous value would be destroyed. Not just in terms of existing workflows but in terms of new workflows that would not be done in R but instead would be done by hand or not at all.
- Nicholas_C 5y agoI’m a hobbyist programmer who has worked in various finance jobs and is now working in investment banking. I would love to use programming in finance but I have to say it’s come in handy very little.
- Jorge1o1 5y agoWell, prolly not in IBD. But in Sales and Trading or at a lot of quant funds it is very common.
- belter 5y ago"Floating-point arithmetic may give inaccurate results in Excel" https://docs.microsoft.com/en-us/office/troubleshoot/excel/floating-point-arithmetic-inaccurate-result https://docs.microsoft.com/en-us/office/troubleshoot/excel/f... "Align numerical precision Excel 2013 and R" https://stackoverflow.com/questions/39531655/align-numerical-precision-excel-2013-and-r https://stackoverflow.com/questions/39531655/align-numerical... "Numeric precision in Microsoft Excel" https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft...
- FabHK 5y agoAlmost all of these are issues with floating point in general, I'd say. Some are with naive implementations of Excel functions, fair enough (for example variance as (sum x_i^2/n - (sum x_i/n)^2) that are indeed better implemented in R or Julia. IEEE 754 just has unintuitive properties. Kahan (the "father of IEEE 754") has a rant (among many others) about Excel as well, and how it tries to hide some of the floating point complexities more or less successfully: Floating-Point Arithmetic Besieged by “Business Decisions” https://people.eecs.berkeley.edu/~wkahan/ARITH_17.pdf https://people.eecs.berkeley.edu/~wkahan/ARITH_17.pdf
- belter 5y agoBut you also have issues like 'Catastrophic cancellation': https://carolomeetsbarolo.wordpress.com/2012/07/20/catastrophic-cancellation-in-action/ https://carolomeetsbarolo.wordpress.com/2012/07/20/catastrop... Or: "OOPS XL Did It Again" https://carolomeetsbarolo.wordpress.com/2014/06/22/oops-xl-did-it-again/ https://carolomeetsbarolo.wordpress.com/2014/06/22/oops-xl-d... From the Wikipedia article: "Although Excel can display 30 decimal places, its precision for a specified number is confined to 15 significant figures, and calculations may have an accuracy that is even less due to five issues: round off,truncation, and binary storage, accumulation of the deviations of the operands in calculations, and worst: cancellation at subtractions resp. 'Catastrophic cancellation' at subtraction of values with similar magnitude."
- 5y ago
- tssva 5y agoExcel doesn't alter data without being asked either. What it does do by default is try to change how the data is displayed. If the column is properly set to the correct data type the data will be shown correctly because Excel doesn't change the actual data. Column types can be set when csv files are imported. The problem described in the article isn't an Excel issue. It is an issue of the geneticist failure to learn the basics of how their tools work.
- tsbinz 5y ago> Excel doesn't alter data without being asked either. What it does do by default is try to change how the data is displayed. This is easily falsifiable. In a "general" cell, when I enter 0002, it gets changed into 2, not just in display, but in actual content. When I change the cell type to text, it'll still be 2. Only if I enter 0002 after changing the type to text is the content kept. Similar when I want to have the text 3/17 or SEPT1 in a cell, I have to format it before typing or the data does get altered. If you try setting it to "text" after you typed it, you'll get some number that's not very useful to you.
- formercoder 5y agoJust type ‘0002.
- tssva 5y agoIf you enter it prior to specifying the formatting you are entering it as a date. The number you see when you change to text is the numeric value reflecting the date/data you entered.
- undreren 5y agoIt also interprets and coerces values in formulas. Empty cells are interpreted as zero, which can be downright catastrophic, if the data is just missing.
- dendriti 5y agothe only way to prevent Excel from converting date-like number values into 'real dates' is to format the number values as text. or escape each and every number value so this doesn't occur. this is not just a workaround - it's recommended by Microsoft. because you literally cannot turn this functionality off. how is that not an "Excel issue"?
- user3939382 5y agoSo if I have Zip codes with leading 0’s, use R?
- jasode 5y ago>So if I have Zip codes with leading 0’s, use R? Several other software tools also mess up leading 0s including R if used in a naive way without specifying extra options. My previous comment about this: https://news.ycombinator.com/item?id=25017116 https://news.ycombinator.com/item?id=25017116 Like R, MS Excel can also preserve leading 0s -- if you specify the option on import. (Click on Excel 2019 Data tab and import via "From Text/CSV" button on the ribbon menu and a dialog pops up that provides option "Do not detect data types" (Earlier version of Excel has different verbiage to interpret numbers as text))
- user3939382 5y agoThe problem that I've had with this issue and Excel is that Excel performs overly-clever-automagic type conversions in certain situations where you don't want them. And then especially when you're collaborating on files with non-technical normal office workers, they're not expecting this behavior. So the end result has been that this issue has bitten me many times, despite knowing how to ideally tell Excel to handle it.
- shawnz 5y agoIf you have zip codes, don't store those in a numeric type (doesn't matter if it's Excel, R, or any other system) Zip codes aren't numbers, they are strings that happen to contain only numeric characters
- TeMPOraL 5y agoWAIT. All these words on SO and here have been written because people don't know to change the column type from "General" to "Text"?
- Linosaurus 5y agoThey would like to be able to just open a csv file in excel without going through a file import wizard. Changing the column type after the fact is too late.
- roenxi 5y agoWhile that is a very real problem, it isn't totally persuasive. It is just as easy to create traps in R (I lost days of my youth to R's challenging-to-defend jungle of an almost-type-system). It is even harder to debug than in Excel too - I would expect less bugs are found in R scripts because less people are up to the challenge of trying. People shouldn't do important work in Excel. If it is important, people should be involved who have invested the time in learning something more powerful. Indeed, we could ask they aspire all the way to good practice and store their data in a database and their code in git. But there needs to be a process to verify model correctness no matter what tool is being used and bugs will exist in R as well as in Excel.
- corty 5y agoI agree that R isn't ideal, and a proper programming language is preferable to both. Which one of course will depend on the domain, but proper control over float precision, fixed-point arithmetics, bignum and debuggability isn't good in R either. It's just that R is still far above Excel.
- fighterpilot 5y agoI mostly agree, but at least Excel is more easily inspectable by a human, which makes a certain class of errors (specifically, silly logic bugs) less likely to get through.
- jrochkind1 5y agoIt's not obvious to me that Excel is more easibly inspectable, can you say more? To me, R seems more easily inspectable, as all the logic of a program is visible just by looking at text files, where in Excel it's hidden "under the surface", you have to click on cells, look at what's there, go click on other cells that relate to it, remember what you were looking at in the first one that's now invisible, etc.
- fighterpilot 5y ago
- temp8964 5y agoActually when you use R functions to import CSV files, they also guess data types and make some conversions by default. The difference is that in R you can turn them off if you want to. But in Excel there are automatic changes you can't turn off.
- capnahab 5y agoAs a total R beginner I was hugely put off by the complexities of data cleansing especially columns with ?different data types.
- HPsquared 5y agoYou can set the data type to "text" when importing CSVs, that turns most things off.
- corty 5y agoMost, but not all. That is the entire point: Excel is never safe in any configuration.
- FabHK 5y agoNot sure about that. It is my understanding that Excel only tries to guess the type of an input if it has not been explicitly specified by the user. The problem with "things that look like dates being interpreted as dates" comes from not specifying that a column has type "text".
- Xelbair 5y agoEven if you do specify a column as a text excel WILL assume that it contains dates sometimes. Happened to me multiple times, even when i set each cell as a text. Seems like copypasting a tab separated values resets the cells to their default state.
- crazygringo 5y agoIf "not all", can you provide an example of what gets converted if you specify text? Just want to know to avoid future gotchas.
- WastingMyTime89 5y agoThere is nothing sneaky about it. Excel imports things it thinks are dates as dates. It's not a silent modification. It's a loading issue. Properly loading data into R is not particularly easy either.
- washadjeffmad 5y agoWe never got the results of our covid tests from early in the pandemic because someone imported the medical data into Excel, which predictably and destructively modified the patient IDs by stripping all leading zeroes. I've used perl, python, and R for scientific data for a really long time but have always made a concerted effort to avoid Excel. My reasoning feels the same as when people say Java is the best language because it can be run on any device, which is like saying anal sex is the best sex because you can do it with any animal. Maybe I'm missing out on a great experience, but the notion has always made me uncomfortable.
- gunshai 5y ago> My reasoning feels the same as when people say Java is the best language because it can be run on any device, which is like saying anal sex is the best sex because you can do it with any animal. Please tell me you have said this to someone in a work meeting. This is hilarious.
- Akronymus 5y agoEven if it wasn't used, I may end up using it at work myself.
- globular-toast 5y agoI haven't used Excel since I was a child and learnt to program. I didn't realise just how bad it was. Someone asked for a list of ids for some inventory and they imported it into Excel. They were all 4 digit alphanumeric but the ones that looked like numbers, like "0001" it just treated it as a number, showed it as 1 no option to disable this ridiculous behaviour.