7 ms·

I work at a large firm which has shifted itself from using Excel, to significantly promoting data analytics tools such as Alteryx in place of Excel. One of the

I work at a large firm which has shifted itself from using Excel, to significantly promoting data analytics tools such as Alteryx in place of Excel. One of the MAJOR improvements from using Alteryx is that it is easier to audit and review the process. With Excel you can be down the rabbit hole of data manipulation or formula driven outputs and realise you have made a mistake but cannot find where or how. Or cannot find where you went wrong without recreating the steps followed. The workflow style interface of Alteryx and other data analytics tools allows everyone to see the process from start to finish of how you are getting to your outputs. It can also ingest and handle much larger datasets than default Excel. I honestly believe that we will see a large shift from using Excel to using better suited analytics tools. I cannot think of anything I can do in Excel that I can do in a GUI based data analytics tool. Not to write Excel off completely as will certainly still be using it, but to a lesser extent and not where complex data manipulation is required.

vikramkr 6y ago
The problem is that everyone has and knows how to use excel. And enough people in finance and consulting see excel skills as a mark of status that there's going to be some resistance to change there. A new tool has to be equally universal, easy to use for basic purposes while having advanced capabilities, etc. There are courses that teach supply chain management with solver and the like - having people shift over from the only software they were trained to use is not trivial. It was easier for geneticists to change gene names than change excel behavior in the end for a reason - excel is powerful and everywhere
- ecpottinger 6y ago
  Not only have I seen spreadsheet used for function they were never intended for like using it as a text notepad. The main problem since spreadsheets are easy to set up, people create them without asking even the most basic safety functions. Testing the range on inputs or outputs is not done enough, but worse using the wrong functions can give a sheet that looks good but is very wrong.
- NoPicklez 6y ago
  I'll stop you right there when I say that the firm I work at is a global finance and consulting firm and has globally started shifting to using data analytics tools such as Alteryx over Excel. The barrier to entry into Alteryx is nowhere near as difficult as say python or R for finance teams and consultants. It is also seen as a mark of status to master a data analytics tool which improves a particular process either within the company or at a client.
1123581321 6y ago
Alteryx appears to be considerably more expensive, though. Shouldn’t it take a tool of comparable cost to replace Excel? Especially in self-serve use. I’m not disputing the quality of the tool, to be clear. It seems like a good change and a totally reasonable line item. Just questioning what it will take to actually replace Excel in a widespread way. https://www.alteryx.com/products/platform-details/pricing https://www.alteryx.com/products/platform-details/pricing
- iagovar 6y ago
  The answer to that is Knime. Not as polished, but still way better than Excel.
  - leetrout 6y ago
    That is a very interesting tool. I’m always interested in visual programming tooling. Thanks for sharing.
- hermitcrab 6y ago
  My own Easy Data Transform software (https://www.easydatatransform.com https://www.easydatatransform.com) does a fair amount of what Alteryx does and is just $99 (one time fee). So cheaper than Excel!
  - pyromine 6y ago
    How is this better than power query? I know this has a visulaization of the transformations, but that's on the roadmap for PQ.
    - hermitcrab 6y ago
      Apart from the visual UI (which is a big deal, especially if you have lots of transforms): -wider range of input and output file formats supported (xls/xlsx, fixed width, json, xml, csv, tsv, yaml, html, markdown, vcard) -much faster -45 transforms to choose from -written from scratch for the purpose, rather than grafted on to an existing product -doesn't mangle your data like Excel does!
claudeganon 6y ago
Also, Excel is just buggy enough to screw you over anyway, even if you’re trying to use it safely. I can’t count how many times valid formulas have failed on me across various instances of spreadsheets. Or I have to save, reload, or restart just to make things work. I’m constantly amazed by how much it’s used for business critical functions while being so routinely unreliable.
- swiley 6y ago
  Most business people don't value and appreciate correct software. It's odd but shouldn't be surprising that most people are unable to effectively evaluate the risk of a software defect, especially in a tool like excel.
  - craftinator 6y ago
    I don't think it's that odd. Marketing in general is the art of ignoring edge cases, focusing on majority tailoring. Why would I care about losing 1 customer due to changing our branding, if it gains me 1000? This is why business is not considered a form of engineering, while software is (just barely). Imprecision is a tool in business, while in engineering it is an enemy.
Spooky23 6y ago
Getting away from using excel is something you demand that everyone else do.
cosmie 6y ago
Excel has similar capabilities (since Excel 2010) in the form of PowerQuery[1]. It can pull in 3rd party data sources (including custom HTTP calls if you want to reach out to an API), has data types, can both process and store substantially larger data volumes (data is cached/stored separately from the spreadsheet data in a compressed, columnar format and processed by a separate data processing engine), and every step of the data processing pipeline is shown and easily referenced/reviewed (including the ability to leave comments in the code). That said, PowerQuery is a graft onto Excel, originating in the SQL Server world. So "Excel" skills aren't very helpful/transferable; it leverages Microsoft's M language[2] instead of Excel functions, and the usage paradigm is far more natural for someone with a programming background than an Excel background. But as someone else mentioned in another comment, Alteryx is anything but cheap. PowerQuery is already widely distributed and available to anyone with Excel installed[3]. So for anyone looking for this type of operational discipline around data, it's a handy tool to pick up. Bonus points that PowerQuery/M-code is portable between a variety of Microsoft tools (SQL Server, PowerBI, Excel, and a variety of Azure's data services). [1] https://support.microsoft.com/en-us/office/power-query-overview-and-learning-ed614c81-4b00-4291-bd3a-55d80767f81d https://support.microsoft.com/en-us/office/power-query-overv... [2] https://docs.microsoft.com/en-us/powerquery-m/ https://docs.microsoft.com/en-us/powerquery-m/ [3] Traditionally only available for Windows, but the base tech has recently been added to Excel for Mac and is being brought up to parity.
- hermitcrab 6y ago
  I tried doing some joins in Power Query and was surprised how slow it is.
  - cosmie 6y ago
    Join performance is a bit of a hairy topic. If you're pulling data from an external source that supports query folding[1] such as a database, it'll try to push those joins to the original source. If you're pulling in data from a source that doesn't (or the join occurs after query folding is no longer possible), there are a host of performance considerations/optimizations. By default PowerQuery knows nothing about your source data and makes no assumptions. If you're joining data using the GUI, the generated code will likely use the NestedJoin[2] function. A single "step" in the processing will have its ram capped at 256MB[3], so depending on the size of the data you're joining and absent of any data processing steps that would give it guarantees about your data, you may or may not be paging massive amounts of data to disk as it loops through the entire dataset you're joining for each row that is being joined. That said, PowerQuery has 6-7 join algorithms available, and a variety of techniques that can be used to optimize the processing. For example, if you add a primary key to the table (via Table.AddKey or Table.Distinct or Table.Group), it can short-circuit the data processing the moment it finds a match. This[4] article series is really helpful for optimizing joins in particular and the link-outs and tidbits scattered within the series are fantastic for getting a better understanding of PowerQuery's inner workings and performance considerations. [1] https://docs.microsoft.com/en-us/power-query/power-query-folding https://docs.microsoft.com/en-us/power-query/power-query-fol... [2] https://docs.microsoft.com/en-us/powerquery-m/table-nestedjoin https://docs.microsoft.com/en-us/powerquery-m/table-nestedjo... [3] Mentioned halfway down this article: https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-container-size/ https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-co... [4] https://blog.crossjoin.co.uk/2020/05/31/optimising-the-performance-of-power-query-merges-in-power-bi-part-1/ https://blog.crossjoin.co.uk/2020/05/31/optimising-the-perfo...
ourlordcaffeine 6y ago
Same where I work. Excel was replaced completely with R a long time ago. Even automated pdf reports are made using R programs. Yet people are often shocked to hear it at first.
- ekianjo 6y ago
  Same here, moving away from Excel with R and Python for pretty much everything.
ZeroCool2u 6y ago
As someone who has personally had to use Alteryx in multiple projects, fuck Alteryx. What an absolute garbage piece of software. I've already written a rant about it before and I don't have the energy to repeat myself, but do not fool yourself into thinking switching from excel to Alteryx is doing yourself any favors. You're just trading one monster for another. Save yourself a huge amount of money, not to mention your sanity and just take the time to learn some Python, Julia, literally anything else to get the same results faster, more reliably, and not be locked into that noveau Oracle-esque nightmare.
- kfk 6y ago
  Alteryx was my first step into analytics and it helped me get 2 heads funded for my team. But yeah at some point I arrived to your same conclusion and moved my team to python, we never looked back. One thing people don’t realize with these UI tools is that there is no way to use a version control system and I nowadays I cannot think of doing any analytics (including basic BI) without git.
  - hermitcrab 6y ago
    Why can't you version them? Do they all use binary formats?
    - kfk 6y ago
      Alteryx files are xml files but the point is change management is hard to do. You need to compare each single element in the UI to understand what was changed between versions. It's a lot like Excel in that regard.
      - hermitcrab 6y ago
        I guess diff'ing 2 xml files isn't ideal to see what changed. Do you have a better solution in mind?
        6y ago

vikramkr 6y ago

The problem is that everyone has and knows how to use excel. And enough people in finance and consulting see excel skills as a mark of status that there's going to be some resistance to change there. A new tool has to be equally universal, easy to use for basic purposes while having advanced capabilities, etc. There are courses that teach supply chain management with solver and the like - having people shift over from the only software they were trained to use is not trivial. It was easier for geneticists to change gene names than change excel behavior in the end for a reason - excel is powerful and everywhere

ecpottinger 6y ago
Not only have I seen spreadsheet used for function they were never intended for like using it as a text notepad. The main problem since spreadsheets are easy to set up, people create them without asking even the most basic safety functions. Testing the range on inputs or outputs is not done enough, but worse using the wrong functions can give a sheet that looks good but is very wrong.
NoPicklez 6y ago
I'll stop you right there when I say that the firm I work at is a global finance and consulting firm and has globally started shifting to using data analytics tools such as Alteryx over Excel. The barrier to entry into Alteryx is nowhere near as difficult as say python or R for finance teams and consultants. It is also seen as a mark of status to master a data analytics tool which improves a particular process either within the company or at a client.

1123581321 6y ago

Alteryx appears to be considerably more expensive, though. Shouldn’t it take a tool of comparable cost to replace Excel? Especially in self-serve use. I’m not disputing the quality of the tool, to be clear. It seems like a good change and a totally reasonable line item. Just questioning what it will take to actually replace Excel in a widespread way. https://www.alteryx.com/products/platform-details/pricing https://www.alteryx.com/products/platform-details/pricing

iagovar 6y ago
The answer to that is Knime. Not as polished, but still way better than Excel.
- leetrout 6y ago
  That is a very interesting tool. I’m always interested in visual programming tooling. Thanks for sharing.
hermitcrab 6y ago
My own Easy Data Transform software (https://www.easydatatransform.com https://www.easydatatransform.com) does a fair amount of what Alteryx does and is just $99 (one time fee). So cheaper than Excel!
- pyromine 6y ago
  How is this better than power query? I know this has a visulaization of the transformations, but that's on the roadmap for PQ.
  - hermitcrab 6y ago
    Apart from the visual UI (which is a big deal, especially if you have lots of transforms): -wider range of input and output file formats supported (xls/xlsx, fixed width, json, xml, csv, tsv, yaml, html, markdown, vcard) -much faster -45 transforms to choose from -written from scratch for the purpose, rather than grafted on to an existing product -doesn't mangle your data like Excel does!

claudeganon 6y ago

Also, Excel is just buggy enough to screw you over anyway, even if you’re trying to use it safely. I can’t count how many times valid formulas have failed on me across various instances of spreadsheets. Or I have to save, reload, or restart just to make things work. I’m constantly amazed by how much it’s used for business critical functions while being so routinely unreliable.

swiley 6y ago
Most business people don't value and appreciate correct software. It's odd but shouldn't be surprising that most people are unable to effectively evaluate the risk of a software defect, especially in a tool like excel.
- craftinator 6y ago
  I don't think it's that odd. Marketing in general is the art of ignoring edge cases, focusing on majority tailoring. Why would I care about losing 1 customer due to changing our branding, if it gains me 1000? This is why business is not considered a form of engineering, while software is (just barely). Imprecision is a tool in business, while in engineering it is an enemy.

Spooky23 6y ago

Getting away from using excel is something you demand that everyone else do.

cosmie 6y ago

Excel has similar capabilities (since Excel 2010) in the form of PowerQuery[1]. It can pull in 3rd party data sources (including custom HTTP calls if you want to reach out to an API), has data types, can both process and store substantially larger data volumes (data is cached/stored separately from the spreadsheet data in a compressed, columnar format and processed by a separate data processing engine), and every step of the data processing pipeline is shown and easily referenced/reviewed (including the ability to leave comments in the code). That said, PowerQuery is a graft onto Excel, originating in the SQL Server world. So "Excel" skills aren't very helpful/transferable; it leverages Microsoft's M language[2] instead of Excel functions, and the usage paradigm is far more natural for someone with a programming background than an Excel background. But as someone else mentioned in another comment, Alteryx is anything but cheap. PowerQuery is already widely distributed and available to anyone with Excel installed[3]. So for anyone looking for this type of operational discipline around data, it's a handy tool to pick up. Bonus points that PowerQuery/M-code is portable between a variety of Microsoft tools (SQL Server, PowerBI, Excel, and a variety of Azure's data services). [1] https://support.microsoft.com/en-us/office/power-query-overview-and-learning-ed614c81-4b00-4291-bd3a-55d80767f81d https://support.microsoft.com/en-us/office/power-query-overv... [2] https://docs.microsoft.com/en-us/powerquery-m/ https://docs.microsoft.com/en-us/powerquery-m/ [3] Traditionally only available for Windows, but the base tech has recently been added to Excel for Mac and is being brought up to parity.

hermitcrab 6y ago
I tried doing some joins in Power Query and was surprised how slow it is.
- cosmie 6y ago
  Join performance is a bit of a hairy topic. If you're pulling data from an external source that supports query folding[1] such as a database, it'll try to push those joins to the original source. If you're pulling in data from a source that doesn't (or the join occurs after query folding is no longer possible), there are a host of performance considerations/optimizations. By default PowerQuery knows nothing about your source data and makes no assumptions. If you're joining data using the GUI, the generated code will likely use the NestedJoin[2] function. A single "step" in the processing will have its ram capped at 256MB[3], so depending on the size of the data you're joining and absent of any data processing steps that would give it guarantees about your data, you may or may not be paging massive amounts of data to disk as it loops through the entire dataset you're joining for each row that is being joined. That said, PowerQuery has 6-7 join algorithms available, and a variety of techniques that can be used to optimize the processing. For example, if you add a primary key to the table (via Table.AddKey or Table.Distinct or Table.Group), it can short-circuit the data processing the moment it finds a match. This[4] article series is really helpful for optimizing joins in particular and the link-outs and tidbits scattered within the series are fantastic for getting a better understanding of PowerQuery's inner workings and performance considerations. [1] https://docs.microsoft.com/en-us/power-query/power-query-folding https://docs.microsoft.com/en-us/power-query/power-query-fol... [2] https://docs.microsoft.com/en-us/powerquery-m/table-nestedjoin https://docs.microsoft.com/en-us/powerquery-m/table-nestedjo... [3] Mentioned halfway down this article: https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-container-size/ https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-co... [4] https://blog.crossjoin.co.uk/2020/05/31/optimising-the-performance-of-power-query-merges-in-power-bi-part-1/ https://blog.crossjoin.co.uk/2020/05/31/optimising-the-perfo...

ourlordcaffeine 6y ago

Same where I work. Excel was replaced completely with R a long time ago. Even automated pdf reports are made using R programs. Yet people are often shocked to hear it at first.

ekianjo 6y ago
Same here, moving away from Excel with R and Python for pretty much everything.

ZeroCool2u 6y ago

As someone who has personally had to use Alteryx in multiple projects, fuck Alteryx. What an absolute garbage piece of software. I've already written a rant about it before and I don't have the energy to repeat myself, but do not fool yourself into thinking switching from excel to Alteryx is doing yourself any favors. You're just trading one monster for another. Save yourself a huge amount of money, not to mention your sanity and just take the time to learn some Python, Julia, literally anything else to get the same results faster, more reliably, and not be locked into that noveau Oracle-esque nightmare.

kfk 6y ago
Alteryx was my first step into analytics and it helped me get 2 heads funded for my team. But yeah at some point I arrived to your same conclusion and moved my team to python, we never looked back. One thing people don’t realize with these UI tools is that there is no way to use a version control system and I nowadays I cannot think of doing any analytics (including basic BI) without git.
- hermitcrab 6y ago
  Why can't you version them? Do they all use binary formats?
  - kfk 6y ago
    Alteryx files are xml files but the point is change management is hard to do. You need to compare each single element in the UI to understand what was changed between versions. It's a lot like Excel in that regard.
    - hermitcrab 6y ago
      I guess diff'ing 2 xml files isn't ideal to see what changed. Do you have a better solution in mind?
      - 6y ago