9 ms·
I recently sat down and built something more complicated than simple accounting in a spreadsheet. It's what I considered to be a pretty typical usecase for a no
by adepressedthrow 4y ago
I recently sat down and built something more complicated than simple accounting in a spreadsheet. It's what I considered to be a pretty typical usecase for a non-math related sheet; taking in several tables of data and selectively joining them. You enter an ID, press a button, and it finds all of the data related to that ID and presents it to you.
I was horrified to find that even with the supporting scripting capabilities, the entire paradigm revolves around knowing the shape of your data in advance. (I was using Google Sheets, but I don't think Excel would have been much different). For example, it is very non-intuitive to write a formula that retrieves all the rows in another sheet that match this rule, and once you do that, since it's a variable number of rows returned, it is difficult to then operate on that data without filling your formulas down for some indeterminate number of rows.
I realize most people don't have the luxury or skills, but I quickly realized that I could spin up a whole CRUD webapp for this problem faster than I, someone who understands indexing and windowing and such, could build it in a spreadsheet.
After this experience, I can't help but wonder if Excel and spreadsheets largely exist due to pre-existing knowledge about how to use them, or if this is _actually_ the best way for non-programming minded people to solve these problems.
- rufus_foreman 4y agoSomeone who has extensive experience in Excel but only basic knowledge in software development will quickly realize that they can develop something in Excel faster than they can build it in a CRUD app.
- adepressedthrow 4y agoI certainly get that, but I'm primarily pointing out that as a non-layman from the software side, it doesn't seem like a particularly amazing tool. It certainly could be the case that it's extremely good for non-programmers, I was simply pointing out that I naively think it's not very well designed for those usecases.
- analog31 4y agoI think this raises another point. An apples-to-apples comparison is only possible if you can do both yourself. If you're not a coder, then you're comparing creating a spreadsheet with managing coding. And the latter is even more difficult to learn than coding itself. I'm in the middle ground -- can code until the cows come home, but can't manage a coding project to save my life. I am extremely sympathetic when someone has to manage me coding. I'm always thinking to myself: How can I avoid turning this into a nightmare for them?
- Dyac 4y agoI'm a happy customer of https://exploratory.io/ https://exploratory.io/ - it's a very user-friendly interface on top of R and I think you might find it helpful.
- eastbound 4y agoMany people have tried launching an Excel-with-SQL-querying product, but it’s extremely hard to do the UI well. Also products where people write SQL are impossible to insure.
- Jtsummers 4y agoGoogle Sheets is pretty basic compared to Excel in terms of the kinds of data analysis and queries it permits (without dropping into another language). Excel added tables over a decade ago that allow for some very useful and much cleaner query and data analysis stuffs in straight Excel. Then there are power queries and pivot tables, not sure how long those two have been around but last I used Google Sheets it had nothing like either. My point being, don't judge spreadsheets by Google Sheets. Actually use Excel and you'll see a much more capable system and get a better understanding of why people (particularly non-programmers in business settings) stick with it. EDIT: Pivot tables are in Google Sheets, so either I missed them before or they were added after I last gave it a serious look. My google-fu is not discovering the date they were added.
- dima_vm 4y agoNot sure what you mean by "power queries", but Google Sheets support SQL queries. Would be easier to see on an example.
- _dain_ 4y agoPowerQuery. It's a tool built into Excel. It's a GUI that wraps an almost purely-functional DSL designed for ETL and data munging, called the M language. You can either use the GUI or write the code directly. It has first class functions and closures and normies are programming in it. It's great. More people should know about it. Btw it's kind of funny seeing so many HN users, many of whom must be working on software that competes with Excel either directly or indirectly, who are so unknowing of the full capabilities of Excel, capabilities that are the bread and butter of any e.g. financial analyst, or logistics manager, or any smart non-programmer white collar worker. Maybe this "hacker repulsion field" is the secret of its dominance -- you can't compete with it if you never learn what it can do.
- pete_nic 4y agoIn this context I’m interpreting DSL to mean “domain-specific language”
- 4y ago
- apancik 4y agoExcel has some functionality that Google Sheets are missing that is used in this particular use case. More specifically, it has a primitive called Table. After you set up your data as Tables, you can then reference whole columns as Table1[Column1] and it also fills your formulas down as you add more rows. I don't want to defend Excel too much, as it is not ideal in many ways. Nevertheless, over time, I found myself using it more and more to prototype and visualize data. With magic features like Pivot charts, Flash fill, and Data tables you can hammer out a one-off "app" in a matter of minutes.
- adepressedthrow 4y agoInteresting. It definitely seems like that would fix a lot of my issues, which prompts the question of why Google hasn't built this functionality
- ghaff 4y agoGoogle has, for better or worse (and I'd argue usually for better), created a 90-95% product across Word/Sheets/Slides. Every now and then I run into limitations but the sparseness is mostly a win. That said, every now and then I run into a limitation (perhaps especially with Excel) that I have to either work around or use the Microsoft product.
- smt88 4y agoThe problems you mentioned are solved by Excel using tables, which are named, variable in size, and can be referenced by column names instead of addresses. Tables can also be joined and queried using Power Query. Excel is still 100x more powerful and sophisticated than Sheets is.
- occamrazor 4y agoI don’t know about GSheets, but Excel has dynamic arrays (of variable lengths) since at least two years ago. Or a couple of lines of VBA can accomplish the same.
- _dain_ 4y agoCan you give a more detailed description of what you were trying? I can't know for sure, but it sounds like Excel can easily handle what you've described, if you use it right. >the entire paradigm revolves around knowing the shape of your data in advance. How exactly do you program without knowing the shape of your data in advance? You need to know your database columns, or your JSON schema, etc. >(I was using Google Sheets, but I don't think Excel would have been much different). It would have been very different, because Excel has tables and Powerquery and Google Sheets doesn't. >since it's a variable number of rows returned, it is difficult to then operate on that data without filling your formulas down for some indeterminate number of rows. Were you using dynamic array formulae? They can handle the old problem of needing to fill down formulae to an arbitrary depth. Or again, tables. Programmers routinely underestimate Excel. Unlike most Microsoft products, it has improved year on year over the past few decades. There are heaps of great power-user features they keep introducing. The skill ceiling is very high .. not as high as proper software engineering, but still damned high. It also really annoys me when I see Linux/FOSS partisans tell Windows normies "oh you can do everything you can do in Excel in LibreOffice Calc" -- no you fucking well cannot. (And I use Linux on my personal computers full time).
- adepressedthrow 4y agoYour comment about FOSS is spot on. While I'm very aware that Google Sheets is not OSS, it felt much more amenable to me than Excel (and I'm sure Excel's online free version isn't particularly fantastic anyway, though it may be better than Sheets from what people are saying here). > How exactly do you program without knowing the shape of your data in advance? You need to know your database columns, or your JSON schema, etc. This was a bit overloaded in my opinion, as in spreadsheets world, "shape" includes the number of rows, hence my comments. I know that the column layout needs to be known. > Were you using dynamic array formulae I looked into it, but couldn't figure out how to handle them without introducing a massive amount of formula duplication. The best I could figure out how to do was to do a single large FILTER (which is dynamic array) and doing a fill down on my other transformation formulas from there. I blacked out the rows past the end of the FILTER using conditional formatting rules (which felt very stupid to do, but I couldn't find anything better). > The skill ceiling is very high I don't doubt you, but if you can't discover the functionality, it might as well not exist. Admittedly I was clearly using the inferior tool, but in my searching for solutions I much more readily found Google's documentation over Excel's. I also realize I'm not in the position of being forced into a corner; as most of us on this forum could, I just wave my magic wand and write the software to solve my problems. I imagine those who don't have that ability available to them will do "crazier and crazier" things to figure out how to accomplish their work in Excel, and therefore will learn much better ways than I have in my little experience with it. ---- I was building a tool to track the completion of finding parts for a given Lego set. You enter the set ID, it pulls the parts list for that set (Rebrickable nicely offers their database as a set of CSVs https://rebrickable.com/downloads/ https://rebrickable.com/downloads/) and formats it nicely for consumption.
- ab_testing 4y ago> but I quickly realized that I could spin up a whole CRUD webapp for this problem faster than I, someone who understands indexing and windowing and such, could build it in a spreadsheet. What would you use to build a CRUD app faster than an excel app
- solardev 4y ago(Not the OP, but) A headless CMS is pretty great. Or Airtable.
- jerry1979 4y ago>I was horrified to find that even with the supporting scripting capabilities, the entire paradigm revolves around knowing the shape of your data in advance. Just record yourself finding the bottom of the data set (Ctrl + down arrow), then take a moment to make the code work in relative terms instead of absolute terms.
- adepressedthrow 4y agoWhat do you mean by this? What am I "recording" as the bottom of the data set? My point was that it is very hard to have a dynamic number of rows feed a proportionate dynamic number of rows. Scripting makes it much simpler, but at least with Google Sheet's scripting, the API seemed pretty lacking for that processing (in the very least, it's very slow, since it's running as a very constrained shared resource).
- spywaregorilla 4y agoExcel let's you create macros by hitting "record" and then doing the operations yourself. It'll generate the code to replicate the impact of your inputs.
- dgudkov 4y agoYour task could be trivially done with EasyMorph (https://easymorph.com https://easymorph.com). We've designed it exactly for such use cases. Data needs of non-technical people have long been neglected. It was believed that any data wrangling should be done by IT people. So all non-technical people had was Excel. Luckily, the no-code movement finally started addressing that issue with a varying degree of success.
- jsmith99 4y ago> it is very non intuitive to write a formula that retrieves all the rows in another sheet that match this rule You can retrieve an entire range of data with a single formula in either excel or Google sheets. The formula is caller FILTER https://support.microsoft.com/en-us/office/filter-function-f4f7cb66-82eb-4767-8f7c-4877ad80c759 https://support.microsoft.com/en-us/office/filter-function-f...
- adepressedthrow 4y agoThat's exactly what I did. Now separate out some columns and perform some additional transformation on that FILTERed data. Can you do it without repeating yourself (duplicating the FILTER statement, or any of the other transformations you need to do, besides just filling down a column). Can you perform these transformations only on the row height of the data, and not have extra rows with broken formulas? I honestly wouldn't even be surprised if the functionality to do the above does exist, but for all of my searching I couldn't find it.
- LasEspuelas 4y agoThis is something I faced in both Excel and GSheets (when working with formulae only). That is the need to repeat myself occasionally.
- deburo 4y agoI will ramble a bit, but for complex data transforms, you can use PowerQuery instead of formulas. For semi-serious programmers like myself, I started doing C# add-ins to Excel using Excel-Dna. There’s also Query Storm. :)
- winphone1974 4y ago"Several tables and selectively joining them" ... "Enter an id and filter" Sure sounds like your creating a relational database in a spreadsheet, which is possible but not really the intended purpose?
- adepressedthrow 4y agoSurely that's what lots of non-developer white collar workers use Excel for? I imagine there's orders of magnitude more people using Excel for data processing rather than Python or R. I'm well aware it's not the best tool for the job, but yet people are using it for purposes such as that. I wanted to learn more about that experience.
- spywaregorilla 4y agoI think no. Most people don't do table joins very often. They wouldn't know how. What they'll do instead is create lookups with VLOOKUP or INDEX(MATCH()) to pull in values from other tables into their one master. And once they have the master flat file they'll use a pivot table for group by aggregations.
- Ruthalas 4y agoThis is very accurate in my experience, down to the suggested formulas.
- collegeburner 4y agoas somebody who is very experienced with both writing crud apps and excel there's still a lot of cases where excel wins. a big one for me is making m&a models, stuff like that, excel is better bc you can organize view and move around data, re run calculations, whatever else in a tighter loop. and it's better at taking data in kinda inconsistent formats. there's a reason it is still king of IB, PE, all the finance world. this is because excel was "low/no code" before it was a tech meme with vc money.
- Jorge1o1 4y agoExactly right about the seeing the data and rerunning calculations. The programmers who look down their nose at Excel are doing the exact same thing in Jupyter Notebooks and in their REPLs. >I can see the state of any given variable at any time >I can rerun the same function on different inputs, or different functions on the same inputs >All without having to restart my program! Remind you of anything? Anybody who does print(df.head()) is pining for Excel…
- potatoicecoffee 4y agoHave you heard of microsoft Access?
- themadturk 4y agoThis stings. I work as one of the less-technical people in group of developers and maintain a fairly extensive flat-file database in Excel. I wanted to re-implement it in Access, but my boss said "No," because he was afraid if I left he wouldn't be able to find anyone to maintain it. Also, Access isn't available with all Microsoft Office licenses. Excel is.
- dav43 4y agoDid you use “Query”?
- bergenty 4y agoI don’t all that much about excel but I remember being blown away as a 17 year old kid at my first job when someone showed me pivot tables.
- rexreed 4y agoIt's much simpler than all that. Excel is just easy to use and quick to get results and powerful enough to extend, easy to share with others, and is a broadly accepted file format. That's why it works. Everything else is more complicated, more vendor-locked in and takes longer to get results or requires more technical skills than the average Excel user has. Is it the best for everything? No. But it's damn good enough for a LOT of things. Its survival is proof.
- jsemrau 4y agoI work in financial services and have build prototypes with Excel that are now multi-million dollar revenue providers (implemented properly). Excel is powerful in this respect because it is a shared experience. Build something with SAS/R/Python, explaining the results is possible but getting buy-in from other teams is harder.
- otikik 4y ago> the entire paradigm revolves around knowing the shape of your data in advance. This is true, but you can improve things significantly by using named ranges. Using `$PAYMENTS` instead of `Sheet2!$B2:$B$21` for your column data and `$TAX_RATE` instead of `$Sheet4!$A$1` clarifies things quite a lot. You still will need to know how your data is structured (e.g. this is a column that goes down, this is a fixed variable) but it is way more readable.