3 ms·

Yes. Yes. Yes! Omg. Yes. The worst I have seen was a team of several hundred people who, on $200 half-broken laptops, were running - in parallel - 500MB Excel

by crdb 11y ago

Yes. Yes. Yes! Omg. Yes. The worst I have seen was a team of several hundred people who, on $200 half-broken laptops, were running - in parallel - 500MB Excel files for hours every day, which basically did a few linear operations on about 30 variables, before joining the results in a superfile itself taking up to 20 minutes just to open on a gaming-specced desktop. In another case, I broke down an Excel model with over 50 tabs and discovered that there were only 2 input variables. Unfortunately the CFO of that business decided to go for the ostrich strategy (after all the monster model impressed investors). Every business I have ever worked in had some convoluted, cumbersome, buggy, opaque, human-dependent Excel processes. My co-founder and I finance our startup replacing these with tight scripts running on AWS (amongst other things) - it's low hanging fruit. We approach these things as a black box, figure out desired behaviour, and rebuild from scratch. Excel wins all the time because it's a UI that managers are familiar with, and they are often either the consumers, or want to audit your data processes. It's got a LOT of tooling that lets non-technical users do things they shouldn't. If you thought Python Pandas was an improvement, try learning about the relational model and how to use a decent relational database (I recommend PostgreSQL, whose error messages will teach you a few things and which has a saner type system where you CAN compare two different integer types without getting NULL). You'd be amazed how far you can go with just SQL. You should also do so from a UNIX OS, and from the command line (using pg, editing a file then running \f blah.sql). It will take you a few hours of pain but will be worth it in added productivity and transparency over any point and click alternatives. What you're touching on is the power of abstraction, functional vs imperative programming, and particularly how proving programs can be enormously productive. SQL is powerful for this because it is declarative and very close to mathematics (Codd brought set theory to databases). The next steps are category and type theory and then the Haskell/Mercury/Idris/Coq rabbit hole... an unpopular one because, as someone once told me, "businesses are resilient to application errors"...

Nicholas_C 11y ago
>The worst I have seen was a team of several hundred people who, on $200 half-broken laptops, were running - in parallel - 500MB Excel files for hours every day, which basically did a few linear operations on about 30 variables, before joining the results in a superfile itself taking up to 20 minutes just to open on a gaming-specced desktop. This is my nightmare. I work on a sharedrive with large Excel files that conk out Excel. We are working on building a tool to do this work but it is slow going.
- jfjeschke 11y ago
  What does the tool do exactly?
  - Nicholas_C 11y ago
    It's a custom developed planning/forecasting tool. It's a P x Q thing, but the P and Q are variable as to the type and mix of the Ps and Qs and need to be combined/sliced into many different ways for analysis/planning/forecasting. The team actually developing it has been very challenged.
    - crdb 11y ago
      You're not really challenged until the growth rate of their Excel monster outpaces the rate at which you automate the existing stuff... thankfully Excel has a 1 million row limit, and it always takes them time to figure out how to shard an Excel database :P
      - Nicholas_C 11y ago
        Well in this case it's actually mine and the rest of the finance group's Excel monsters (plural) that are being automated. The devs are doing yeoman's work.
- crdb 11y ago
  I'm sure there's automated ways of doing it within the Microsoft ecosystem... it's just not a specialty I want to acquire!
viraptor 11y ago
Oooh! Horror story time. There's a power plant which used an excel spreadsheet connected to SQL server stored procedures (live connect? active connect? can't remember) to calculate and show cooling conditions and active states of the generators. With lots of macros written by a teenager who didn't even understand the SQL part properly at the time. (don't judge me, they paid and I had no idea how bad that project was...) Fortunately, if I remember correctly it was just read-only for monitoring. If it wasn't, I'm sorry... so so sorry.
- crdb 11y ago
  Wait... a POWER PLANT?! I undertand relying on piles of crappy Excel models built by underpaid, skill-lacking interns in cash-strapped e-commerce companies where investors don't really care about the real profit margin, but... I mean, we're talking about millions of dollars of hardware and human lives here...
  - viraptor 11y ago
    Yup. A power plant. It wasn't an official, approved project BTW. Just a "this would make our lives so much easier, you know excel, right?" kind of deal. I'm more worried about IT culture which creates a need for such secret side projects really. That this existed and someone potentially made decisions based on that spreadsheet (no testing happened, I got the formulas and test data, but no results for comparison) is just a reflection of the priorities in that place.
    - crdb 11y ago
      Oh, that's always the case unfortunately. Doing things "properly" means you can't do them "tomorrow" or even better, "yesterday". It's funny to hear the same story everywhere - including on something as critical as a power plant, and including the "secret" part. That several-hundred-person Excel nightmare? It had grown from "quick and dirty" shortcuts from middle managers, none of which were still around, none of which did the work (that's why you have unpaid interns), designed to bypass the official data warehouse project. They got promoted because they delivered the processes, and nobody checked the numbers (that's another "feature" of Excel shops - "if it looks like it works, it works"). The problem is that it's hard to sell "your numbers are off" to new clients. I toyed with the idea of making it conditional: "if your profit margin is more than 5% off, you pay us"...

by crdb 11y ago

Nicholas_C 11y ago

>The worst I have seen was a team of several hundred people who, on $200 half-broken laptops, were running - in parallel - 500MB Excel files for hours every day, which basically did a few linear operations on about 30 variables, before joining the results in a superfile itself taking up to 20 minutes just to open on a gaming-specced desktop. This is my nightmare. I work on a sharedrive with large Excel files that conk out Excel. We are working on building a tool to do this work but it is slow going.

jfjeschke 11y ago
What does the tool do exactly?
- Nicholas_C 11y ago
  It's a custom developed planning/forecasting tool. It's a P x Q thing, but the P and Q are variable as to the type and mix of the Ps and Qs and need to be combined/sliced into many different ways for analysis/planning/forecasting. The team actually developing it has been very challenged.
  - crdb 11y ago
    You're not really challenged until the growth rate of their Excel monster outpaces the rate at which you automate the existing stuff... thankfully Excel has a 1 million row limit, and it always takes them time to figure out how to shard an Excel database :P
    - Nicholas_C 11y ago
      Well in this case it's actually mine and the rest of the finance group's Excel monsters (plural) that are being automated. The devs are doing yeoman's work.
crdb 11y ago
I'm sure there's automated ways of doing it within the Microsoft ecosystem... it's just not a specialty I want to acquire!

viraptor 11y ago

Oooh! Horror story time. There's a power plant which used an excel spreadsheet connected to SQL server stored procedures (live connect? active connect? can't remember) to calculate and show cooling conditions and active states of the generators. With lots of macros written by a teenager who didn't even understand the SQL part properly at the time. (don't judge me, they paid and I had no idea how bad that project was...) Fortunately, if I remember correctly it was just read-only for monitoring. If it wasn't, I'm sorry... so so sorry.

crdb 11y ago
Wait... a POWER PLANT?! I undertand relying on piles of crappy Excel models built by underpaid, skill-lacking interns in cash-strapped e-commerce companies where investors don't really care about the real profit margin, but... I mean, we're talking about millions of dollars of hardware and human lives here...
- viraptor 11y ago
  Yup. A power plant. It wasn't an official, approved project BTW. Just a "this would make our lives so much easier, you know excel, right?" kind of deal. I'm more worried about IT culture which creates a need for such secret side projects really. That this existed and someone potentially made decisions based on that spreadsheet (no testing happened, I got the formulas and test data, but no results for comparison) is just a reflection of the priorities in that place.
  - crdb 11y ago
    Oh, that's always the case unfortunately. Doing things "properly" means you can't do them "tomorrow" or even better, "yesterday". It's funny to hear the same story everywhere - including on something as critical as a power plant, and including the "secret" part. That several-hundred-person Excel nightmare? It had grown from "quick and dirty" shortcuts from middle managers, none of which were still around, none of which did the work (that's why you have unpaid interns), designed to bypass the official data warehouse project. They got promoted because they delivered the processes, and nobody checked the numbers (that's another "feature" of Excel shops - "if it looks like it works, it works"). The problem is that it's hard to sell "your numbers are off" to new clients. I toyed with the idea of making it conditional: "if your profit margin is more than 5% off, you pay us"...