6 ms·

There's many situations where manually dealing with 100+ records is a decent tradeoff. I feel people in our field overestimate the actual complexity of many dum

There's many situations where manually dealing with 100+ records is a decent tradeoff. I feel people in our field overestimate the actual complexity of many dumb looking tasks. From your bank employee example, it looks like your solution would be to open the bank's internal tool to her excel instance, have it somewhat find the right records and inject the right value, while checking the listed codes. That looks like a series of APIs if they care about data exposure, or huge CSV exports that they need to then track ? And as it seems to be important enough to warrant verification, it means your program is not some random batch, it needs significant validation. Then picture the internal tool changing its output format. Reports getting categorized differently because of organizational change. New fields being added to follow new legislations. The tool could break at any of these changes, and the lady's left with a broken system except a lot of money has been invested, it's supposed to have solved her problem, she might not have the same access to data she had before the integration, and overall the situation could be worse than before. That feels like a bleak scenario, but it happens all the time as well.

jiggawatts 3y ago
It’s basically a JOIN, the most fundamental data manipulation operator. If you can’t automate this, then you don’t “get” computers at all. I once had two guys arguing next to me about how it was impossible to produce a simple report about VM storage utilisation in under a month. I produced the report, printed it out, and shoved it in their hands just to shut them up while they were still busy arguing. A “month” of work required five joins.
- radlad 3y ago
  > It's basically a JOIN The comment you're replying just explained why this is not "basically a JOIN." There are two unrelated systems (not SQL databases) which don't expose any APIs to the analyst.
  - tuatoru 3y ago
    It's still basically a join. If it's on a computer, it can be converted to text files, and then it can be JOINed, one way or another.
    - deleted 3y ago
      [deleted]
    - ethbr1 3y ago
      My sibling from another mother. That's exactly what we did.
    - csomar 3y ago
      In these cases, the work is usually about connecting the systems together than the data modification itself.
      - TeMPOraL 3y ago
        No, that's just made up work that's not required to solve a problem. Systems are already connected enough by virtue of being accessible by the same person on the same computer at the same time. Turning this into software project and building Proper Integration is only creating a tight coupling between the systems, which will only keep creating bugs and requires ongoing maintenance.
  - c0pium 3y ago
    [flagged]
    - nicbou 3y ago
      This is a pointlessly mean and unpleasant comment. It was polite discussion up until that point.
ethbr1 3y ago
We were able to fix it up with some automation in a couple days. Critically, still running visually on her machine, and with internal resources for lifecycle support if the process changes.
- radlad 3y ago
  Finding internal resources to fix the tool when things change is very prescient. Growing up, my mom performed a job where she would essentially copy car crash data from one system into another, row by row. As a teenager learning to program, it was obvious to me this could be easily automated. She (perhaps rightfully) didn't want to run anything I cooked up on her company's computer! A year or so later she was laid off a long with many others in the company performing similar roles. And this is the problem, isn't it? Once the job is automated, the analyst is no longer necessary. You really need to "learn to fish."
- troupe 3y ago
  Do you have any guess how long it will take to break even? Presumably you charged some amount to spend a few days automating it and hopefully it frees her up to do something more valuable for the company. Sometimes the reason things aren't automated is because there isn't anything more valuable for that person to do so automation doesn't have a positive return on investment.
  - ethbr1 3y ago
    It was an internal citizen developer initiative that I was supporting externally. Don't know her salary, so couldn't say. > because there isn't anything more valuable for that person to do I've seen an interesting shift post-COVID where (remaining) ops workers are often more coordinators of / responsible for their area of responsibility. Which is a distinction because in that sort of job, not having Task X leads directing into finding something else you can be doing to improve your area.
  - concordDance 3y ago
    Can't speak for them, but I've done office automation work with a breakeven measured in days.
bloak 3y ago
I think you're right. I've sometimes been in the situation of having to maintain a script that depends on APIs or data formats that change from time to time without warning. A fair amount of skill and effort is required. (There is skill involved in writing the script so that it is maintainable and not too fragile but still detects when something has gone wrong.) If the script is doing a task that has to be completed on a particular day every month, rather than whenever the script hacker is next available, then you'd have to choose between paying the hacker a reasonable retainer or having someone around who you know can still do the task manually, which means you might have to get them to do the task manually from time to time to stay in practice. A dilemma I keep facing in practice is converting a text that someone else has produced in their own special way, like the ghastly cluttered HTML from a word processor. For example, do I attempt to use a Perl script to automatically convert some of the mark-up, or do I throw away all the mark-up and put back the italics by hand, with the risk of making a mistake? (If anyone knows of a good configurable or modifiable tool for converting cluttered HTML into sane HTML/XML I'd be interested to hear about it. My Perl scripts for doing it are not great.)
- danielbln 3y ago
  Sounds like a good usecase for an LLM.
  - ben_w 3y ago
    Perhaps, but LLMs while may be slightly better (or slightly worse) than average humans, if you have 20 instances of the same LLM will have correlated errors whereas 20 humans will be… well, not entirely[0] uncorrelated, but less correlated, in their errors. [0] We'll still sometimes get an entire office of telephone help-desk people who are all convinced that $1/megabyte is the same as $0.01/megabyte, or whatever that story was a few decades back (I can't Google it any more to find it, there was a recording of the phone call).
    - genewitch 3y ago
      it was 0.01¢ and the rep / software was billing it as $0.01 - saying 0.01 of a penny is $0.01. I may be off by a factor of 10.

jiggawatts 3y ago

It’s basically a JOIN, the most fundamental data manipulation operator. If you can’t automate this, then you don’t “get” computers at all. I once had two guys arguing next to me about how it was impossible to produce a simple report about VM storage utilisation in under a month. I produced the report, printed it out, and shoved it in their hands just to shut them up while they were still busy arguing. A “month” of work required five joins.

radlad 3y ago
> It's basically a JOIN The comment you're replying just explained why this is not "basically a JOIN." There are two unrelated systems (not SQL databases) which don't expose any APIs to the analyst.
- tuatoru 3y ago
  It's still basically a join. If it's on a computer, it can be converted to text files, and then it can be JOINed, one way or another.
  - deleted 3y ago
    [deleted]
  - ethbr1 3y ago
    My sibling from another mother. That's exactly what we did.
  - csomar 3y ago
    In these cases, the work is usually about connecting the systems together than the data modification itself.
    - TeMPOraL 3y ago
      No, that's just made up work that's not required to solve a problem. Systems are already connected enough by virtue of being accessible by the same person on the same computer at the same time. Turning this into software project and building Proper Integration is only creating a tight coupling between the systems, which will only keep creating bugs and requires ongoing maintenance.
- c0pium 3y ago
  [flagged]
  - nicbou 3y ago
    This is a pointlessly mean and unpleasant comment. It was polite discussion up until that point.

ethbr1 3y ago

We were able to fix it up with some automation in a couple days. Critically, still running visually on her machine, and with internal resources for lifecycle support if the process changes.

radlad 3y ago
Finding internal resources to fix the tool when things change is very prescient. Growing up, my mom performed a job where she would essentially copy car crash data from one system into another, row by row. As a teenager learning to program, it was obvious to me this could be easily automated. She (perhaps rightfully) didn't want to run anything I cooked up on her company's computer! A year or so later she was laid off a long with many others in the company performing similar roles. And this is the problem, isn't it? Once the job is automated, the analyst is no longer necessary. You really need to "learn to fish."
troupe 3y ago
Do you have any guess how long it will take to break even? Presumably you charged some amount to spend a few days automating it and hopefully it frees her up to do something more valuable for the company. Sometimes the reason things aren't automated is because there isn't anything more valuable for that person to do so automation doesn't have a positive return on investment.
- ethbr1 3y ago
  It was an internal citizen developer initiative that I was supporting externally. Don't know her salary, so couldn't say. > because there isn't anything more valuable for that person to do I've seen an interesting shift post-COVID where (remaining) ops workers are often more coordinators of / responsible for their area of responsibility. Which is a distinction because in that sort of job, not having Task X leads directing into finding something else you can be doing to improve your area.
- concordDance 3y ago
  Can't speak for them, but I've done office automation work with a breakeven measured in days.

bloak 3y ago

I think you're right. I've sometimes been in the situation of having to maintain a script that depends on APIs or data formats that change from time to time without warning. A fair amount of skill and effort is required. (There is skill involved in writing the script so that it is maintainable and not too fragile but still detects when something has gone wrong.) If the script is doing a task that has to be completed on a particular day every month, rather than whenever the script hacker is next available, then you'd have to choose between paying the hacker a reasonable retainer or having someone around who you know can still do the task manually, which means you might have to get them to do the task manually from time to time to stay in practice. A dilemma I keep facing in practice is converting a text that someone else has produced in their own special way, like the ghastly cluttered HTML from a word processor. For example, do I attempt to use a Perl script to automatically convert some of the mark-up, or do I throw away all the mark-up and put back the italics by hand, with the risk of making a mistake? (If anyone knows of a good configurable or modifiable tool for converting cluttered HTML into sane HTML/XML I'd be interested to hear about it. My Perl scripts for doing it are not great.)

danielbln 3y ago
Sounds like a good usecase for an LLM.
- ben_w 3y ago
  Perhaps, but LLMs while may be slightly better (or slightly worse) than average humans, if you have 20 instances of the same LLM will have correlated errors whereas 20 humans will be… well, not entirely[0] uncorrelated, but less correlated, in their errors. [0] We'll still sometimes get an entire office of telephone help-desk people who are all convinced that $1/megabyte is the same as $0.01/megabyte, or whatever that story was a few decades back (I can't Google it any more to find it, there was a recording of the phone call).
  - genewitch 3y ago
    it was 0.01¢ and the rep / software was billing it as $0.01 - saying 0.01 of a penny is $0.01. I may be off by a factor of 10.