32 ms·

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a syst

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a system IT had built for us with Hadoop, API gateways, multiple developers and hundreds of thousands of yearly cost. I told him that at our scale (now and any foreseeable future) I could easily run the whole thing on a USB drive attached to his laptop and a few python scripts. He looked really annoyed and I was never involved again with this project. I think it’s part of the BS cycle that’s prevalent in companies. You can’t admit that you are doing something simple.

noisy_boy 2y ago
In most non-tech companies, it comes down to the motive of the manager and in most cases it is expansion of reporting line and grabbing as much budget as possible. Using "simple" solutions runs counter to this central motivation.
- disqard 2y ago
  This is also true of tech companies. Witness how the "GenAI" hammer is being used right now at MS, Google, Meta, etc.
- eloisant 2y ago
  - the manager wants expansion - the developers want to get experience in a fancy stack to build up their resume Everyone benefits from the collective hallucination
boh 2y ago
That's the tech sector in a nutshell. Very few innovations actually matter to non-tech companies. Most companies could survive on Windows 98 software.
sien 2y ago
The flipside of this is that people at some places are fully aware of this and get very suspicious of consultants offering to handle their 'big data' challenges with loads of highly proprietary stuff. Data organisation can be more of an issue, but the general issue with this is often a lack of internal discipline on the data owners to carefully manage their data. But that isn't nearly as attractive for management. Putting 'brought in new cloud vendor / technology' looks better than 'improved data organisation' on a CV even if the new vendor was a waste of money.
treflop 2y ago
Assuming that it’s not me that’s being an ass (saying “I could do it with a few Python off a USB drive” would be being an ass), if my interviewer is offended because I offered the practical solution, I am not working there, especially if they’re from the team I’m going to work on. It’s like a reverse signal.
arvinsim 2y ago
Resume Driven Development is real. Yes, you can optimize the solution to just a USB drive. But that wouldn't check the boxes of people looking at your resume.

noisy_boy 2y ago

In most non-tech companies, it comes down to the motive of the manager and in most cases it is expansion of reporting line and grabbing as much budget as possible. Using "simple" solutions runs counter to this central motivation.

disqard 2y ago
This is also true of tech companies. Witness how the "GenAI" hammer is being used right now at MS, Google, Meta, etc.
eloisant 2y ago
- the manager wants expansion - the developers want to get experience in a fancy stack to build up their resume Everyone benefits from the collective hallucination

boh 2y ago

That's the tech sector in a nutshell. Very few innovations actually matter to non-tech companies. Most companies could survive on Windows 98 software.

sien 2y ago

The flipside of this is that people at some places are fully aware of this and get very suspicious of consultants offering to handle their 'big data' challenges with loads of highly proprietary stuff. Data organisation can be more of an issue, but the general issue with this is often a lack of internal discipline on the data owners to carefully manage their data. But that isn't nearly as attractive for management. Putting 'brought in new cloud vendor / technology' looks better than 'improved data organisation' on a CV even if the new vendor was a waste of money.

treflop 2y ago

Assuming that it’s not me that’s being an ass (saying “I could do it with a few Python off a USB drive” would be being an ass), if my interviewer is offended because I offered the practical solution, I am not working there, especially if they’re from the team I’m going to work on. It’s like a reverse signal.

arvinsim 2y ago

Resume Driven Development is real. Yes, you can optimize the solution to just a USB drive. But that wouldn't check the boxes of people looking at your resume.