5 ms·

I've had similar experiences. Sometimes we'll have a dataset with tens of thousands records and it will give rise to the belief that it's a problem that require

by tootie 6y ago

I've had similar experiences. Sometimes we'll have a dataset with tens of thousands records and it will give rise to the belief that it's a problem that requires a highly scalable solution because "tens of thousands" is more than a human can hold in their head. In reality, if the records are just a few columns of data, the whole set can be serialized to a single file and consumed in one gulp into a single object in memory on commodity hardware no sweat. Then process it with a for loop. Very few enterprises actually have big big data.

Cthulhu_ 6y ago
Said enterprise WISH they had big data. Or maybe it's fear, as in, 'what IF we get big data?'
- SketchySeaBeast 6y ago
  I'm aware of a couple of companies who behave like that - "well, we could increase our user based by an order of magnitude at any point here, so better spring for the order of magnitude more expensive database, just in case we need it." Feels like toxic optimism.
  - dvfjsdhgfv 6y ago
    It's not just about scaling databases, some people are simply unable to assess reasonable limits on any system. A few years ago certain Scandinavian publisher decided to replace their standard industry tools by a single "Digital Experience Platform" that was expected to do everything. After a couple of years they understood it's a stupid idea and gave up. Then later someone in the management thought that since they already spent some millions of euros they should continue anyway. This behemoth is so slow and buggy the end users work at 1/4th speed but everyone is afraid to say anything as the ones who did have been fired. The current PM is sending weekly success messages. It's hilarious. And all because someone once had a fantasy of having one huge system that does everything.
  - spratzt 6y ago
    I love the phrase 'toxic optimism'.
- tim333 6y ago
  I've noticed business people have a different idea of what 'big data' means to tech guys. The business guys think it means a lot of data like the records of a million people which is a lot of data but not the tech guy definition which tends to be data too large to process on a single machine. Those come out at something like 1GB and 10TB which are obviously rather different.
SatvikBeri 6y ago
My solution started out as a 10-line Python script where I would manually clean the data we received, then process it. CEO: "Will this scale?" Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing." As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db... I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.

by tootie 6y ago

Cthulhu_ 6y ago

Said enterprise WISH they had big data. Or maybe it's fear, as in, 'what IF we get big data?'

SketchySeaBeast 6y ago
I'm aware of a couple of companies who behave like that - "well, we could increase our user based by an order of magnitude at any point here, so better spring for the order of magnitude more expensive database, just in case we need it." Feels like toxic optimism.
- dvfjsdhgfv 6y ago
  It's not just about scaling databases, some people are simply unable to assess reasonable limits on any system. A few years ago certain Scandinavian publisher decided to replace their standard industry tools by a single "Digital Experience Platform" that was expected to do everything. After a couple of years they understood it's a stupid idea and gave up. Then later someone in the management thought that since they already spent some millions of euros they should continue anyway. This behemoth is so slow and buggy the end users work at 1/4th speed but everyone is afraid to say anything as the ones who did have been fired. The current PM is sending weekly success messages. It's hilarious. And all because someone once had a fantasy of having one huge system that does everything.
- spratzt 6y ago
  I love the phrase 'toxic optimism'.
tim333 6y ago
I've noticed business people have a different idea of what 'big data' means to tech guys. The business guys think it means a lot of data like the records of a million people which is a lot of data but not the tech guy definition which tends to be data too large to process on a single machine. Those come out at something like 1GB and 10TB which are obviously rather different.

SatvikBeri 6y ago

My solution started out as a 10-line Python script where I would manually clean the data we received, then process it. CEO: "Will this scale?" Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing." As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db... I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.