4 ms·
One of my anecdata: My team does a ton of large-scale processing (most of which is ETL), and we have TONS of servers at our disposal. We effectively throw hard
by mynameisash 5y ago
One of my anecdata:
My team does a ton of large-scale processing (most of which is ETL), and we have TONS of servers at our disposal. We effectively throw hardware at our compute, and even then, jobs can take 5-10 hours to run. No big deal - just run the job before you call it a day and it's done when you start tomorrow morning. Or when you have daily jobs, you schedule it and it's just done whenever it's done.
I can't tell you the number of times I've looked at a job someone wrote, made a 5 LOC change in 2 minutes, and cut processing from 8 hours to 5 hours. I literally had one that went from 5 hours to 5 minutes.
These extremely simple investments save a lot of money, but it's a problem that's spread across hundreds of different jobs, and it takes so much effort to convince people that we should invest in it. A daily job that costs $1500/mo to run (based on amortized hardware costs) needs only be optimized once, and those savings are yours forever. But I've had the conversation that is effectively: my time is more costly than a measly $1500 job, so it's okay if the job is expensive.
- resonious 5y agoThis is a pretty interesting example. It sounds like even with the numbers, it's a pretty hard sell to focus on performance..