2 ms·
> moving the ETL onto the fresh server. On which... it'll run at maybe 20% faster, because that's the scale of single-threaded processor speed improvements the
by jiggawatts 4y ago
> moving the ETL onto the fresh server.
On which... it'll run at maybe 20% faster, because that's the scale of single-threaded processor speed improvements these days. Not to mention that now there's a network hop involved, which will eat into any CPU gains.
Very few apps scale well with increasing core counts, and then hit a wall around 64 cores for almost everything.
Okay, okay, fine. The ETL is natively parallel code and somehow, magically, it can read inherently sequential file formats like multi-gigabyte CSV or JSON files in parallel. This tiny org already has 10 gigabit switches, SFPs, and everything.
Did you upgrade the database server too? No? Now the shiny new ETL server is twiddling its thumbs while the database server is getting overloaded.
Suddenly this option is "not so cheap". You have to buy a new database server and... uh-oh... it's Microsoft SQL Server, Oracle, or SAP HANA, and the licensing is going to eat half your tiny little company's profits for the year.
Did you forget the OS license, backup agent license, anti-malware license, and so on? I bet you did. All of those are extra, and either per-machine or per-core.
Someone has to set this all up. Small non-IT shops typically outsource this to an IT service company. They'll explain all the extras that turn a $3K purchase into a $30K purchase (including on-site assistance to install everything).
Or that 1 guy could have just taken a 10 minute look at the ETL logs, discovered that "SELECT * FROM HugeTable" is unnecessary, and fixed the problem.
- xupybd 4y agoThat's a totally different scenario. Imagine it's all Linux no back up required and the database server has lots of capacity. The lone developer can do it in half a day. No extra cost. But yes there exist situations where that is no possible. This is just one example of where it is