4 ms·

Am I missing something or is this blog post just someone discovering that ETL exists

pjmlp 8y ago
Lots of "modern" blog posts seem like people rediscovering that X exists. A consequence of many skipping on traditional learning processes and how developers are now also forced into castings and portfolios.
- wild_preference 8y ago
  There is no traditional process that has you knowing everything about everything much less be able to reverse identify an implementation with the underpinning concept. There are things I know much more of than you do. Seems a bit masturbatory for me to see you miss a concept and go “ah, the modern lack of traditional learning processes!”
- Spooky23 8y ago
  It's also a negative side-effect of the improvement of software and automation in this space. In ye olden times, there would be an grumpy DBA in this story terminating the query and yelling at the user. Many, many people are consuming databases without understanding how to approach SQL optimization, because nobody is forcing them to!
  - sidlls 8y ago
    There's a happy middle ground between what we have now and those DBAs playing hardass without even trying to understand the need.
    - mmt 8y ago
      > DBAs playing hardass without even trying to understand the need. Although the parent posed a caricature with "yelling", this is an even more extreme one. The "forcing" function merely requires a bit of "hardass" but not the willful ignorance you associate with it. At the risk of going true-scotsman, I believe that competent DBAs capable of the forcing function would already understand the need without trying. As such, I don't think there needs to be any kind of "medium" between two extremes of ignorance, but, rather, back to the GP's point, a reduction of ignorance in "what we have now". Of course, I don't believe hardass/yelling is the best way to achieve that, either, instead favoring (original) Devops culture, but that also doesn't work if Ops (including DBA) skills aren't recognized as valuable any more (and "Devops" just means "replace Ops with Devs").
  - hermitdev 8y ago
    Ah, yes. I interned in the Database Administration team of a Fortune 500 company. I definitely learned a lot as a developer under the tutelage of some old school mainframers (I was primarily targeting DB2 on OS/390). I learned a lot about query optimization in my 3 years as an intern. I also caused a few issues along the way, including a number of PR reports to IBM. Some of which were benign, like incorrect documentation. Others were more problematic, such as semantically identical queries returning different results (e.g. coalesce vs case when null in a column spec) to knocking out the entire development LPAR on the mainframe with a simple select statement (hit a bug in the query optimizer and it crashed not just the DB, but the whole LPAR). Still, those 3 years were invaluable to me. They developed both an intuition on how to write optimal queries, but also the knowledge of how to utilize the tools to identify problems, measure and fix them. It has served me well as a developer in the 15 years since I left my internship. I don't get into fights with DBAs since they recognize I know as much or more than them about optimizing queries on datasets I work with.
seeker61 8y ago
Re-inventing it. (And yes, that's worse.)
- sampleinajar 8y ago
  The number of times I reinvented something due to lack of knowledge, assistance, training, or from being thrown into a tech stack I am unfamiliar with, etc. was quite maddening. I'm pretty sure that's how most CS courses are designed really, "Figure this out. Did you get? Great. That will never be useful because you can just do this now." Teaching people to think, blah blah. Complain when they don't know.
sidlls 8y ago
Well it's hardly surprising given the emphasis on CS trivia and fad-devops that exists in this industry. Basics like this are often neglected because employees are too busy chasing Kafka and Kubernetes or re-implementing skip lists on whiteboards. After all, if you can recall your textbook CS surely you can just re-invent the thousands of PhD and engineering hours that went into this stuff previously on an ad hoc basis for our projects and scale it with K8s.
- blub 8y ago
  Methinks you're unfair to the author: https://en.m.wikipedia.org/wiki/Diomidis_Spinellis https://en.m.wikipedia.org/wiki/Diomidis_Spinellis And at the risk of sounding like a clown, I'll go ahead and ask: ETL seems to be such a basic and generic thing to the point that it's useless as a concept. What did those Phds achieve in those thousand hours that the author missed?
phyzome 8y ago
https://en.wikipedia.org/wiki/Extract,_transform,_load https://en.wikipedia.org/wiki/Extract,_transform,_load for anyone who didn't catch the acronym.
blub 8y ago
As someone that doesn't know much about ETL, I don't really see why it's worth mentioning in this context. Yes, what the OP is doing can be called extract/transform/load, but that's an extremely general set of operations. Do you have some specific tool in mind that would make solving their problem a breeze? Is there a specific process that one should follow for ETL which was ignored?
- camel_Snake 8y ago
  Note: I'm just a backend dev, but I've been looking into data engineering a bit recently so I'm not terribly experienced. Happy to be corrected Problem here is they are using a system designed for the typical webapp use-case (OLTP). It's optimized to find a particular row or rows and then maybe update them with low latency. ETL tools are more optimized towards this sort of bulk data processing (OLAP) where we want to scan all the rows for a particular column and run some sort of analysis. A simple optimization some databases make for this is to orient the data so that each whole column is stored together rather than storing each row together. This makes it much quicker to run queries over an entire column or two in a table. Generally modern ETL is optimized towards working with LARGE amounts of data so there are many tools that are optimized for this.

Lots of "modern" blog posts seem like people rediscovering that X exists. A consequence of many skipping on traditional learning processes and how developers are now also forced into castings and portfolios.

wild_preference 8y ago
There is no traditional process that has you knowing everything about everything much less be able to reverse identify an implementation with the underpinning concept. There are things I know much more of than you do. Seems a bit masturbatory for me to see you miss a concept and go “ah, the modern lack of traditional learning processes!”
Spooky23 8y ago
It's also a negative side-effect of the improvement of software and automation in this space. In ye olden times, there would be an grumpy DBA in this story terminating the query and yelling at the user. Many, many people are consuming databases without understanding how to approach SQL optimization, because nobody is forcing them to!
- sidlls 8y ago
  There's a happy middle ground between what we have now and those DBAs playing hardass without even trying to understand the need.
  - mmt 8y ago
    > DBAs playing hardass without even trying to understand the need. Although the parent posed a caricature with "yelling", this is an even more extreme one. The "forcing" function merely requires a bit of "hardass" but not the willful ignorance you associate with it. At the risk of going true-scotsman, I believe that competent DBAs capable of the forcing function would already understand the need without trying. As such, I don't think there needs to be any kind of "medium" between two extremes of ignorance, but, rather, back to the GP's point, a reduction of ignorance in "what we have now". Of course, I don't believe hardass/yelling is the best way to achieve that, either, instead favoring (original) Devops culture, but that also doesn't work if Ops (including DBA) skills aren't recognized as valuable any more (and "Devops" just means "replace Ops with Devs").
- hermitdev 8y ago
  Ah, yes. I interned in the Database Administration team of a Fortune 500 company. I definitely learned a lot as a developer under the tutelage of some old school mainframers (I was primarily targeting DB2 on OS/390). I learned a lot about query optimization in my 3 years as an intern. I also caused a few issues along the way, including a number of PR reports to IBM. Some of which were benign, like incorrect documentation. Others were more problematic, such as semantically identical queries returning different results (e.g. coalesce vs case when null in a column spec) to knocking out the entire development LPAR on the mainframe with a simple select statement (hit a bug in the query optimizer and it crashed not just the DB, but the whole LPAR). Still, those 3 years were invaluable to me. They developed both an intuition on how to write optimal queries, but also the knowledge of how to utilize the tools to identify problems, measure and fix them. It has served me well as a developer in the 15 years since I left my internship. I don't get into fights with DBAs since they recognize I know as much or more than them about optimizing queries on datasets I work with.

seeker61 8y ago

Re-inventing it. (And yes, that's worse.)

sampleinajar 8y ago
The number of times I reinvented something due to lack of knowledge, assistance, training, or from being thrown into a tech stack I am unfamiliar with, etc. was quite maddening. I'm pretty sure that's how most CS courses are designed really, "Figure this out. Did you get? Great. That will never be useful because you can just do this now." Teaching people to think, blah blah. Complain when they don't know.

sidlls 8y ago

Well it's hardly surprising given the emphasis on CS trivia and fad-devops that exists in this industry. Basics like this are often neglected because employees are too busy chasing Kafka and Kubernetes or re-implementing skip lists on whiteboards. After all, if you can recall your textbook CS surely you can just re-invent the thousands of PhD and engineering hours that went into this stuff previously on an ad hoc basis for our projects and scale it with K8s.

blub 8y ago
Methinks you're unfair to the author: https://en.m.wikipedia.org/wiki/Diomidis_Spinellis https://en.m.wikipedia.org/wiki/Diomidis_Spinellis And at the risk of sounding like a clown, I'll go ahead and ask: ETL seems to be such a basic and generic thing to the point that it's useless as a concept. What did those Phds achieve in those thousand hours that the author missed?

phyzome 8y ago

https://en.wikipedia.org/wiki/Extract,_transform,_load https://en.wikipedia.org/wiki/Extract,_transform,_load for anyone who didn't catch the acronym.

blub 8y ago

As someone that doesn't know much about ETL, I don't really see why it's worth mentioning in this context. Yes, what the OP is doing can be called extract/transform/load, but that's an extremely general set of operations. Do you have some specific tool in mind that would make solving their problem a breeze? Is there a specific process that one should follow for ETL which was ignored?

camel_Snake 8y ago
Note: I'm just a backend dev, but I've been looking into data engineering a bit recently so I'm not terribly experienced. Happy to be corrected Problem here is they are using a system designed for the typical webapp use-case (OLTP). It's optimized to find a particular row or rows and then maybe update them with low latency. ETL tools are more optimized towards this sort of bulk data processing (OLAP) where we want to scan all the rows for a particular column and run some sort of analysis. A simple optimization some databases make for this is to orient the data so that each whole column is stored together rather than storing each row together. This makes it much quicker to run queries over an entire column or two in a table. Generally modern ETL is optimized towards working with LARGE amounts of data so there are many tools that are optimized for this.