5 ms·

Being a total stubborn asshole to my boss over last decade because I refused to even investigate about how we could replace our old and "complicated" SQL integr

Being a total stubborn asshole to my boss over last decade because I refused to even investigate about how we could replace our old and "complicated" SQL integration/export workflows with a modern and "intuitive" visual proprietary ETL this article is a real relief for me! I'm now officially a bleeding edge DevOps with 10 years expertise on the brand new "old school" ELT (Extract,Load, Transform). LoL

dkersten 7y ago
Hah. I guess that happens semi regularly in our industry, that by being stubborn and sticking with old technologies you may one day wake up and find out its suddenly bleeding edge again (kind of, in some form, anyway)
- pjmlp 7y ago
  Like writing microservices in Cobol on Amazon Cloud. :) https://www.infoq.com/articles/migrate-mainframe-aws/ https://www.infoq.com/articles/migrate-mainframe-aws/
deleted 7y ago
[deleted]
pjmlp 7y ago
Yeah, that is why I long learned to be skeptical in a pop culture driven industry, always looking after the new fad to sell stuff.
- kingludite 7y ago
  Like that it's funny to be a human. When all features have been added its so complicated one might as well start from scratch.... Wait I can put it even worse: What the Father Does, the Son Does - John 5:19-23
barrkel 7y ago
Complicated SQL is definitely complicated, though. Get an expected 1:1 join wrong (so it ends up 1:n) and duplicates muck things up; and typically unit tests don't catch these kinds of errors because the test set is minimal. Long scripts of SQL with intermediate steps (e.g. building temp tables) are hard to reason about and debug, and especially painful when they fail only in rare production scenarios. Developers find it a lot easier to reason on an item by item basis, and not on a set or batch basis, and it's very tricky to get error handling and transactions right on a batch basis unless you want to fail the whole batch. Putting all the work on the database also means you need to scale up that layer. From another POV, it can be simpler to have cheap point lookup, secondary indexes in stores that are tuned for the specifics of the queries, and get performance back by scaling out the compute.
- Twisell 7y ago
  1.If you rely on an ETL to automagically remove duplicates you are lying to yourself because you treat external symptoms, the illness is the underlying query and hiding this fact can only lead to more headaches later on. 2.Testing for duplicate is a SQL one-liner 3.If your data model is well conceived you can actually unwanted 1-n relationship to be inserted in the first place. Yeah sharding database is coming late to the game but ACID compliance is a killer feature. Code to manage inconsistencies in NoSQL is way less trivial as far as I know... unless it's ok to have an inconsistent database... Guess it depend on use case but ACID seems nice for any important dataset.
  - barrkel 7y ago
    I think you misunderstood me on the duplicates. I'm talking about how easy it is to e.g. left join an attribute from another table and under-qualify the association so that you accidentally pull in unwanted rows. It's not that you have duplicates in the data; the data is correct and the model is correct. It's rather that the query is incorrect and the set of tests, owing to them typically being written by the same person who wrote the code, doesn't populate the association table with more rows than expected. I mention this because it's a bug I've seen crop up several times in production. When you write 'join' you're typically fetching data to add extra columns to your result set; it's relatively unusual to want a row product in the absence of group by & aggregates, or an ORM doing eager loading of a detail table. But nothing in SQL requires you to state your product expectations on the join, apart from left/right outer vs inner join. I personally think SQL could do with forcing people to be more explicit on this point. I fully agree on the inconsistencies in NoSQL; I think the most sensible way to use NoSQL is as a document store without relations, and keep the relations somewhere else. Manually maintaining invariants (especially of the denormalized bidirectional kind that some stores require) in an environment with less than complete transaction support isn't my idea of fun.

dkersten 7y ago

Hah. I guess that happens semi regularly in our industry, that by being stubborn and sticking with old technologies you may one day wake up and find out its suddenly bleeding edge again (kind of, in some form, anyway)

pjmlp 7y ago
Like writing microservices in Cobol on Amazon Cloud. :) https://www.infoq.com/articles/migrate-mainframe-aws/ https://www.infoq.com/articles/migrate-mainframe-aws/

deleted 7y ago

[deleted]

pjmlp 7y ago

Yeah, that is why I long learned to be skeptical in a pop culture driven industry, always looking after the new fad to sell stuff.

kingludite 7y ago
Like that it's funny to be a human. When all features have been added its so complicated one might as well start from scratch.... Wait I can put it even worse: What the Father Does, the Son Does - John 5:19-23

barrkel 7y ago

Complicated SQL is definitely complicated, though. Get an expected 1:1 join wrong (so it ends up 1:n) and duplicates muck things up; and typically unit tests don't catch these kinds of errors because the test set is minimal. Long scripts of SQL with intermediate steps (e.g. building temp tables) are hard to reason about and debug, and especially painful when they fail only in rare production scenarios. Developers find it a lot easier to reason on an item by item basis, and not on a set or batch basis, and it's very tricky to get error handling and transactions right on a batch basis unless you want to fail the whole batch. Putting all the work on the database also means you need to scale up that layer. From another POV, it can be simpler to have cheap point lookup, secondary indexes in stores that are tuned for the specifics of the queries, and get performance back by scaling out the compute.

Twisell 7y ago
1.If you rely on an ETL to automagically remove duplicates you are lying to yourself because you treat external symptoms, the illness is the underlying query and hiding this fact can only lead to more headaches later on. 2.Testing for duplicate is a SQL one-liner 3.If your data model is well conceived you can actually unwanted 1-n relationship to be inserted in the first place. Yeah sharding database is coming late to the game but ACID compliance is a killer feature. Code to manage inconsistencies in NoSQL is way less trivial as far as I know... unless it's ok to have an inconsistent database... Guess it depend on use case but ACID seems nice for any important dataset.
- barrkel 7y ago
  I think you misunderstood me on the duplicates. I'm talking about how easy it is to e.g. left join an attribute from another table and under-qualify the association so that you accidentally pull in unwanted rows. It's not that you have duplicates in the data; the data is correct and the model is correct. It's rather that the query is incorrect and the set of tests, owing to them typically being written by the same person who wrote the code, doesn't populate the association table with more rows than expected. I mention this because it's a bug I've seen crop up several times in production. When you write 'join' you're typically fetching data to add extra columns to your result set; it's relatively unusual to want a row product in the absence of group by & aggregates, or an ORM doing eager loading of a detail table. But nothing in SQL requires you to state your product expectations on the join, apart from left/right outer vs inner join. I personally think SQL could do with forcing people to be more explicit on this point. I fully agree on the inconsistencies in NoSQL; I think the most sensible way to use NoSQL is as a document store without relations, and keep the relations somewhere else. Manually maintaining invariants (especially of the denormalized bidirectional kind that some stores require) in an environment with less than complete transaction support isn't my idea of fun.