4 ms·

" Up till a few years ago, the traditional way of managing data (in SQL-based RDBMSs) was considered a relic of the past, as these systems couldn't scale to cop

by swalsh 7y ago

" Up till a few years ago, the traditional way of managing data (in SQL-based RDBMSs) was considered a relic of the past, as these systems couldn't scale to cope with such a huge amount of data. " I must have missed the boat on this one. I remember in 2010 there was a brief period of time where NoSQL was in fashion, but it rightfully died pretty quickly to a small set of specialized use cases. There have been some cases where big data systems have replaced more traditional rdbms systems, but now you can use SQL for those too (like Hive SQL). SQL is the one skill that has not become obsolete in the course of my career. Frankly I've started relying on it more, because it never goes obsolete. Also it's fast as hell. When I first started my career I did C#, and the .net Framework 2 was fairly new. Since then WinForms, and WebForms have gone away. ORM's changed, Javascript changed. Then I moved to Ruby, and Python, and PHP. Those ecosystems have evolved too. But the one thing that I learned 15 years ago, that I still use every day is SQL.

oyoun 7y ago
I totally agree with you. And SQL keep evolving
larrydag 7y ago
Unfortunately for this data analyst NoSQL is alive and well. It is being rolled out to enterprise levels without the downstream impacts fully realized. This requires all sorts of ad-hoc ETL data methods which in turn puts more time and strain on budgets and resources. At least this is true in my organization.
- baldfat 7y ago
  I think there is much more non-SQL then the OP realizes. I have been protesting non-SQL, SQL is more powerful than most realize. Also the term "Big Data" is being used 99% of the time about data bases I could load into my ram but counted in millions. This then causes people to search for things and find non-SQL.
  - deleted 7y ago
    [deleted]
  - EdwardDiego 7y ago
    That said, every big data tool I've encountered in current usage supports SQL - very few people writing Map/Reduce jobs these days. SQL has become the lingua franca of data analysis.
gwbas1c 7y ago
> I must have missed the boat on this one. I remember in 2010 there was a brief period of time where NoSQL was in fashion, but it rightfully died pretty quickly to a small set of specialized use cases. There have been some cases where big data systems have replaced more traditional rdbms systems, but now you can use SQL for those too (like Hive SQL). A few weeks ago I had a lead server-side architect call me up and ask if my client could tolerate eventual consistency. My answer was that "he could get away with that 10 years ago but that NoSQL went out of fashion and SQL came back in again." To put it in a different context: Some people just go through periods in their life where they don't keep up and don't realize what was "new and exciting" 10 years ago turned out to be smoke and mirrors.
- pdexter 7y ago
  What's the connection between NoSQL vs SQL and eventual consistency?
  - cle 7y ago
    NoSQL in some cases trades off consistency for scalability (generally by sharding data, so propagating data across shards introduces eventual consistency). As you can imagine, data that reside in a single shard don’t need to be propagated, so you can get strong consistency easily (e.g. fetching a single document). It’s when you start doing queries, joins, aggregations, etc. that you start encountering eventual consistency.
  - dragonwriter 7y ago
    NoSQL is a bad name that really doesn't have anything to do with SQL per se, but is a shorthand for non-relational datastores (no one calls non-SQL relational systems “NoSQL”), which, while they depart from the relational model in a number of different ways, typically include eventual rather than strong consistency to support liveness in a distributed context.
  - jsight 7y ago
    Mainly that most people who ask that question are using it as a lead-in to promoting a NoSQL product.
oyoun 7y ago
And database could scale up at this time: Pan-STARRS is one of the bigger database with PB
exabrial 7y ago
> brief period of time where NoSQL was in fashion, but it rightfully died pretty quickly That what is not standardized, dies
jstrong 7y ago
> Also it's fast as hell compared to what? excel?
jimbokun 7y ago
The nugget of truth was that joins and transactions are very difficult to implement in a performant manner in large scale, distributed systems. So it was possible to write very highly scalable databases if you give up on "SQL", which was a stand in for joins and transactions. Since then, developers have been coming up with clever ways to implement distributed joins and transactions more efficiently and scalably on top of distributed database engines, and thus "the return of SQL".
- 6gvONxR4sf7o 7y ago
  Where can I learn more about these clever developments?
  - jimbokun 7y ago
    I find CockroachDB one of the most interesting systems adding distributed transactions, joins and SQL support on top of a highly scalable key value store engine: https://www.cockroachlabs.com/docs/stable/architecture/overview.html https://www.cockroachlabs.com/docs/stable/architecture/overv...
  - jwatte 7y ago
    For example: http://info.snowflake.net/rs/252-RFO-227/images/Snowflake_SIGMOD.pdf http://info.snowflake.net/rs/252-RFO-227/images/Snowflake_SI... https://www.snowflake.com/wp-content/uploads/2014/10/A-Detailed-View-Inside-Snowflake.pdf https://www.snowflake.com/wp-content/uploads/2014/10/A-Detai...
collyw 7y ago
The youngsters in my work don't really know SQL. I am probably see as one of these old developers that refuses to get with the times (rather than seeing NoSQL for a step backwards that it is). Its this constant churn of bullshit that makes me want to do a different job after nearly 20 years (as well as the nonsense that is the hiring process).
- mumblemumble 7y ago
  This, I think, is the true cost of technologies like ORM. And it's hard to notice, because it doesn't show up right away; it's a generational thing. It reminds me a bit of a paper I saw recently on The Morning Paper called Ironies of Automation: https://blog.acolyer.org/2020/01/08/ironies-of-automation/ https://blog.acolyer.org/2020/01/08/ironies-of-automation/
pastor_elm 7y ago
So long as fresh faced developers arrive on the scene, there will always be people looking to get around using SQL.

by swalsh 7y ago

oyoun 7y ago

I totally agree with you. And SQL keep evolving

larrydag 7y ago

Unfortunately for this data analyst NoSQL is alive and well. It is being rolled out to enterprise levels without the downstream impacts fully realized. This requires all sorts of ad-hoc ETL data methods which in turn puts more time and strain on budgets and resources. At least this is true in my organization.

baldfat 7y ago
I think there is much more non-SQL then the OP realizes. I have been protesting non-SQL, SQL is more powerful than most realize. Also the term "Big Data" is being used 99% of the time about data bases I could load into my ram but counted in millions. This then causes people to search for things and find non-SQL.
- deleted 7y ago
  [deleted]
- EdwardDiego 7y ago
  That said, every big data tool I've encountered in current usage supports SQL - very few people writing Map/Reduce jobs these days. SQL has become the lingua franca of data analysis.

gwbas1c 7y ago

> I must have missed the boat on this one. I remember in 2010 there was a brief period of time where NoSQL was in fashion, but it rightfully died pretty quickly to a small set of specialized use cases. There have been some cases where big data systems have replaced more traditional rdbms systems, but now you can use SQL for those too (like Hive SQL). A few weeks ago I had a lead server-side architect call me up and ask if my client could tolerate eventual consistency. My answer was that "he could get away with that 10 years ago but that NoSQL went out of fashion and SQL came back in again." To put it in a different context: Some people just go through periods in their life where they don't keep up and don't realize what was "new and exciting" 10 years ago turned out to be smoke and mirrors.

pdexter 7y ago
What's the connection between NoSQL vs SQL and eventual consistency?
- cle 7y ago
  NoSQL in some cases trades off consistency for scalability (generally by sharding data, so propagating data across shards introduces eventual consistency). As you can imagine, data that reside in a single shard don’t need to be propagated, so you can get strong consistency easily (e.g. fetching a single document). It’s when you start doing queries, joins, aggregations, etc. that you start encountering eventual consistency.
- dragonwriter 7y ago
  NoSQL is a bad name that really doesn't have anything to do with SQL per se, but is a shorthand for non-relational datastores (no one calls non-SQL relational systems “NoSQL”), which, while they depart from the relational model in a number of different ways, typically include eventual rather than strong consistency to support liveness in a distributed context.
- jsight 7y ago
  Mainly that most people who ask that question are using it as a lead-in to promoting a NoSQL product.

oyoun 7y ago

And database could scale up at this time: Pan-STARRS is one of the bigger database with PB

exabrial 7y ago

> brief period of time where NoSQL was in fashion, but it rightfully died pretty quickly That what is not standardized, dies

jstrong 7y ago

> Also it's fast as hell compared to what? excel?

jimbokun 7y ago

The nugget of truth was that joins and transactions are very difficult to implement in a performant manner in large scale, distributed systems. So it was possible to write very highly scalable databases if you give up on "SQL", which was a stand in for joins and transactions. Since then, developers have been coming up with clever ways to implement distributed joins and transactions more efficiently and scalably on top of distributed database engines, and thus "the return of SQL".

6gvONxR4sf7o 7y ago
Where can I learn more about these clever developments?
- jimbokun 7y ago
  I find CockroachDB one of the most interesting systems adding distributed transactions, joins and SQL support on top of a highly scalable key value store engine: https://www.cockroachlabs.com/docs/stable/architecture/overview.html https://www.cockroachlabs.com/docs/stable/architecture/overv...
- jwatte 7y ago
  For example: http://info.snowflake.net/rs/252-RFO-227/images/Snowflake_SIGMOD.pdf http://info.snowflake.net/rs/252-RFO-227/images/Snowflake_SI... https://www.snowflake.com/wp-content/uploads/2014/10/A-Detailed-View-Inside-Snowflake.pdf https://www.snowflake.com/wp-content/uploads/2014/10/A-Detai...

collyw 7y ago

The youngsters in my work don't really know SQL. I am probably see as one of these old developers that refuses to get with the times (rather than seeing NoSQL for a step backwards that it is). Its this constant churn of bullshit that makes me want to do a different job after nearly 20 years (as well as the nonsense that is the hiring process).

mumblemumble 7y ago
This, I think, is the true cost of technologies like ORM. And it's hard to notice, because it doesn't show up right away; it's a generational thing. It reminds me a bit of a paper I saw recently on The Morning Paper called Ironies of Automation: https://blog.acolyer.org/2020/01/08/ironies-of-automation/ https://blog.acolyer.org/2020/01/08/ironies-of-automation/

pastor_elm 7y ago

So long as fresh faced developers arrive on the scene, there will always be people looking to get around using SQL.