16 ms·

Why SQL is beating NoSQL, and what this means for the future of data

405 points by nreece 9y ago

manigandham 9y ago
There's really no such thing as NoSQL first of all. Relational, graph, key/value (including wide column), document-store, etc. There are lots of database types but it turns out relational works 95% of the time and we're getting better at recognizing the use-cases for the others. SQL is also just a query language, that's literally the name. Any database can implement it, not just relational. Is SQL a great interface for both OLTP and OLAP? Yes, it's proven itself over decades. Is it the only valid interface? No. Does it work with other newer/different data systems? Yes, Spark, Hadoop, Kafka, and even these new distributed relational databases are all examples of such. It would be far better for the industry if we can get past these ideological and faulty "debates" and move on to more important things.
- jevgeni 9y ago
  Thank you!
takeda 9y ago
Great article, but I think the author puts too much emphasis on SQL instead on the fact that the data being stored in a relational way. Personally I actually would like an alternative language that would be capable to be integrated with the language in such way that it could also fall with type checking. So for example if I rename a column in a database, the type checker would highlight all places in my code that were broken by this change. JOOQ[1] seems to do something like that, but it's only for Java. Also, looks like QUEL[2] would be a bit easier to be integrated with a language, too bad it died. [1] http://www.jooq.org/ http://www.jooq.org/ [2] https://en.wikipedia.org/wiki/QUEL_query_languages https://en.wikipedia.org/wiki/QUEL_query_languages
labster 9y ago
If SQL is to databases as Javascript is to browsers — ubiquitous and largely standard across the market — are there any languages that transpile to SQL? People in this thread have commented that SQL is clunky, and JS definitely fit that description for a while. So I'm wondering if there are any alternatives that would prove SQL's clunkiness to me.
- yenwel 9y ago
  LINQ?
  - deleted 9y ago
    [deleted]
- evv555 9y ago
  Ecto for Elixir might fit that description. Queries are composable unlike SQL.
- amirouche 9y ago
  ORMs transpile to SQL. Just like JavaScript is not perfect, SQL is not perfect. I would rather have a general purpose programming language like Scheme to run queries against my database.
- bigger_cheese 9y ago
  I'm an engineer (the non software kind) that works in an industrial plant. I often have to do ad-hoc querying manipulation etc of data from a lot of diverse sources. Usually to do fault finding and analytics - sometimes also for reporting. I don't know about transpiling but I use SAS to work around SQL's clunkiness. I can embed SQL seamlessly into my SAS code so it works well. I use SQL for things that are simpler in SQL (selecting, filtering, aggregating data and "case when" statements etc) and I use SAS's data step language for when I have to do more complicated stuff with the data like Arrays, loops, macros, weighted averaging, time series based stuff and regressions. The thing I love about SQL is it is the lowest common denominator for all of my organisations data it doesn't mater if data is stored in an Oracle table, a Microsoft SQL Server table, or even Access DB or spreadsheet. The language to extract/merge/join all the data is the same and standardized. We also have things called "Data historians" which are basically time-series datastores (usually they are linked to PLC vendors) and nothing frustrates me more when they use some proprietary "sql like" language which seems similar to ANSI SQL except things like aggregations (SUM, AVG, MAX etc) don't work or they do weird things with data types (especially dates) or similar everyone of these seem to be subtly different - annoying.
- 9y ago
russellbeattie 9y ago
Wow... I wish I could just wait a few years and have everyone come to their senses on other things as well.
amirouche 9y ago
What is good about SQL is not the programming theory behind it, or the fact that it's supposed to be simple enough for non-programmers. Just like JavaScript, the good thing about SQL is that it's a standard.
nextInt 9y ago
Lol SQL is making a comeback? It never left.
- SQL2219 9y ago
  This.
- krf 9y ago
  It only left in terms of attention from startups I suppose. I also disagree with the reason the article suggested - SQL could not handle the loads. My opinion is that startups simply liked the idea of not having a schema as it fit their agile approach. So, they went NoSQL because it allowed them to get going faster and change easier.
  - xirdstl 9y ago
    That, and they like to believe they need to scale like Facebook.
    - Myrmornis 9y ago
      Sadly, this is a common reason.
- wernercd 9y ago
  SQL is dead. Long live SQL.
- 20years 9y ago
  Exactly! SQL was never dead.
Clubber 9y ago
So the long and short of it is SQL would be a good common language for relational and NoSQL style databases. I would agree with that, but since they different monsters (RDBMS and NoSQL), it will take a bit of tweaking to find a good dialect (how do joins work, etc). Of course it makes sense to adapt the NoSQL databases to the existing ANSI SQL rather than make existing ANSI SQL users switch to a new type of SQL that accommodates NoSQL, but we'll see what happens. Google create a "Standard SQL," but I'm not familiar with it. Here is a link on Google Standard SQL: https://cloud.google.com/bigquery/docs/reference/standard-sql/migrating-from-legacy-sql https://cloud.google.com/bigquery/docs/reference/standard-sq...
- nkozyra 9y ago
  > So the long and short of it is SQL would be a good common language for relational and NoSQL style databases. Yes, I'm sure the inevitable "SQL for NoSQL Databases" book will be a good read :)
mmaunder 9y ago
It's not one versus the other. That's like suggesting JSON is going to 'beat' HTML. They're different tools for different jobs and both are used alongside each other to great effect.
wildchild 9y ago
Because nosql is hype and snake-oil. Barely usable tools to dump startups on ignorant investors.
peterwwillis 9y ago
SQL wins because big iron wins. At the end of the day, the big, clunky, slow, complicated, aggravating, old systems stick around because they are built to stick around. Looking at, say, an eventually consistent distributed decentralized kv store, one might be tricked into believing it's simple enough to deploy with enough nodes and general enough interfaces that you can build a complicated system on top of it if needed, and rely on its supppsed simplicity and scalability the rest of the time. But nobody tells you about the shitty implementation. The replication that doesn't resume, the stodgy transfers, the imbalanced distribution, the consensus conflict, the infinitely expanding disk, the churning CPU. How at scale, if all the other aspects of your infra aren't scaling along with your data size, the whole thing blows. Traditional SQL databases end up being many times simpler in practice, and because of their inherent scaling limitations, much easier to manage. And most importantly: their implementations aren't buggy. SQL is just more reliable.
- wjossey 9y ago
  As someone who helped design, then manage a massive NoSQL store which hundreds of terabytes of data, of which most of it was hot, I’ll humbly disagree and say it’s not necessarily as bad as you make it sound. Did we have someone who managed that system? Sometimes, but mostly it just did it’s own thing. We ironically invested way more time on our MySQL database over the years because we couldn’t get that to scale the way we wanted to, but I think that was specific to a problem we were having. Did we later invest a lot of time in that system (the NoSQL one)? Yes, because it was very cost effective for us to do so. At certain scale throwing people at optimization problems can pay huge dividends. But, this can be said of most infrstracture. It’s usually worth revisiting every year or two and seeing how what you can squeeze out. Did we have churning CPU, infinitely expanding disk, replication issues, and more? Sure, but not very commonly and mostly it was fairly easily resolved. More importantly though, it was a solid system that was the underpinning of a colossal system, and it behaved admiraly more than 99.9% of the time. Will most projects benefit from a hugely distributed KV store? Nope. But I’m still glad they exist!
  - cachemiss 9y ago
    Caveat, this comment isn't directed at you (I agree with your comment), but rather the points around what you are saying. One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated. When talking about stores, it's important to be explicit about a few things: 1. Storage model 2. Distribution model 3. Access model 4. Transaction model 5. Maturity and competence of implementation What happens is people talk about "SQL" as either an NSM or DSM storage model, over either a single node, or possibly more than that in some of the MPP systems, using SQL as an access model, with linearizable transactions, and a mature competent implementation. NoSQL when most people refer to it can be any combination of those things, as long as the access model isn't SQL. I work on database engines, and it's important to decouple these things and be explicit about them when discussing various tradeoffs. You can do SQL the language over a distributed k/v store (not always a great idea) and other non-tabular / relational models and you can distribute relational engines (though scaling linearizable transactions is difficult and doesn't scale for certain use cases due to physics, but that's unrelated to the relational part of it). Generally people talk about joins not scaling in some normalized form, but then what they do is just materialize the join into whatever they are using to store things in a denormalized model, which has its own drawbacks. As to the comment above you, SQL vs NoSQL also doesn't have anything to do with the relative maturity of anything. Some of the newer non-relational engines have some operational issues, but that doesn't really have anything to do with their storage model or access method, it just has to due with the competence of the implementation. MongoDB is difficult operationally not because it's not a relational engine, but because it wasn't well designed. Just like people put SQL over non-tabular stores, you can build non-tabular / relational engines over relational engines (sharding PostgreSQL etc.). In fact major cloud vendors do just that.
taeric 9y ago
I do find the amusing "backronym" rather hard to take at face value. I also found the old critics of SQL databases odd focus on the query language hard to take. The problem wasn't the queries, per se. The problem was the power of the queries for interactive uses and how that doesn't scale up. That is, the problem with most SQL databases is that you need some pretty specialized knowledge in order to construct good queries. Not shockingly, the problem with most modern key/value (or otherwise) datastores is that you need some pretty specialized knowledge in order to construct good queries. Now, for things getting off the ground, this is probably fine. Most of the places you will go south with queries is in the ad-hoc style query. Of course, that style query is perfect for interactive use. Ideal, even. It falls on its face if it is supposed to support an automated case at high TPS. Unfortunately, automated cases at low TPS often turn into automated cases at high TPS. Worse, interactive cases at ridiculously low tps often turn into automated cases at low TPS. Which, of course, just feeds itself. How this feeds itself, of course, is you can explore your data much more effectively if there is an ad-hoc query engine. So, we are now seeing the resurgence of ad-hoc queries and the learning that those can lead to some powerful insights.
- aaron-lebo 9y ago
  The thing is ORMs have made writing easy queries easy and hard queries possible since at the very least 2005 with Rails, and they are easily extensible and composable. It's mostly a solved issue. If you want to do complex data manipulation, surprise, it takes a powerful tool. Seems a lot of the hype is the result of VC $ + naiveity/overselling about what MongoDB and co would be capable of. The marketing is natural - MongoDB made databases "easy" and easy is the only way to sell lowest common denominator tech to become a unicorn. Bad incentives for all involved. Sql is a kind of rite of passage? If you can't learn it or can't be bothered to learn it, you don't need to go anywhere near a database because you can't be bothered to understand the right tool. Granted, nosql has some uses, but those are few and far between and probably would have been handled more effectively and easily with old approaches like Prevlayer, but Prevlayer was a thousand lines of code and couldn't be monetized.
  - taeric 9y ago
atomical 9y ago
Most data is relational. If you put relational data in a NoSQL database you end up writing an API that is similar to SQL.
- SmellTheGlove 9y ago
  Yup, and you move data logic from a stable, baked solution (your RDBMS of choice) to the application layer. Best case, you end up with something as well-implemented as SQL, but you're maintaining it as part of the app logic. Might as well let the database do it.
- Clubber 9y ago
  Most structured data is relational. If you include data like (someone else's) web site content, images, sound files, video files, etc, it's a different picture (pardon the pun).
  - LeoPanthera 9y ago
    How is NoSQL better than simply using the filesystem?
    - Clubber 9y ago
      I'm not an expert on NoSQL, but based on general computer knowledge I would say some advantages over a filesystem would be: 1. Single entry point. 2. Not having to share your filesystem. 3. Centralized security separated from your OS 4. Simple key/value access (no folder structure) 5. Abstraction of the filesystem (like #2) 6. Distribution of data over multiple nodes. 7. Separation of NoSQL admin and network admins. 8. Shiney new thing :) I'm sure now a days, an advanced filesystem/sharing protocol would handle some of these.
      - dom0 9y ago
        The biggest advantage is that you get transactions. (Or Not). Possibly faster small objects with less (memory, storage) overhead.
SmellTheGlove 9y ago
This was a nice read for the history. Honestly, being in a big legacy industry (Insurance), it's as though NoSQL never happened. We're too big, at least my organization, to have made the wholesale change and we've been plugging along mostly in Teradata and DB2 for a long time. Teradata in particular has performed well across a variety of use cases, its only large downside being the cost. Anyhow, only point I'd add is that I'm not sure that SQL-like languages added on top of NoSQL solutions was really that bad of a thing. It's not as though there's one flavor of SQL to rule them all on RDBMSs. I get that the bones are largely the same - SELECT, FROM, WHERE, most join types, aggregates, etc. But take a really common analytic use case - persisted transactional data. Something like this - SELECT A, B, C, FROM SomeTbl WHERE SomeDate BETWEEN SomeBegin AND SomeEnd QUALIFY RANK() OVER (PARTITION BY A, B ORDER BY SomeOtherDate DESC) = 1 Pretty sure that SELECT, FROM, WHERE will work just about anywhere. That QUALIFY clause is written for Teradata, and once you get into the ordered analytic functions (as one example), you need to know the implementation relevant to the RDBMS you're dealing with. It's not uncommon for a large enterprise to have MS SQL, DB2, Teradata, Oracle, etc. all under one roof. As a data engineer, you still need to know the differences - first to get the data out, and then to optimize. Dunno, I could be way off on this, but that's been my experience at least. I think the bigger issue is that you can use an RDBMS to do NoSQL-like things, such as key-value stores, with the flexibility to structure your data if you need that later. So why not start with a relational database?
- Clubber 9y ago
  >So why not start with a relational database? I asked the same question. The biggest reason I've heard is RDBMS's don't horizontally scale well, meaning you can't easily have 50 replicated nodes across the globe and expect it to perform well, or setup easily, because it's fairly complicated with an RDBMS. There's things like Oracle's grid or SQL's high availability clusters, but they get complicated fast, particularly when you need a bunch of nodes. Obviously it was a solution for a very large amount of unstructured or semi-structured data that needed to be redundant across networks. With NoSQL, the key/values are a lot simpler to replicate apparantly. It's a Google/Twitter size problem that most industries wouldn't have, but since it's the new shiny thing, you know how that goes.
myrandomcomment 9y ago
I started in tech at IBM supporting the finance clients. When I moved to an ISP in the early 90s I was dumb struck by the fact that things broke and it was okay. Oh the Usenet server failed again....etc...it never sat well with me. Look SNA was complex, but uptime on the system I support was measured in decades. I have done a ton of startups since then and while I love the speed at which we move the falgility of the system always rubbed me wrong. Looking at the need for speed (mongo, et.al) was always painful. I am happy to see that rational thought is winning (which is how I see this).
jugg1es 9y ago
Just like anything in the tech world, it all depends on what you are trying to do. There are scenarios where nosql makes more sense and cases where relational databases make more sense. There is no panacea in regards to data storage.
ChicagoDave 9y ago
I had the task of replicating data from DynamoDB to Redshift on AWS on two large projects on the last few years. The primary factor in using DynamoDB is that is's cheap and scales like nothing else in the galaxy. Then the other shoe fell. The data was needed for reports. Reports require a static schema. NoSQL (and the developers who love it) despises static schema. "I can add properties whenever I want!" This process of analyzing and reporting on production data becomes a very time-consuming, costly, and brittle exercise. So then you have to determine, did we save enough money on the design side (using NoSQL over SQL) and then piss it away on reporting? I'd argue AWS and other cloud providers need to create a SQL capable relational database as a service. This would (I hope) solve the problem. But in the meantime, let's build our micro-services on relational databases so we can actually get aggregate data to stakeholders in real-time.
- MBCook 9y ago
  > I'd argue AWS and other cloud providers need to create a SQL capable relational database as a service. This would (I hope) solve the problem. Isn’t that Aurora?
  - nkozyra 9y ago
    It's RDS in general, which supports a number of RDBMS.
  - PKop 9y ago
    OP probably means not worrying about instances, scaling or over/under provisioning etc, just service usage at a more fine grained level.. more like Spanner
    - ChicagoDave 9y ago
      Exactly. I don't want to care about the server or the hardware or the OS. Just give me a database with tables and standard SQL querying capabilities and performance.
      - 9y ago
pjungwir 9y ago
I've always found it a little funny that SQL was originally designed for non-programmers, sort of like AppleScript. I used to think neither of those panned out, but in fact there really are a lot of smart not-programmers who can use it. At a company I work with many of the support staff have been learning SQL to help customers pull reports from our data warehousey reporting database. So maybe the article is onto something about the value of the language. On the other hand, SQL-the-language isn't essential to relational databases. I have often wondered where C.J. Date has been the last few years. I actually love SQL, but it does have its limitations. I wouldn't mind a solid relational database with an alternative query language. It's such a missed opportunity for a great VC pitch: Tutorial D, NoSQL before it was cool. :-)
- Clubber 9y ago
  Also, SQL was designed back when most programming was done in assembler or C. Compared to assembler, SQL is easy. Programming itself has gotten a lot easier since.
  - skissane 9y ago
    SQL was designed by IBM in the 1970s, to run on mainframes. The community by whom and for whom it was developed was focused on PL/I and COBOL and largely didn't use C. The first implementation of SQL in C was probably Oracle V3 (released in 1983). IBM's equivalent SQL offerings around the same time (SQL/DS for VM/CMS and DB2 for MVS) were written in PL/I dialects and assembly, and COBOL, Fortran and PL/I were supported as application languages. I believe IBM's first forays into using C with SQL was the development of OS/2 EE Database Manager in C, somewhat later in the 1980s. (The current DB2 Linux/Unix/Windows code base is originally descended from OS/2 EE Database Manager.)
    - Clubber 9y ago
      I was going by Oracle's "first commercial implementation," as a starting point. I found it it was originally written in Fortran, but later ported to C. I wouldn't have guessed fortran. https://asktom.oracle.com/pls/asktom/f?p=100:11:0::NO::P11_QUESTION_ID:5228516811673 https://asktom.oracle.com/pls/asktom/f?p=100:11:0::NO::P11_Q... Oracle was originally written in fortran and then redone in C, which it has been written in ever since. In fact, when we ported Oracle to the mainframe for the first time we were faced with an issue -- do we rewrite Oracle in assembler (there was no c compiler on the mainframe back then) or do we write a C compiler? We wrote a C compiler. On a side note, I remember watching an interview or reading an article (probably from Larry Ellison) who said IBM gave 1/3 of it's business to Intel, another 1/3 to Microsoft and the final 1/3 to Oracle. Pretty crazy.
deleted 9y ago
[deleted]
systematical 9y ago
For me the author nailed it when he brought up the PostgreSQL JSON datatype. I just couldn't ever get into MongoDB, but recognized the value of JSON structure data. Perfect balance for me.
gsylvie 9y ago
SSD's.
- jjirsa 9y ago
  If SSDs solve your scaling problem, you never had a scaling problem
  - gsylvie 9y ago
    NVMe SSD's are 100x faster than 15k disks. How many shops run 100+ machine NoSQL instances?
    - jjirsa 9y ago
      Lots? Certainly most adtech firms and IOT type firms Also: scaling isn't just speed, it's also capacity, HA, failover. Are you running 100x4T NVME in a single server? Edit: fun little story. Some time back, I wrote a compaction strategy for Cassandra designed for time-series data. That strategy (known as TWCS) is now widely deployed, but for the first year or so, it was an uphill battle to get it accepted into the project over the existing options (high barrier to entry for fairly obvious reasons). I had given a talk at a conference and made the code available online - my employer was going to use it, so giving it out was a good way to help iron out any wrinkles, and maybe some early adopters could help me get it into the project. There was a Fortune 500 company who sent me one of their products - unsolicited - as a thank-you, because they had deployed my compaction strategy, and it apparently helped them a lot. I asked how much, and their answer was something like "About $20k/month". In my mind, I thought that was great - probably let them decommission somewhere in the range of 5-20 servers, which sounded pretty good to me. Then they said "per cluster, and there's 10 clusters so far". Now, the $200k/month savings is cool and all, but the real point is think about how many machines are involved here to be able to SAVE $200k/month just by changing the way data files are combined.
ww520 9y ago
Way back when I took my first database class by Mike Stonebraker, he talked of some of the old war stories between RDBMS and network-based database, which was hot before RDBMS. He said the relational model by Date and Codd had won out back then, for the simple fact that relational data stood by themselves while the network-based data were tightly coupled with the applications. When NoSQL came around, yep, it looked like the old network-based database again. History has repeated itself. Relational model, where SQL is merely the querying language, will win again and again, for the simple fact that it enables data to stand by themselves, and data tend to outlive applications, both in longevity and in scope. Edit: PostgreSQL came from Postgres, the research project started by Stonebraker and others in UCB.
- elvinyung 9y ago
  Stonebraker wrote MapReduce: A Huge Step Backwards [1], which basically tells exactly the story you mentioned, the "Hadoop is Codasyl" comparison. [1] https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...
lmickh 9y ago
I like how it pretty much just glosses over decades of familiarity. If there is anything the last 20 years prove, it is that the majority of developers will stick with what they know over what might be a good fit for the job. It goes even deeper in the SQL world down to the specific database flavor. From the ops side I actually find RDBMS more difficult to deal with cause the power of relationships is easy to abuse and they are not anti-fragile. Instead of smartly reasoning about the data, it is all to easy to just "JOIN ALL THE THINGS WITH MEGA TEMP TABLES!". I've taken more database outages from bad queries then anything else. There are bad implementations on both sides. There are reasons to pick both sides over the other given a set a circumstances. At the end of the day, the technical facts don't matter to most people's decision making though.
- nkozyra 9y ago
  > it is that the majority of developers will stick with what they know over what might be a good fit for the job Anecdotally, I have seen this shift quite a bit in the last decade. There has been a big move toward "try whatever is new and shiny" and I think a lot of that drove the NoSQL craze. And don't get me wrong, there are some great NoSQL options if you select the right tool for the right job. But a lot of people who gravitated to NoSQL did so because they designed poor queries in RDBMS. Guilty myself. Often the quick reaction was "wow, this is faster" instead of "well of course it's faster, I'm not getting X, Y and Z features of a relational database. Do I need those? Did I abuse the RDBMS?" etc.
- viraptor 9y ago
  > Instead of smartly reasoning about the data, it is all to easy to just "JOIN ALL THE THINGS WITH MEGA TEMP TABLES!". This is so common, I'd love for the popular databases to add table flags that prevent it by accident. Letting me configure "this table most not full scan or file sort implicitly" would get rid of half the incident callouts I've been involved in. You could always override it in the query where needed.
  - paulddraper 9y ago
    MySQL has a slow query log with all non-indexed queries.
randomdrake 9y ago
I don't buy it. The makers of the software that this appears to be a carefully written advertisement for, came to the same conclusion as the rest of the IT world (that some of us saw a mile away): NoSQL was, and still is, only good for very specific things in very specific cases; it's generally dreadful for anything that SQL engines could already do well. The subtitle for this contains: "After years of being left for dead" and the author throws phrases around like: "And boy did the software developer community eat up NoSQL, embracing it arguably much more broadly than the original Google/Amazon authors intended." Who? Where's the data? This blog post has a lot of links, references, and studies, but where's the data to back up the premise? A quick search on Google Trends comparing SQL databases to NoSQL as an entire term, or any of the popular flavors of NoSQL, reveals that it is not even a blip in comparison. But don't take my word for it, the author and their company had the "DUH" moment too (emphasis mine): > ...we soon realized that we’d have to do a lot more work: e.g., deciding syntax, building various connectors, educating users, etc. We also found ourselves constantly looking up the proper syntax to queries that we could already express in SQL, for a query language we had written ourselves! One day we realized that building our own query language made no sense. That the key was to embrace SQL. You might have had hubris stemming from discarding or not knowing all of the history that you decided to share with us in this blog post. A great deal of the NoSQL community was completely unpalatable for this reason. The folks who had been doing data for decades, and built stable and powerful systems on top of many prior decades of mistakes, built them for a reason. And here we see the author trying to proclaim that suddenly SQL is back? SQL never went anywhere. NoSQL is a neat tool that was developed and continues to be developed and probably isn't going anywhere. But the idea that NoSQL suddenly overtook SQL, and now SQL is seeing some huge resurgence, feels like it comes from the perspective of someone who only saw a window into the last 6-8 years of development. The king is dead. Long live the king.
- paulddraper 9y ago
  I think if there were data to support the authors perception, it would be number of blog posts written.
- Clubber 9y ago
  It's a marketing piece, no doubt, but he is explicitly talking about SQL the language on top of NoSQL, not SQL as in RDBMS. But to your point SQL the language or RDBMS was never dead, it just wasn't the prettiest girl in the room for all the new GI's that just shipped in.
fadzlan 9y ago
Based from what I read of the article, its more about SQL the query language instead of the normal ACID compliant RDBMS over NoSQL databases. Of course, having a common query language is a big advantage, but NoSQL varies in paradigms and implementation due to the subset of problems they choose to solve, thus having a standard query language does not make sense for NoSQL.
shepardrtc 9y ago
SQL never really left the table. People just like to try out new things and talk a lot about them. Its actually a pretty great feature of tech culture; we don't accept that everything is perfect, we always want to try for something better. And in this case, people tried to have a database with a different and potentially "easier" paradigm. Nothing wrong with that at all. But its hard to compete with years and years of tuning and optimization that SQL databases have gone through.
emodendroket 9y ago
Because it's better in every single way except maybe for massive scale, a problem fewer people have than think they have, would be my answer to that question.
- nimchimpsky 9y ago
  And NoSQL is only better for massive scale when you don't need to do anything with the data.
craigkerstiens 9y ago
I don't think the article could have said it much better. SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them. Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) were talking about JSON like it's fad and how it's going to come to pass. They were saying so because they'd heard about this disruption before... there was this new thing XML and these document databases were so much more powerful. They waited a few years and added an XML datatype [1]... And then XML databases came and passed. At first they scoffed a bit on JSON as the new hip thing, but came around a little in Postgres 9.2, and then in 9.4 [2] we all really got what we wanted. SQL has never been a particularly elegant language, but it's always been a powerful one. The lone knock against has either been usability or scalability [3]. On the usability side that's a bit of a hard one as well it's not super pretty. But once you understand it's foundations around relational algebra or relational calculus you're so far ahead. On the scalability side there are definitely some options such as Citus, but also core Postgres is massively improving here. [1] https://www.postgresql.org/docs/9.3/static/datatype-xml.html https://www.postgresql.org/docs/9.3/static/datatype-xml.html [2] https://blog.codeship.com/unleash-the-power-of-storing-json-in-postgres/ https://blog.codeship.com/unleash-the-power-of-storing-json-... [3] https://www.citusdata.com/blog/2017/02/16/citus61-released/ https://www.citusdata.com/blog/2017/02/16/citus61-released/
- unclebucknasty 9y ago
  I think I'm one of those graybeards. I see it in so many things tech. It's a pattern, and once you've seen it repeat a half-dozen times and also gain a depth of experience over that time, you can actually recognize when something represents genuine progress vs yet another passing fad. Spoiler alert: those that are most rabidly promoted are often the latter. But, if you try to raise the point in the midst of the latest fad, you generally get shouted down. So, you wait until the less-jaded figure it out...again. It was plainly obvious for NoSQl, just as it now is for SPAs (or at least our current approach). Don't believe me? Wait 5 years. And, the thing with SQL that's so much like other tech is that we largely figured it out years ago. Improvements have been incremental at best and, frequently, not improvements at all.
methodin 9y ago
No tool is good when abused. NoSQL was a knee-jerk reaction and the truth lies somewhere in the middle. Putting everything in relational databases is as bad as putting none of it there. If everything is treated like a tool the world becomes brighter at the cost of having to learn and understand more.
aneutron 9y ago
By no means am I an expert, but here's the thing. You can't say "Cars are the new thing. Airplanes came to be but now they're not the shiny thing anymore." You don't take your car to go intercontinent, and you don't take the plane to go a couple of kilometers. One thing NoSQL databases excel at, is scaling. Scaling in Cassandra for example means adding a node, everything else is just magic. THAT is what made MySQL and the SQL language popular. SQL may be popular with developers, but scaling MySQL was a nightmare for quite some time, even with the introduction of vitesse etc. Point is, please do not compare apples to oranges.
- nimchimpsky 9y ago
  I disagree with this. And MySQL != SQL
- emodendroket 9y ago
  I don't honestly think many people have the kind of scale where this is a sensible trade-off to make.
  - sk5t 9y ago
    Absolutely. Even a bigint / long sequence column so often feels a bit ridiculous.
    - shub 9y ago
      I have 5 tables which have overrun the range of int in the id column. Happily they were defined bigint. A while ago one ran out of tinyints, but doing the schema change...was pretty terrible actually due to the foreign keys. The tinyint decision wasn't me, I want to make clear. New tables all get bigint. Either you don't have many rows, so you can spare the space, or you have a lot of rows and the thought of changing the type of your primary key gives you a queasy feeling in your belly.
      - oldandtired 9y ago
        This is only a problem because of the laziness of those who have built the DBMS. Not necessarily the programmers fault but those who mandate the requirements of the DBMS (usually managers). If you specify that a field is an integer, you shouldn't have to care how big that integer is. That is a concern for the DBMS and should not be yours. In point of fact, your only concern should be what fields you are using not how they are to be stored. You should never ever have to concern yourself with whether a field is a tiny_int, small_int, integer or big_int. Any changes to the sizes that the DBMS has chosen for specific relations should be automatically propagated by the DBMS to any other area that refers to the original field. It should never require intervention by a DBA or programmer, ever. Over the decades, I have read lots of recommendations for DBA's about how to manage the various databases stored in the DBMS. Most of these recommendations should have been put into the DBMS as a part of the attributes by which the DBMS controlled the database. These recommendations were often portrayed as a result of running various sets of queries and other functions in specific orders. All of them could have been automated out of existence, but to keep the DBMS simple, they forced these processes onto the DBA's and database designers. Since we know what has to be done, surely the DBMS producers (like IBM, ORACLE, Microsoft, etc.) should be able to do so as a part of their product development.
jonbaer 9y ago
A bit surprised to find no mention of the GPU database field (ie: https://www.mapd.com/demos/ https://www.mapd.com/demos/, etc) because straight SQL on this hardware is extremely fast when optimized. Feels like all this NoSQL stuff came around when developers no longer wanted to deal w/ DBAs and do full stack(s) on their own. DBAs spent long hours on DDL and DML (stored procedures) for a good reason.
- deleted 9y ago
  [deleted]
- arnon 9y ago
  I'd mention an actual capable SQL database a lot before MapD... Like SQream or Kinetica
bischofs 9y ago
The author seems to conflate sql with single node classical rdbms'. The query language side of things hasn't changed much but coming from automotive which is currently generating daily terabytes of long and flat data in the form of time series streams with very little relational characteristics nosql systems are very useful. I think there is a tendency to go after old relational systems because of how they handled themselves with the data that was generated mostly by users. At this point my industry is dealing with data generated by machines and nature and thus the need for extremely scalable and distributed systems. Whether these systems use sql i don't think is that important ( most of them will support sql or sql like )
known 9y ago
https://en.wikipedia.org/wiki/Comparison_of_structured_storage_software https://en.wikipedia.org/wiki/Comparison_of_structured_stora...
odammit 9y ago
Heard a great quote once, "If you don't understand SQL, you probably need SQL."
jondubois 9y ago
this article is overly opinionated and narrow-minded. Many startups use NoSQL. Personally I prefer NoSQL for most use cases. That said I have nothing against SQL itself but I will NEVER go back to using an ORM - They're a dirty hack; always have been and always will be. I'd rather write the SQL by hand. I love using MongoDB; it's very simple to use. I liked RethinkDB's ReQL even more (even though the company itself had to shut down, the open source project is still great). The main problem with SQL databases is that the powerful features offered by SQL don't scale; so if you want to build a future-proof system, you have to avoid using certain features like joins; the advantage of NoSQL is that it's built to scale; the constraints are are feature.
- arkh 9y ago
  > you have to avoid using certain features like joins Too bad it is kinda sorta the thing with a RDBM. Relationnal. When you have to work with data which are related and don't want to do stupid shit like updating the infos of an author in each of its written article you want SQL. But nothing prevents you from using a NoSQL database on top or even multiple ones. If you have multiple way to use your data, having multiple read or write models is a plus. About scale: rare are the people who can't just add more hardware to the problem.
- naranha 9y ago
  I really agree with you. I think ORM is responsible for a lot of failed software projects, since it leads to over-engineering. I saw a lot of projects where 90% of the mapped columns or tables could be replaced with JSON, because they were simply never queried against. One thing I learned writing SPAs is that joins are often unnecessary because with proper state management a lot of data is already present on the client side. NoSQL Query languages (for example Mongo or CouchDB queries) are more than sufficient for these use cases.
- rayascott 9y ago
  Well said. It's amazing how many engineers will make sweeping generalisations like it's a provable fact. You'd think from their training, they'd know better. But hey, he's got a blog and service, and everyone listen up now.
skc 9y ago
I like SQL. I've never failed to do anything with it and find it very powerful. But while that aspect of the SQL experience I'm comfortable with, I'm still just a developer and not a DBA. I don't even _want_ to be a DBA. And as someone putting the pieces of a startup together in a country where DBA skills are practically non-existent (meaning I can't reliably train someone up myself) a cloud hosted NoSQL solution seems a much better bet on paper. I've noticed that when the topic of SQL vs NoSQL comes up, the issue of cost of infrastructure maintenance always seems to get glossed over.
HumanDrivenDev 9y ago
NoSQL is a really terrible term. I don't know why so many people seem eager to the argue the merits of an incredibly disparate umbrella label that includes databases that have almost nothing in common. What meaningful things can you say about a category that includes Cassandara, Datalog, LDMB and Neo4J? I think when a lot of people talk about NoSQL they just want to rant against a certain kind of strawman programmer. You know the one. Young, stupid, naiive, too arrogant to learn nth normal form or define schemas. This programmer probably uses nodejs or some other such heresy and only wants quick results, integrity be damned! Don't get me wrong, Relational Databases are really good, and fit a lot of problems really well. But there do exist legitimate use cases (not necessarily scale!) where an RDBMS will simply be a poor fit, or a lot more work. Don't dredge up the strawman of the programmer too arrogant to learn SQL because you're too arogant to learn the merits of something that isn't SQL.
sgt 9y ago
What are the implications for apps being designed in Firebase?
sandGorgon 9y ago
Nobody has mentioned Postgres 10 which is going to be pretty cool in its release - https://www.ashnik.com/new-features-coming-in-postgresql-10/ https://www.ashnik.com/new-features-coming-in-postgresql-10/ Postgres is creeping up on Cassandra faster than Cassandra becomes usable. Especially the really cool work that Citus is doing.
- ddorian43 9y ago
  Nah, postgresql still doesn't have a nice/easy sharding policy. Don't forget that citus is still single-master (the opensource one). Postgresql-xl also doesn't have nice/easy replication. Scylladb is creeping up on Cassandra though.
  - sandGorgon 9y ago
    citus is all opensource now - https://www.citusdata.com/blog/2016/03/24/citus-unforks-goes-open-source/ https://www.citusdata.com/blog/2016/03/24/citus-unforks-goes... https://github.com/citusdata/citus https://github.com/citusdata/citus Which is why Postgres is creeping up on both scylladb and cassandra. Plus PG 10 will have declarative partitioning built in. Pretty cool.
    - ddorian43 9y ago
      1. I know it's open-source, but they have a `citus-mx` product which has multiple masters which isn't open-source (currently you have to do coordinators manually). 2. Kind hard to creep up on scylladb by seeing it's architecture and assuming you use it for what it's best at, the asyncness and raw performance.
primeblue 9y ago
NoSQL is practically useless for succinct and efficient queries
memsom 9y ago
About 15 years ago, I worked on a team with a product that used an object based database. No SQL didn't exist, but it was kind of in a similar vein. The database was a pure object storage and there was an index b-tree to make sense of it. We could store anything at all in the database. As long as it inherited from a specific class. It used the object serialization features of the language (Object Pascal) to read and write the data chunks. It had a query layer that allowed collection interrogation of data with specific values set. But, it was also very problematic. The structure was fairly chaotic. Ignoring the specific implementation, it taught me that databases - despite the rigidity of the storage mechanism, were a bloody good way to keep data organised. And SQL, despite the warts and inconsistencies of the major dialects, is actually a very powerful way to query data. When you strip away the integrity and the ability to easily pull data out in a uniform way, it makes things a lot less fluid.
- reboog711 9y ago
  What makes you think SQL didn't exist 15 years ago? SQL has been around since 1974 according to Wikipedia. There was a ton of SQL Work during the first dot com era of the mid-90s. On another note, I did a lot of Lotus Notes working during that time, which was clearly a no-SQL database.
  - sb8244 9y ago
    Author means that NoSQL as a term / fad didn't exist
    - reboog711 9y ago
      I see that now. His space between no and SQL confused me.
  - deleted 9y ago
    [deleted]
- le-mark 9y ago
  It's almost as if "everyone" forgot that before SQL (relational database) there was only NoSQL databases of various sorts (hierarchical, document, etc) and SQL and the relational model arose to address their shortcomings. Personally, I knew stuff had gotten stupid when I sat through a presentation by a gemfire evangalist who advised everyone present to just "do your joins in code". If you need to join data you should be using SQL.
crimsonalucard 9y ago
If NoSQL never worked because of a lack of a common interface, then switching back to SQL is just giving up on finding a real solution because SQL is one of the big mistakes in web development. Allow me to elucidate: We query the database in a "query language" that is declarative and based on a expression. This SQL expression is then compiled down into imperative instructions. This is where the mistake lies: SQL is a leaky abstraction. We understand algorithms, we understand how to manipulate the Big-O of imperative instructions to get better runtimes, but in order to understand how to optimize SQL we have to understand how this "expression" compiles down into imperative code. There's no rhyme or reason for why "SELECT STAR" is bad unless you know beforehand the instructions it compiles down to. Actually, you don't even need to understand the instructions "SELECT STAR" compiles down to, you can build an entire career (DB admins) around just remembering that and other random rules as optimization hacks. The bottleneck in web development is the database. The reason why we can use python or ruby as application servers instead super fast C++ apps is because the database will always be way slower. So it would make sense to optimize the database, yet decades of imperative algorithm theory is thrown out the window because you can't query the database imperatively. Instead of optimizing the bottleneck, we write out web apps in imperative code and our database queries in highly abstract leaky expressions. The web app is not the bottleneck. Having a highly abstract functional language as the web app is perfectly ok as you tend to not have the web app run through complex search algorithms anyway. Instead we decide to write our webapps in optimizable imperative code and our bottlenecked database in SQL. It's all backwards. I'm thinking the mistake with NOSQL was the fact that the databases weren't relational. Can we make a relational database with a query language that is NOSQL and imperative?
mirekrusin 9y ago
Why, again, this obsession with absolute statements? A doean't need to be "a killer" of B; Redis is a great NoSQL database and it occupies specific place in the stack, graph databases have their own, invaluable traversal query languages and SQL databases are perfect for many other cases. Nobody needs to "beat to death" or "be the only future". I'm a bit bored with this hyperbolic, nonsense rhetoric.
joeminichino 9y ago
the whole debate always seems very confused to me. SQL refers to the language with which you interact with a database, NoSQL refers to the database system itself and it's normally used as opposed to traditional RDBSs. So really the debate should be either SQL vs NON-SQL interfaces or RDBS vs NoSQL databases. Also NoSQL meaning "not only SQL" means SQL can be a subset of NoSQL, so the debate is over. Really we should review the terminology of this entire subject...
smarx007 9y ago
I am really surprised that while everyone talks about SQL vs NoSQL, nobody has mentioned RDF as a model and SPARQL as a language. Graph-like structures based on triples allow the data relations to be represented properly (relational properties), while not limiting the data structure. This is not a shiny hot thing, but instead something developed over many years by (mostly) academics. Take a look at the tutorial: https://www.w3.org/TR/2014/NOTE-rdf11-primer-20140225/ https://www.w3.org/TR/2014/NOTE-rdf11-primer-20140225/ And just to prove that RDF is a model, but not a format, look at JSON-LD as a serialisation format for RDF and at the SQLGraph [1] paper from Google to see how RDF can be implemented on top of an SQL RDBMS. [1]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43287.pdf https://static.googleusercontent.com/media/research.google.c...
- bobdc 9y ago
  SPARQL is also a standard from a well-known standards body with plenty of open source and commercial implementations, and it's quite popular at many commercial enterprises: http://www.sparql.club http://www.sparql.club
zimablue 9y ago
It feels like everyone mixes relational databases with SQL in discussions. Relational databases are fine but SQL is disgusting. It's a query language that's very hard to parse or generate and has multiple incompatible variants. Ironically the Lambda calculus version looks much better, I'd never seen those before. I wish we'd go back to that as default, unify around a standard and expose block text SQL as an extra for users.
- jeremiep 9y ago
  The one downside I see with SQL is that its a string. Would it be a data structure instead it would be incredibly easier to manipulate, without third party query builders. Even without that its still more convenient than most NoSQL. Still, Datalog as implemented by Datomic is a breath of fresh air.
  - zimablue 9y ago
    It's more than just being a string though, it's the multiple incompatible versions and the stunted mini language of it's own. A good solution would be able to run arbitrary expressions in some "real" programming language. It also in my experience doesn't seem to deliver on its central premise of converting declarative expressions to fast operations. So often in a SP I've had to force a write to an intermediate table to jimmy the execution plan. I think it's like excel, it's not considered good because the thing itself is good it seems good because the thing it does is very useful (lightweight interactive dataflow programming, talking to relational databases) and it has a near monopoly. LINQ is really cool but doesn't exist in python (macropython has an unused attempt PINQ I think). Probably because a) SQL is bad b) only MS actually has the definition of the mssql language!
- oldandtired 9y ago
  Not quite, the confusion is that SQL DBMS's are assumed to be relational when they are not. None of the major players have produced a relational DBMS (there is one proviso, but IBM didn't promote that specific product out of the UK over its DB2 product out of the USA).
amigoingtodie 9y ago
Larry Ellison ate IBM's lunch, using Codd's idea (which was funded by IBM). The math is solid, right?
newsmania 9y ago
everybody hates on Mongodb, but at least it's sql-injection proof.
turowicz 9y ago
Comparing SQL to NoSQL is like comparing a boat to a car. They serve different purpose and each come with their own benefits and disadvantages.
mr_overalls 9y ago
SQL has all of the benefits mentioned in the article, but a few small changes to the ANSI standard - mostly taking design cues from C#'s LINQ - would make it much more usable and modern as a query language. https://www.linqpad.net/WhyLINQBeatsSQL.aspx https://www.linqpad.net/WhyLINQBeatsSQL.aspx Also, it's such a small thing, but why not place the SELECT clause after the FROM clause? This would allow for easier auto-completion help from the query editor.
Glyptodon 9y ago
SQL is great. If it needs anything it's built in tools for data expiration and a means of creating results that are more complexly structured than just rows/columns, though both can be worked around.
bogomipz 9y ago
The premise that SQL is "beating" NoSQL is pretty dumb. I have never been aware that such a competition between datastore designs exists. Who is that competition between developers? Users? Most of the places I have worked in the last 5 or 6 years have had both relational and non-relational databases. This is not uncommon. In none of those shops was there a competition between the two databases but rather they were complimentary. This article and title seem to be very self-serving for their own product. Its seems to willfully conflate SQL the interface and SQL a general term for a relational database. NoSQL has always been something of a misnomer - non-relational would have been a better term but it doesn't sound as buzz-worthy I suppose. The SQL like interface has been in Cassandra for longer now than it was absent. The gain in SQL like interfaces for non-relational databases is because it's familiar and works really well. Anywhere there is a database there is going to be a need for selection, filtering and projections of tuples. >"In Amazon’s own words, its PostgreSQL- and MySQL-compatible database Aurora database product has been the “fastest growing service in the history of AWS”. Is this really surprising that the world's largest cloud provider is selling more databases than anything else? Almost everyone needs a database, given howrelational database, there are more people around that have experience with it. >"To understand why SQL is making a comeback ..." No, SQL never went away. Full stop. >"But don’t take our word for it. Take Google’s" No, this practice of blindly adopting ideas just because they work for Google needs to stop. This post sounds as if the author(s) themselves bought into all of the NoSQL hype that buzzword-obsessed tech journalists were spinning and they are just now having an epiphany that much of that hype was just that.
- geebee 9y ago
  "I have never been aware that such a competition between datastore designs exists" Good! There certainly shouldn't be. Wish I could say the same. Magpie developers, people who want to seem edgy and forward looking, people who like making others look backwards at meetings, people who are worried they'll look like dinosaurs, people who've been around the block long enough to know an approach is totally unnecessary for their project but will be required on the resume for the next project. Trust me, they were all out there, in force. And of course, many data stores that aren't SQL are absolutely great for the task. Try to solve the 6 degrees of separation problem with SQL joins, then do it with a graph database. There are all kinds of applications that weren't a great fit for a relational model, and many that are. This article does mention the moment where NoSql was defined as "Not Always SQL" (and dismisses it with "nice try"), but honestly, if this had been the approach up front, it would have saved us a lot of grief. Probably. Maybe. I hope. Oh, who am I kidding. All that stuff I described above would have happened anyway.
AzzieElbab 9y ago
Why is no one stating the obvious? 1) hardware improvments along with db hosted solutions simplified sql management and scaling 2) most data is simply useless or of very temporary value. Having said that, I do not think sql can compete with specialized solutions when it comes to unstructured or semi-structured data like EMR for example. Also, with notable exception of datalog/datomic NoSQL solutions remain kV stores hacked together by some horrible programming models
TheVikingOwain 9y ago
I always get a little tick when I read the nosql is a new thing. Pick databases like Unidata/Universe and mvBase have been around for a long time and experienced a lot of the problems that newer nosql implementations have/had.
botskonet 9y ago
I've long struggled with NoSQL. My data is relational, and I've been long taught to normalize and de-dup my data, so how do I use this? I keep feeling like I'm missing the genius because I keep coming back to wishing I had a true relational table. However, there have been times where being forced to define a schema has been painful. If our app has to store dynamic keys/values it all winds up as records in a relational db table acting like key/value store. Whereas in Mongo we can just store the document, without affecting the entire table schema. I want the best of both worlds...
mrjn 9y ago
SQL is not beating NoSQL. NoSQL is evolving. https://blog.dgraph.io/post/sql-vs-nosql/ https://blog.dgraph.io/post/sql-vs-nosql/

405 points by nreece 9y ago

There's really no such thing as NoSQL first of all. Relational, graph, key/value (including wide column), document-store, etc. There are lots of database types but it turns out relational works 95% of the time and we're getting better at recognizing the use-cases for the others. SQL is also just a query language, that's literally the name. Any database can implement it, not just relational. Is SQL a great interface for both OLTP and OLAP? Yes, it's proven itself over decades. Is it the only valid interface? No. Does it work with other newer/different data systems? Yes, Spark, Hadoop, Kafka, and even these new distributed relational databases are all examples of such. It would be far better for the industry if we can get past these ideological and faulty "debates" and move on to more important things.

jevgeni 9y ago
Thank you!

takeda 9y ago

Great article, but I think the author puts too much emphasis on SQL instead on the fact that the data being stored in a relational way. Personally I actually would like an alternative language that would be capable to be integrated with the language in such way that it could also fall with type checking. So for example if I rename a column in a database, the type checker would highlight all places in my code that were broken by this change. JOOQ[1] seems to do something like that, but it's only for Java. Also, looks like QUEL[2] would be a bit easier to be integrated with a language, too bad it died. [1] http://www.jooq.org/ http://www.jooq.org/ [2] https://en.wikipedia.org/wiki/QUEL_query_languages https://en.wikipedia.org/wiki/QUEL_query_languages

labster 9y ago

If SQL is to databases as Javascript is to browsers — ubiquitous and largely standard across the market — are there any languages that transpile to SQL? People in this thread have commented that SQL is clunky, and JS definitely fit that description for a while. So I'm wondering if there are any alternatives that would prove SQL's clunkiness to me.

yenwel 9y ago
LINQ?
- deleted 9y ago
  [deleted]
evv555 9y ago
Ecto for Elixir might fit that description. Queries are composable unlike SQL.
amirouche 9y ago
ORMs transpile to SQL. Just like JavaScript is not perfect, SQL is not perfect. I would rather have a general purpose programming language like Scheme to run queries against my database.
bigger_cheese 9y ago
I'm an engineer (the non software kind) that works in an industrial plant. I often have to do ad-hoc querying manipulation etc of data from a lot of diverse sources. Usually to do fault finding and analytics - sometimes also for reporting. I don't know about transpiling but I use SAS to work around SQL's clunkiness. I can embed SQL seamlessly into my SAS code so it works well. I use SQL for things that are simpler in SQL (selecting, filtering, aggregating data and "case when" statements etc) and I use SAS's data step language for when I have to do more complicated stuff with the data like Arrays, loops, macros, weighted averaging, time series based stuff and regressions. The thing I love about SQL is it is the lowest common denominator for all of my organisations data it doesn't mater if data is stored in an Oracle table, a Microsoft SQL Server table, or even Access DB or spreadsheet. The language to extract/merge/join all the data is the same and standardized. We also have things called "Data historians" which are basically time-series datastores (usually they are linked to PLC vendors) and nothing frustrates me more when they use some proprietary "sql like" language which seems similar to ANSI SQL except things like aggregations (SUM, AVG, MAX etc) don't work or they do weird things with data types (especially dates) or similar everyone of these seem to be subtly different - annoying.
9y ago

russellbeattie 9y ago

Wow... I wish I could just wait a few years and have everyone come to their senses on other things as well.

amirouche 9y ago

What is good about SQL is not the programming theory behind it, or the fact that it's supposed to be simple enough for non-programmers. Just like JavaScript, the good thing about SQL is that it's a standard.

nextInt 9y ago

Lol SQL is making a comeback? It never left.

SQL2219 9y ago
This.
krf 9y ago
It only left in terms of attention from startups I suppose. I also disagree with the reason the article suggested - SQL could not handle the loads. My opinion is that startups simply liked the idea of not having a schema as it fit their agile approach. So, they went NoSQL because it allowed them to get going faster and change easier.
- xirdstl 9y ago
  That, and they like to believe they need to scale like Facebook.
  - Myrmornis 9y ago
    Sadly, this is a common reason.
wernercd 9y ago
SQL is dead. Long live SQL.
20years 9y ago
Exactly! SQL was never dead.

Clubber 9y ago

So the long and short of it is SQL would be a good common language for relational and NoSQL style databases. I would agree with that, but since they different monsters (RDBMS and NoSQL), it will take a bit of tweaking to find a good dialect (how do joins work, etc). Of course it makes sense to adapt the NoSQL databases to the existing ANSI SQL rather than make existing ANSI SQL users switch to a new type of SQL that accommodates NoSQL, but we'll see what happens. Google create a "Standard SQL," but I'm not familiar with it. Here is a link on Google Standard SQL: https://cloud.google.com/bigquery/docs/reference/standard-sql/migrating-from-legacy-sql https://cloud.google.com/bigquery/docs/reference/standard-sq...

nkozyra 9y ago
> So the long and short of it is SQL would be a good common language for relational and NoSQL style databases. Yes, I'm sure the inevitable "SQL for NoSQL Databases" book will be a good read :)

mmaunder 9y ago

It's not one versus the other. That's like suggesting JSON is going to 'beat' HTML. They're different tools for different jobs and both are used alongside each other to great effect.

wildchild 9y ago

Because nosql is hype and snake-oil. Barely usable tools to dump startups on ignorant investors.

peterwwillis 9y ago

SQL wins because big iron wins. At the end of the day, the big, clunky, slow, complicated, aggravating, old systems stick around because they are built to stick around. Looking at, say, an eventually consistent distributed decentralized kv store, one might be tricked into believing it's simple enough to deploy with enough nodes and general enough interfaces that you can build a complicated system on top of it if needed, and rely on its supppsed simplicity and scalability the rest of the time. But nobody tells you about the shitty implementation. The replication that doesn't resume, the stodgy transfers, the imbalanced distribution, the consensus conflict, the infinitely expanding disk, the churning CPU. How at scale, if all the other aspects of your infra aren't scaling along with your data size, the whole thing blows. Traditional SQL databases end up being many times simpler in practice, and because of their inherent scaling limitations, much easier to manage. And most importantly: their implementations aren't buggy. SQL is just more reliable.

wjossey 9y ago
As someone who helped design, then manage a massive NoSQL store which hundreds of terabytes of data, of which most of it was hot, I’ll humbly disagree and say it’s not necessarily as bad as you make it sound. Did we have someone who managed that system? Sometimes, but mostly it just did it’s own thing. We ironically invested way more time on our MySQL database over the years because we couldn’t get that to scale the way we wanted to, but I think that was specific to a problem we were having. Did we later invest a lot of time in that system (the NoSQL one)? Yes, because it was very cost effective for us to do so. At certain scale throwing people at optimization problems can pay huge dividends. But, this can be said of most infrstracture. It’s usually worth revisiting every year or two and seeing how what you can squeeze out. Did we have churning CPU, infinitely expanding disk, replication issues, and more? Sure, but not very commonly and mostly it was fairly easily resolved. More importantly though, it was a solid system that was the underpinning of a colossal system, and it behaved admiraly more than 99.9% of the time. Will most projects benefit from a hugely distributed KV store? Nope. But I’m still glad they exist!
- cachemiss 9y ago
  Caveat, this comment isn't directed at you (I agree with your comment), but rather the points around what you are saying. One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated. When talking about stores, it's important to be explicit about a few things: 1. Storage model 2. Distribution model 3. Access model 4. Transaction model 5. Maturity and competence of implementation What happens is people talk about "SQL" as either an NSM or DSM storage model, over either a single node, or possibly more than that in some of the MPP systems, using SQL as an access model, with linearizable transactions, and a mature competent implementation. NoSQL when most people refer to it can be any combination of those things, as long as the access model isn't SQL. I work on database engines, and it's important to decouple these things and be explicit about them when discussing various tradeoffs. You can do SQL the language over a distributed k/v store (not always a great idea) and other non-tabular / relational models and you can distribute relational engines (though scaling linearizable transactions is difficult and doesn't scale for certain use cases due to physics, but that's unrelated to the relational part of it). Generally people talk about joins not scaling in some normalized form, but then what they do is just materialize the join into whatever they are using to store things in a denormalized model, which has its own drawbacks. As to the comment above you, SQL vs NoSQL also doesn't have anything to do with the relative maturity of anything. Some of the newer non-relational engines have some operational issues, but that doesn't really have anything to do with their storage model or access method, it just has to due with the competence of the implementation. MongoDB is difficult operationally not because it's not a relational engine, but because it wasn't well designed. Just like people put SQL over non-tabular stores, you can build non-tabular / relational engines over relational engines (sharding PostgreSQL etc.). In fact major cloud vendors do just that.

taeric 9y ago

I do find the amusing "backronym" rather hard to take at face value. I also found the old critics of SQL databases odd focus on the query language hard to take. The problem wasn't the queries, per se. The problem was the power of the queries for interactive uses and how that doesn't scale up. That is, the problem with most SQL databases is that you need some pretty specialized knowledge in order to construct good queries. Not shockingly, the problem with most modern key/value (or otherwise) datastores is that you need some pretty specialized knowledge in order to construct good queries. Now, for things getting off the ground, this is probably fine. Most of the places you will go south with queries is in the ad-hoc style query. Of course, that style query is perfect for interactive use. Ideal, even. It falls on its face if it is supposed to support an automated case at high TPS. Unfortunately, automated cases at low TPS often turn into automated cases at high TPS. Worse, interactive cases at ridiculously low tps often turn into automated cases at low TPS. Which, of course, just feeds itself. How this feeds itself, of course, is you can explore your data much more effectively if there is an ad-hoc query engine. So, we are now seeing the resurgence of ad-hoc queries and the learning that those can lead to some powerful insights.

aaron-lebo 9y ago
The thing is ORMs have made writing easy queries easy and hard queries possible since at the very least 2005 with Rails, and they are easily extensible and composable. It's mostly a solved issue. If you want to do complex data manipulation, surprise, it takes a powerful tool. Seems a lot of the hype is the result of VC $ + naiveity/overselling about what MongoDB and co would be capable of. The marketing is natural - MongoDB made databases "easy" and easy is the only way to sell lowest common denominator tech to become a unicorn. Bad incentives for all involved. Sql is a kind of rite of passage? If you can't learn it or can't be bothered to learn it, you don't need to go anywhere near a database because you can't be bothered to understand the right tool. Granted, nosql has some uses, but those are few and far between and probably would have been handled more effectively and easily with old approaches like Prevlayer, but Prevlayer was a thousand lines of code and couldn't be monetized.
- taeric 9y ago

atomical 9y ago

Most data is relational. If you put relational data in a NoSQL database you end up writing an API that is similar to SQL.

SmellTheGlove 9y ago
Yup, and you move data logic from a stable, baked solution (your RDBMS of choice) to the application layer. Best case, you end up with something as well-implemented as SQL, but you're maintaining it as part of the app logic. Might as well let the database do it.
Clubber 9y ago
Most structured data is relational. If you include data like (someone else's) web site content, images, sound files, video files, etc, it's a different picture (pardon the pun).
- LeoPanthera 9y ago
  How is NoSQL better than simply using the filesystem?
  - Clubber 9y ago
    I'm not an expert on NoSQL, but based on general computer knowledge I would say some advantages over a filesystem would be: 1. Single entry point. 2. Not having to share your filesystem. 3. Centralized security separated from your OS 4. Simple key/value access (no folder structure) 5. Abstraction of the filesystem (like #2) 6. Distribution of data over multiple nodes. 7. Separation of NoSQL admin and network admins. 8. Shiney new thing :) I'm sure now a days, an advanced filesystem/sharing protocol would handle some of these.
    - dom0 9y ago
      The biggest advantage is that you get transactions. (Or Not). Possibly faster small objects with less (memory, storage) overhead.

SmellTheGlove 9y ago

This was a nice read for the history. Honestly, being in a big legacy industry (Insurance), it's as though NoSQL never happened. We're too big, at least my organization, to have made the wholesale change and we've been plugging along mostly in Teradata and DB2 for a long time. Teradata in particular has performed well across a variety of use cases, its only large downside being the cost. Anyhow, only point I'd add is that I'm not sure that SQL-like languages added on top of NoSQL solutions was really that bad of a thing. It's not as though there's one flavor of SQL to rule them all on RDBMSs. I get that the bones are largely the same - SELECT, FROM, WHERE, most join types, aggregates, etc. But take a really common analytic use case - persisted transactional data. Something like this - SELECT A, B, C, FROM SomeTbl WHERE SomeDate BETWEEN SomeBegin AND SomeEnd QUALIFY RANK() OVER (PARTITION BY A, B ORDER BY SomeOtherDate DESC) = 1 Pretty sure that SELECT, FROM, WHERE will work just about anywhere. That QUALIFY clause is written for Teradata, and once you get into the ordered analytic functions (as one example), you need to know the implementation relevant to the RDBMS you're dealing with. It's not uncommon for a large enterprise to have MS SQL, DB2, Teradata, Oracle, etc. all under one roof. As a data engineer, you still need to know the differences - first to get the data out, and then to optimize. Dunno, I could be way off on this, but that's been my experience at least. I think the bigger issue is that you can use an RDBMS to do NoSQL-like things, such as key-value stores, with the flexibility to structure your data if you need that later. So why not start with a relational database?

Clubber 9y ago
>So why not start with a relational database? I asked the same question. The biggest reason I've heard is RDBMS's don't horizontally scale well, meaning you can't easily have 50 replicated nodes across the globe and expect it to perform well, or setup easily, because it's fairly complicated with an RDBMS. There's things like Oracle's grid or SQL's high availability clusters, but they get complicated fast, particularly when you need a bunch of nodes. Obviously it was a solution for a very large amount of unstructured or semi-structured data that needed to be redundant across networks. With NoSQL, the key/values are a lot simpler to replicate apparantly. It's a Google/Twitter size problem that most industries wouldn't have, but since it's the new shiny thing, you know how that goes.

myrandomcomment 9y ago

I started in tech at IBM supporting the finance clients. When I moved to an ISP in the early 90s I was dumb struck by the fact that things broke and it was okay. Oh the Usenet server failed again....etc...it never sat well with me. Look SNA was complex, but uptime on the system I support was measured in decades. I have done a ton of startups since then and while I love the speed at which we move the falgility of the system always rubbed me wrong. Looking at the need for speed (mongo, et.al) was always painful. I am happy to see that rational thought is winning (which is how I see this).

jugg1es 9y ago

Just like anything in the tech world, it all depends on what you are trying to do. There are scenarios where nosql makes more sense and cases where relational databases make more sense. There is no panacea in regards to data storage.

ChicagoDave 9y ago

I had the task of replicating data from DynamoDB to Redshift on AWS on two large projects on the last few years. The primary factor in using DynamoDB is that is's cheap and scales like nothing else in the galaxy. Then the other shoe fell. The data was needed for reports. Reports require a static schema. NoSQL (and the developers who love it) despises static schema. "I can add properties whenever I want!" This process of analyzing and reporting on production data becomes a very time-consuming, costly, and brittle exercise. So then you have to determine, did we save enough money on the design side (using NoSQL over SQL) and then piss it away on reporting? I'd argue AWS and other cloud providers need to create a SQL capable relational database as a service. This would (I hope) solve the problem. But in the meantime, let's build our micro-services on relational databases so we can actually get aggregate data to stakeholders in real-time.

MBCook 9y ago
> I'd argue AWS and other cloud providers need to create a SQL capable relational database as a service. This would (I hope) solve the problem. Isn’t that Aurora?
- nkozyra 9y ago
  It's RDS in general, which supports a number of RDBMS.
- PKop 9y ago
  OP probably means not worrying about instances, scaling or over/under provisioning etc, just service usage at a more fine grained level.. more like Spanner
  - ChicagoDave 9y ago
    Exactly. I don't want to care about the server or the hardware or the OS. Just give me a database with tables and standard SQL querying capabilities and performance.
    - 9y ago

pjungwir 9y ago

I've always found it a little funny that SQL was originally designed for non-programmers, sort of like AppleScript. I used to think neither of those panned out, but in fact there really are a lot of smart not-programmers who can use it. At a company I work with many of the support staff have been learning SQL to help customers pull reports from our data warehousey reporting database. So maybe the article is onto something about the value of the language. On the other hand, SQL-the-language isn't essential to relational databases. I have often wondered where C.J. Date has been the last few years. I actually love SQL, but it does have its limitations. I wouldn't mind a solid relational database with an alternative query language. It's such a missed opportunity for a great VC pitch: Tutorial D, NoSQL before it was cool. :-)

Clubber 9y ago
Also, SQL was designed back when most programming was done in assembler or C. Compared to assembler, SQL is easy. Programming itself has gotten a lot easier since.
- skissane 9y ago
  SQL was designed by IBM in the 1970s, to run on mainframes. The community by whom and for whom it was developed was focused on PL/I and COBOL and largely didn't use C. The first implementation of SQL in C was probably Oracle V3 (released in 1983). IBM's equivalent SQL offerings around the same time (SQL/DS for VM/CMS and DB2 for MVS) were written in PL/I dialects and assembly, and COBOL, Fortran and PL/I were supported as application languages. I believe IBM's first forays into using C with SQL was the development of OS/2 EE Database Manager in C, somewhat later in the 1980s. (The current DB2 Linux/Unix/Windows code base is originally descended from OS/2 EE Database Manager.)
  - Clubber 9y ago
    I was going by Oracle's "first commercial implementation," as a starting point. I found it it was originally written in Fortran, but later ported to C. I wouldn't have guessed fortran. https://asktom.oracle.com/pls/asktom/f?p=100:11:0::NO::P11_QUESTION_ID:5228516811673 https://asktom.oracle.com/pls/asktom/f?p=100:11:0::NO::P11_Q... Oracle was originally written in fortran and then redone in C, which it has been written in ever since. In fact, when we ported Oracle to the mainframe for the first time we were faced with an issue -- do we rewrite Oracle in assembler (there was no c compiler on the mainframe back then) or do we write a C compiler? We wrote a C compiler. On a side note, I remember watching an interview or reading an article (probably from Larry Ellison) who said IBM gave 1/3 of it's business to Intel, another 1/3 to Microsoft and the final 1/3 to Oracle. Pretty crazy.

deleted 9y ago

[deleted]

systematical 9y ago

For me the author nailed it when he brought up the PostgreSQL JSON datatype. I just couldn't ever get into MongoDB, but recognized the value of JSON structure data. Perfect balance for me.

gsylvie 9y ago

SSD's.

jjirsa 9y ago
If SSDs solve your scaling problem, you never had a scaling problem
- gsylvie 9y ago
  NVMe SSD's are 100x faster than 15k disks. How many shops run 100+ machine NoSQL instances?
  - jjirsa 9y ago
    Lots? Certainly most adtech firms and IOT type firms Also: scaling isn't just speed, it's also capacity, HA, failover. Are you running 100x4T NVME in a single server? Edit: fun little story. Some time back, I wrote a compaction strategy for Cassandra designed for time-series data. That strategy (known as TWCS) is now widely deployed, but for the first year or so, it was an uphill battle to get it accepted into the project over the existing options (high barrier to entry for fairly obvious reasons). I had given a talk at a conference and made the code available online - my employer was going to use it, so giving it out was a good way to help iron out any wrinkles, and maybe some early adopters could help me get it into the project. There was a Fortune 500 company who sent me one of their products - unsolicited - as a thank-you, because they had deployed my compaction strategy, and it apparently helped them a lot. I asked how much, and their answer was something like "About $20k/month". In my mind, I thought that was great - probably let them decommission somewhere in the range of 5-20 servers, which sounded pretty good to me. Then they said "per cluster, and there's 10 clusters so far". Now, the $200k/month savings is cool and all, but the real point is think about how many machines are involved here to be able to SAVE $200k/month just by changing the way data files are combined.

ww520 9y ago

Way back when I took my first database class by Mike Stonebraker, he talked of some of the old war stories between RDBMS and network-based database, which was hot before RDBMS. He said the relational model by Date and Codd had won out back then, for the simple fact that relational data stood by themselves while the network-based data were tightly coupled with the applications. When NoSQL came around, yep, it looked like the old network-based database again. History has repeated itself. Relational model, where SQL is merely the querying language, will win again and again, for the simple fact that it enables data to stand by themselves, and data tend to outlive applications, both in longevity and in scope. Edit: PostgreSQL came from Postgres, the research project started by Stonebraker and others in UCB.

elvinyung 9y ago
Stonebraker wrote MapReduce: A Huge Step Backwards [1], which basically tells exactly the story you mentioned, the "Hadoop is Codasyl" comparison. [1] https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...

lmickh 9y ago

I like how it pretty much just glosses over decades of familiarity. If there is anything the last 20 years prove, it is that the majority of developers will stick with what they know over what might be a good fit for the job. It goes even deeper in the SQL world down to the specific database flavor. From the ops side I actually find RDBMS more difficult to deal with cause the power of relationships is easy to abuse and they are not anti-fragile. Instead of smartly reasoning about the data, it is all to easy to just "JOIN ALL THE THINGS WITH MEGA TEMP TABLES!". I've taken more database outages from bad queries then anything else. There are bad implementations on both sides. There are reasons to pick both sides over the other given a set a circumstances. At the end of the day, the technical facts don't matter to most people's decision making though.

nkozyra 9y ago
> it is that the majority of developers will stick with what they know over what might be a good fit for the job Anecdotally, I have seen this shift quite a bit in the last decade. There has been a big move toward "try whatever is new and shiny" and I think a lot of that drove the NoSQL craze. And don't get me wrong, there are some great NoSQL options if you select the right tool for the right job. But a lot of people who gravitated to NoSQL did so because they designed poor queries in RDBMS. Guilty myself. Often the quick reaction was "wow, this is faster" instead of "well of course it's faster, I'm not getting X, Y and Z features of a relational database. Do I need those? Did I abuse the RDBMS?" etc.
viraptor 9y ago
> Instead of smartly reasoning about the data, it is all to easy to just "JOIN ALL THE THINGS WITH MEGA TEMP TABLES!". This is so common, I'd love for the popular databases to add table flags that prevent it by accident. Letting me configure "this table most not full scan or file sort implicitly" would get rid of half the incident callouts I've been involved in. You could always override it in the query where needed.
- paulddraper 9y ago
  MySQL has a slow query log with all non-indexed queries.

randomdrake 9y ago

I don't buy it. The makers of the software that this appears to be a carefully written advertisement for, came to the same conclusion as the rest of the IT world (that some of us saw a mile away): NoSQL was, and still is, only good for very specific things in very specific cases; it's generally dreadful for anything that SQL engines could already do well. The subtitle for this contains: "After years of being left for dead" and the author throws phrases around like: "And boy did the software developer community eat up NoSQL, embracing it arguably much more broadly than the original Google/Amazon authors intended." Who? Where's the data? This blog post has a lot of links, references, and studies, but where's the data to back up the premise? A quick search on Google Trends comparing SQL databases to NoSQL as an entire term, or any of the popular flavors of NoSQL, reveals that it is not even a blip in comparison. But don't take my word for it, the author and their company had the "DUH" moment too (emphasis mine): > ...we soon realized that we’d have to do a lot more work: e.g., deciding syntax, building various connectors, educating users, etc. We also found ourselves constantly looking up the proper syntax to queries that we could already express in SQL, for a query language we had written ourselves! One day we realized that building our own query language made no sense. That the key was to embrace SQL. You might have had hubris stemming from discarding or not knowing all of the history that you decided to share with us in this blog post. A great deal of the NoSQL community was completely unpalatable for this reason. The folks who had been doing data for decades, and built stable and powerful systems on top of many prior decades of mistakes, built them for a reason. And here we see the author trying to proclaim that suddenly SQL is back? SQL never went anywhere. NoSQL is a neat tool that was developed and continues to be developed and probably isn't going anywhere. But the idea that NoSQL suddenly overtook SQL, and now SQL is seeing some huge resurgence, feels like it comes from the perspective of someone who only saw a window into the last 6-8 years of development. The king is dead. Long live the king.

paulddraper 9y ago
I think if there were data to support the authors perception, it would be number of blog posts written.
Clubber 9y ago
It's a marketing piece, no doubt, but he is explicitly talking about SQL the language on top of NoSQL, not SQL as in RDBMS. But to your point SQL the language or RDBMS was never dead, it just wasn't the prettiest girl in the room for all the new GI's that just shipped in.

fadzlan 9y ago

Based from what I read of the article, its more about SQL the query language instead of the normal ACID compliant RDBMS over NoSQL databases. Of course, having a common query language is a big advantage, but NoSQL varies in paradigms and implementation due to the subset of problems they choose to solve, thus having a standard query language does not make sense for NoSQL.

shepardrtc 9y ago

SQL never really left the table. People just like to try out new things and talk a lot about them. Its actually a pretty great feature of tech culture; we don't accept that everything is perfect, we always want to try for something better. And in this case, people tried to have a database with a different and potentially "easier" paradigm. Nothing wrong with that at all. But its hard to compete with years and years of tuning and optimization that SQL databases have gone through.

emodendroket 9y ago

Because it's better in every single way except maybe for massive scale, a problem fewer people have than think they have, would be my answer to that question.

nimchimpsky 9y ago
And NoSQL is only better for massive scale when you don't need to do anything with the data.

craigkerstiens 9y ago

I don't think the article could have said it much better. SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them. Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) were talking about JSON like it's fad and how it's going to come to pass. They were saying so because they'd heard about this disruption before... there was this new thing XML and these document databases were so much more powerful. They waited a few years and added an XML datatype [1]... And then XML databases came and passed. At first they scoffed a bit on JSON as the new hip thing, but came around a little in Postgres 9.2, and then in 9.4 [2] we all really got what we wanted. SQL has never been a particularly elegant language, but it's always been a powerful one. The lone knock against has either been usability or scalability [3]. On the usability side that's a bit of a hard one as well it's not super pretty. But once you understand it's foundations around relational algebra or relational calculus you're so far ahead. On the scalability side there are definitely some options such as Citus, but also core Postgres is massively improving here. [1] https://www.postgresql.org/docs/9.3/static/datatype-xml.html https://www.postgresql.org/docs/9.3/static/datatype-xml.html [2] https://blog.codeship.com/unleash-the-power-of-storing-json-in-postgres/ https://blog.codeship.com/unleash-the-power-of-storing-json-... [3] https://www.citusdata.com/blog/2017/02/16/citus61-released/ https://www.citusdata.com/blog/2017/02/16/citus61-released/

unclebucknasty 9y ago
I think I'm one of those graybeards. I see it in so many things tech. It's a pattern, and once you've seen it repeat a half-dozen times and also gain a depth of experience over that time, you can actually recognize when something represents genuine progress vs yet another passing fad. Spoiler alert: those that are most rabidly promoted are often the latter. But, if you try to raise the point in the midst of the latest fad, you generally get shouted down. So, you wait until the less-jaded figure it out...again. It was plainly obvious for NoSQl, just as it now is for SPAs (or at least our current approach). Don't believe me? Wait 5 years. And, the thing with SQL that's so much like other tech is that we largely figured it out years ago. Improvements have been incremental at best and, frequently, not improvements at all.

methodin 9y ago

No tool is good when abused. NoSQL was a knee-jerk reaction and the truth lies somewhere in the middle. Putting everything in relational databases is as bad as putting none of it there. If everything is treated like a tool the world becomes brighter at the cost of having to learn and understand more.

aneutron 9y ago

By no means am I an expert, but here's the thing. You can't say "Cars are the new thing. Airplanes came to be but now they're not the shiny thing anymore." You don't take your car to go intercontinent, and you don't take the plane to go a couple of kilometers. One thing NoSQL databases excel at, is scaling. Scaling in Cassandra for example means adding a node, everything else is just magic. THAT is what made MySQL and the SQL language popular. SQL may be popular with developers, but scaling MySQL was a nightmare for quite some time, even with the introduction of vitesse etc. Point is, please do not compare apples to oranges.

nimchimpsky 9y ago
I disagree with this. And MySQL != SQL
emodendroket 9y ago
I don't honestly think many people have the kind of scale where this is a sensible trade-off to make.
- sk5t 9y ago
  Absolutely. Even a bigint / long sequence column so often feels a bit ridiculous.
  - shub 9y ago
    I have 5 tables which have overrun the range of int in the id column. Happily they were defined bigint. A while ago one ran out of tinyints, but doing the schema change...was pretty terrible actually due to the foreign keys. The tinyint decision wasn't me, I want to make clear. New tables all get bigint. Either you don't have many rows, so you can spare the space, or you have a lot of rows and the thought of changing the type of your primary key gives you a queasy feeling in your belly.
    - oldandtired 9y ago
      This is only a problem because of the laziness of those who have built the DBMS. Not necessarily the programmers fault but those who mandate the requirements of the DBMS (usually managers). If you specify that a field is an integer, you shouldn't have to care how big that integer is. That is a concern for the DBMS and should not be yours. In point of fact, your only concern should be what fields you are using not how they are to be stored. You should never ever have to concern yourself with whether a field is a tiny_int, small_int, integer or big_int. Any changes to the sizes that the DBMS has chosen for specific relations should be automatically propagated by the DBMS to any other area that refers to the original field. It should never require intervention by a DBA or programmer, ever. Over the decades, I have read lots of recommendations for DBA's about how to manage the various databases stored in the DBMS. Most of these recommendations should have been put into the DBMS as a part of the attributes by which the DBMS controlled the database. These recommendations were often portrayed as a result of running various sets of queries and other functions in specific orders. All of them could have been automated out of existence, but to keep the DBMS simple, they forced these processes onto the DBA's and database designers. Since we know what has to be done, surely the DBMS producers (like IBM, ORACLE, Microsoft, etc.) should be able to do so as a part of their product development.

jonbaer 9y ago

A bit surprised to find no mention of the GPU database field (ie: https://www.mapd.com/demos/ https://www.mapd.com/demos/, etc) because straight SQL on this hardware is extremely fast when optimized. Feels like all this NoSQL stuff came around when developers no longer wanted to deal w/ DBAs and do full stack(s) on their own. DBAs spent long hours on DDL and DML (stored procedures) for a good reason.

deleted 9y ago
[deleted]
arnon 9y ago
I'd mention an actual capable SQL database a lot before MapD... Like SQream or Kinetica

bischofs 9y ago

The author seems to conflate sql with single node classical rdbms'. The query language side of things hasn't changed much but coming from automotive which is currently generating daily terabytes of long and flat data in the form of time series streams with very little relational characteristics nosql systems are very useful. I think there is a tendency to go after old relational systems because of how they handled themselves with the data that was generated mostly by users. At this point my industry is dealing with data generated by machines and nature and thus the need for extremely scalable and distributed systems. Whether these systems use sql i don't think is that important ( most of them will support sql or sql like )

known 9y ago

https://en.wikipedia.org/wiki/Comparison_of_structured_storage_software https://en.wikipedia.org/wiki/Comparison_of_structured_stora...

odammit 9y ago

Heard a great quote once, "If you don't understand SQL, you probably need SQL."

jondubois 9y ago

this article is overly opinionated and narrow-minded. Many startups use NoSQL. Personally I prefer NoSQL for most use cases. That said I have nothing against SQL itself but I will NEVER go back to using an ORM - They're a dirty hack; always have been and always will be. I'd rather write the SQL by hand. I love using MongoDB; it's very simple to use. I liked RethinkDB's ReQL even more (even though the company itself had to shut down, the open source project is still great). The main problem with SQL databases is that the powerful features offered by SQL don't scale; so if you want to build a future-proof system, you have to avoid using certain features like joins; the advantage of NoSQL is that it's built to scale; the constraints are are feature.

arkh 9y ago
> you have to avoid using certain features like joins Too bad it is kinda sorta the thing with a RDBM. Relationnal. When you have to work with data which are related and don't want to do stupid shit like updating the infos of an author in each of its written article you want SQL. But nothing prevents you from using a NoSQL database on top or even multiple ones. If you have multiple way to use your data, having multiple read or write models is a plus. About scale: rare are the people who can't just add more hardware to the problem.
naranha 9y ago
I really agree with you. I think ORM is responsible for a lot of failed software projects, since it leads to over-engineering. I saw a lot of projects where 90% of the mapped columns or tables could be replaced with JSON, because they were simply never queried against. One thing I learned writing SPAs is that joins are often unnecessary because with proper state management a lot of data is already present on the client side. NoSQL Query languages (for example Mongo or CouchDB queries) are more than sufficient for these use cases.
rayascott 9y ago
Well said. It's amazing how many engineers will make sweeping generalisations like it's a provable fact. You'd think from their training, they'd know better. But hey, he's got a blog and service, and everyone listen up now.

skc 9y ago

I like SQL. I've never failed to do anything with it and find it very powerful. But while that aspect of the SQL experience I'm comfortable with, I'm still just a developer and not a DBA. I don't even _want_ to be a DBA. And as someone putting the pieces of a startup together in a country where DBA skills are practically non-existent (meaning I can't reliably train someone up myself) a cloud hosted NoSQL solution seems a much better bet on paper. I've noticed that when the topic of SQL vs NoSQL comes up, the issue of cost of infrastructure maintenance always seems to get glossed over.

HumanDrivenDev 9y ago

NoSQL is a really terrible term. I don't know why so many people seem eager to the argue the merits of an incredibly disparate umbrella label that includes databases that have almost nothing in common. What meaningful things can you say about a category that includes Cassandara, Datalog, LDMB and Neo4J? I think when a lot of people talk about NoSQL they just want to rant against a certain kind of strawman programmer. You know the one. Young, stupid, naiive, too arrogant to learn nth normal form or define schemas. This programmer probably uses nodejs or some other such heresy and only wants quick results, integrity be damned! Don't get me wrong, Relational Databases are really good, and fit a lot of problems really well. But there do exist legitimate use cases (not necessarily scale!) where an RDBMS will simply be a poor fit, or a lot more work. Don't dredge up the strawman of the programmer too arrogant to learn SQL because you're too arogant to learn the merits of something that isn't SQL.

sgt 9y ago

What are the implications for apps being designed in Firebase?

sandGorgon 9y ago

Nobody has mentioned Postgres 10 which is going to be pretty cool in its release - https://www.ashnik.com/new-features-coming-in-postgresql-10/ https://www.ashnik.com/new-features-coming-in-postgresql-10/ Postgres is creeping up on Cassandra faster than Cassandra becomes usable. Especially the really cool work that Citus is doing.

ddorian43 9y ago
Nah, postgresql still doesn't have a nice/easy sharding policy. Don't forget that citus is still single-master (the opensource one). Postgresql-xl also doesn't have nice/easy replication. Scylladb is creeping up on Cassandra though.
- sandGorgon 9y ago
  citus is all opensource now - https://www.citusdata.com/blog/2016/03/24/citus-unforks-goes-open-source/ https://www.citusdata.com/blog/2016/03/24/citus-unforks-goes... https://github.com/citusdata/citus https://github.com/citusdata/citus Which is why Postgres is creeping up on both scylladb and cassandra. Plus PG 10 will have declarative partitioning built in. Pretty cool.
  - ddorian43 9y ago
    1. I know it's open-source, but they have a `citus-mx` product which has multiple masters which isn't open-source (currently you have to do coordinators manually). 2. Kind hard to creep up on scylladb by seeing it's architecture and assuming you use it for what it's best at, the asyncness and raw performance.

primeblue 9y ago

NoSQL is practically useless for succinct and efficient queries

memsom 9y ago

About 15 years ago, I worked on a team with a product that used an object based database. No SQL didn't exist, but it was kind of in a similar vein. The database was a pure object storage and there was an index b-tree to make sense of it. We could store anything at all in the database. As long as it inherited from a specific class. It used the object serialization features of the language (Object Pascal) to read and write the data chunks. It had a query layer that allowed collection interrogation of data with specific values set. But, it was also very problematic. The structure was fairly chaotic. Ignoring the specific implementation, it taught me that databases - despite the rigidity of the storage mechanism, were a bloody good way to keep data organised. And SQL, despite the warts and inconsistencies of the major dialects, is actually a very powerful way to query data. When you strip away the integrity and the ability to easily pull data out in a uniform way, it makes things a lot less fluid.

reboog711 9y ago
What makes you think SQL didn't exist 15 years ago? SQL has been around since 1974 according to Wikipedia. There was a ton of SQL Work during the first dot com era of the mid-90s. On another note, I did a lot of Lotus Notes working during that time, which was clearly a no-SQL database.
- sb8244 9y ago
  Author means that NoSQL as a term / fad didn't exist
  - reboog711 9y ago
    I see that now. His space between no and SQL confused me.
- deleted 9y ago
  [deleted]
le-mark 9y ago
It's almost as if "everyone" forgot that before SQL (relational database) there was only NoSQL databases of various sorts (hierarchical, document, etc) and SQL and the relational model arose to address their shortcomings. Personally, I knew stuff had gotten stupid when I sat through a presentation by a gemfire evangalist who advised everyone present to just "do your joins in code". If you need to join data you should be using SQL.

crimsonalucard 9y ago

If NoSQL never worked because of a lack of a common interface, then switching back to SQL is just giving up on finding a real solution because SQL is one of the big mistakes in web development. Allow me to elucidate: We query the database in a "query language" that is declarative and based on a expression. This SQL expression is then compiled down into imperative instructions. This is where the mistake lies: SQL is a leaky abstraction. We understand algorithms, we understand how to manipulate the Big-O of imperative instructions to get better runtimes, but in order to understand how to optimize SQL we have to understand how this "expression" compiles down into imperative code. There's no rhyme or reason for why "SELECT STAR" is bad unless you know beforehand the instructions it compiles down to. Actually, you don't even need to understand the instructions "SELECT STAR" compiles down to, you can build an entire career (DB admins) around just remembering that and other random rules as optimization hacks. The bottleneck in web development is the database. The reason why we can use python or ruby as application servers instead super fast C++ apps is because the database will always be way slower. So it would make sense to optimize the database, yet decades of imperative algorithm theory is thrown out the window because you can't query the database imperatively. Instead of optimizing the bottleneck, we write out web apps in imperative code and our database queries in highly abstract leaky expressions. The web app is not the bottleneck. Having a highly abstract functional language as the web app is perfectly ok as you tend to not have the web app run through complex search algorithms anyway. Instead we decide to write our webapps in optimizable imperative code and our bottlenecked database in SQL. It's all backwards. I'm thinking the mistake with NOSQL was the fact that the databases weren't relational. Can we make a relational database with a query language that is NOSQL and imperative?

mirekrusin 9y ago

Why, again, this obsession with absolute statements? A doean't need to be "a killer" of B; Redis is a great NoSQL database and it occupies specific place in the stack, graph databases have their own, invaluable traversal query languages and SQL databases are perfect for many other cases. Nobody needs to "beat to death" or "be the only future". I'm a bit bored with this hyperbolic, nonsense rhetoric.

joeminichino 9y ago

the whole debate always seems very confused to me. SQL refers to the language with which you interact with a database, NoSQL refers to the database system itself and it's normally used as opposed to traditional RDBSs. So really the debate should be either SQL vs NON-SQL interfaces or RDBS vs NoSQL databases. Also NoSQL meaning "not only SQL" means SQL can be a subset of NoSQL, so the debate is over. Really we should review the terminology of this entire subject...

smarx007 9y ago

I am really surprised that while everyone talks about SQL vs NoSQL, nobody has mentioned RDF as a model and SPARQL as a language. Graph-like structures based on triples allow the data relations to be represented properly (relational properties), while not limiting the data structure. This is not a shiny hot thing, but instead something developed over many years by (mostly) academics. Take a look at the tutorial: https://www.w3.org/TR/2014/NOTE-rdf11-primer-20140225/ https://www.w3.org/TR/2014/NOTE-rdf11-primer-20140225/ And just to prove that RDF is a model, but not a format, look at JSON-LD as a serialisation format for RDF and at the SQLGraph [1] paper from Google to see how RDF can be implemented on top of an SQL RDBMS. [1]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43287.pdf https://static.googleusercontent.com/media/research.google.c...

bobdc 9y ago
SPARQL is also a standard from a well-known standards body with plenty of open source and commercial implementations, and it's quite popular at many commercial enterprises: http://www.sparql.club http://www.sparql.club

zimablue 9y ago

It feels like everyone mixes relational databases with SQL in discussions. Relational databases are fine but SQL is disgusting. It's a query language that's very hard to parse or generate and has multiple incompatible variants. Ironically the Lambda calculus version looks much better, I'd never seen those before. I wish we'd go back to that as default, unify around a standard and expose block text SQL as an extra for users.

jeremiep 9y ago
The one downside I see with SQL is that its a string. Would it be a data structure instead it would be incredibly easier to manipulate, without third party query builders. Even without that its still more convenient than most NoSQL. Still, Datalog as implemented by Datomic is a breath of fresh air.
- zimablue 9y ago
  It's more than just being a string though, it's the multiple incompatible versions and the stunted mini language of it's own. A good solution would be able to run arbitrary expressions in some "real" programming language. It also in my experience doesn't seem to deliver on its central premise of converting declarative expressions to fast operations. So often in a SP I've had to force a write to an intermediate table to jimmy the execution plan. I think it's like excel, it's not considered good because the thing itself is good it seems good because the thing it does is very useful (lightweight interactive dataflow programming, talking to relational databases) and it has a near monopoly. LINQ is really cool but doesn't exist in python (macropython has an unused attempt PINQ I think). Probably because a) SQL is bad b) only MS actually has the definition of the mssql language!
oldandtired 9y ago
Not quite, the confusion is that SQL DBMS's are assumed to be relational when they are not. None of the major players have produced a relational DBMS (there is one proviso, but IBM didn't promote that specific product out of the UK over its DB2 product out of the USA).

amigoingtodie 9y ago

Larry Ellison ate IBM's lunch, using Codd's idea (which was funded by IBM). The math is solid, right?

newsmania 9y ago

everybody hates on Mongodb, but at least it's sql-injection proof.

turowicz 9y ago

Comparing SQL to NoSQL is like comparing a boat to a car. They serve different purpose and each come with their own benefits and disadvantages.

mr_overalls 9y ago

SQL has all of the benefits mentioned in the article, but a few small changes to the ANSI standard - mostly taking design cues from C#'s LINQ - would make it much more usable and modern as a query language. https://www.linqpad.net/WhyLINQBeatsSQL.aspx https://www.linqpad.net/WhyLINQBeatsSQL.aspx Also, it's such a small thing, but why not place the SELECT clause after the FROM clause? This would allow for easier auto-completion help from the query editor.

Glyptodon 9y ago

SQL is great. If it needs anything it's built in tools for data expiration and a means of creating results that are more complexly structured than just rows/columns, though both can be worked around.

bogomipz 9y ago

The premise that SQL is "beating" NoSQL is pretty dumb. I have never been aware that such a competition between datastore designs exists. Who is that competition between developers? Users? Most of the places I have worked in the last 5 or 6 years have had both relational and non-relational databases. This is not uncommon. In none of those shops was there a competition between the two databases but rather they were complimentary. This article and title seem to be very self-serving for their own product. Its seems to willfully conflate SQL the interface and SQL a general term for a relational database. NoSQL has always been something of a misnomer - non-relational would have been a better term but it doesn't sound as buzz-worthy I suppose. The SQL like interface has been in Cassandra for longer now than it was absent. The gain in SQL like interfaces for non-relational databases is because it's familiar and works really well. Anywhere there is a database there is going to be a need for selection, filtering and projections of tuples. >"In Amazon’s own words, its PostgreSQL- and MySQL-compatible database Aurora database product has been the “fastest growing service in the history of AWS”. Is this really surprising that the world's largest cloud provider is selling more databases than anything else? Almost everyone needs a database, given howrelational database, there are more people around that have experience with it. >"To understand why SQL is making a comeback ..." No, SQL never went away. Full stop. >"But don’t take our word for it. Take Google’s" No, this practice of blindly adopting ideas just because they work for Google needs to stop. This post sounds as if the author(s) themselves bought into all of the NoSQL hype that buzzword-obsessed tech journalists were spinning and they are just now having an epiphany that much of that hype was just that.

geebee 9y ago
"I have never been aware that such a competition between datastore designs exists" Good! There certainly shouldn't be. Wish I could say the same. Magpie developers, people who want to seem edgy and forward looking, people who like making others look backwards at meetings, people who are worried they'll look like dinosaurs, people who've been around the block long enough to know an approach is totally unnecessary for their project but will be required on the resume for the next project. Trust me, they were all out there, in force. And of course, many data stores that aren't SQL are absolutely great for the task. Try to solve the 6 degrees of separation problem with SQL joins, then do it with a graph database. There are all kinds of applications that weren't a great fit for a relational model, and many that are. This article does mention the moment where NoSql was defined as "Not Always SQL" (and dismisses it with "nice try"), but honestly, if this had been the approach up front, it would have saved us a lot of grief. Probably. Maybe. I hope. Oh, who am I kidding. All that stuff I described above would have happened anyway.

AzzieElbab 9y ago

Why is no one stating the obvious? 1) hardware improvments along with db hosted solutions simplified sql management and scaling 2) most data is simply useless or of very temporary value. Having said that, I do not think sql can compete with specialized solutions when it comes to unstructured or semi-structured data like EMR for example. Also, with notable exception of datalog/datomic NoSQL solutions remain kV stores hacked together by some horrible programming models

TheVikingOwain 9y ago

I always get a little tick when I read the nosql is a new thing. Pick databases like Unidata/Universe and mvBase have been around for a long time and experienced a lot of the problems that newer nosql implementations have/had.

botskonet 9y ago

I've long struggled with NoSQL. My data is relational, and I've been long taught to normalize and de-dup my data, so how do I use this? I keep feeling like I'm missing the genius because I keep coming back to wishing I had a true relational table. However, there have been times where being forced to define a schema has been painful. If our app has to store dynamic keys/values it all winds up as records in a relational db table acting like key/value store. Whereas in Mongo we can just store the document, without affecting the entire table schema. I want the best of both worlds...

mrjn 9y ago

SQL is not beating NoSQL. NoSQL is evolving. https://blog.dgraph.io/post/sql-vs-nosql/ https://blog.dgraph.io/post/sql-vs-nosql/