47 ms·

SQL Databases Don't Scale

100 points by sant0sk1 17y ago

brlewis 17y ago
This article is about relational databases that you read from and write to. It isn't SQL-specific. It's a good article, not another "SQL is obsolete" as you might think from the title.
wildwood 17y ago
"Sharding kills most of the value of a relational database." I can appreciate the point the author makes about partitioning schemes requiring heavy integration with the business logic, but I disagree with the claim that sharding doesn't work. It's probably more accurate to say that sharding only works well if you design your database very carefully, or just get lucky about how your data model maps to sharding schemes. A bad sharded database can probably hobble your app, but a good one does get you remarkably close to true horizontal scaling. In my experience, at least.
- psadauskas 17y ago
  If you data model maps well to sharding, it maps well to distributed databases, and you're not gaining much by being on an RDBMS. I think the point they're trying to make is if you're going through the contortions to shard a relational database, you're better off just doing it in a distributed database in the first place.
  - TheXenocide 17y ago
    There's more than one way to skin a mongoose and in this case the more common shard solutions are fundamentally different from distributed DB in that a distributed DB often times contains large amounts of a a subset of data that has to be merged with data on other DBs whereas a shard can have entire schema duplication on different shards where you can use relational technology just fine and even if the query is distributed you can just add rows to the end of a dataset rather than joining the data columns manually (implementing your own relational joins). In many cases you partition your sharded data in such a way that the majority of your queries are not distributed so that they can continue to leverage the relational model you had before, only performing a distributed query for more complex actions like reporting and such. Also going back to the first place isn't really an option, shards usually come of a system that has grown, not ground up design. Ultimately I still support the distributed model myself, but a shard model does support relational data much more so than a distributed DB (at least out of the box).
- prodigal_erik 17y ago
  Sharding means you can't do "select * from a join b using (k)" unless you can prove the a record and the b record will always be hosted on the same shard. So of course you often have to write application code that knows where they are and goes and gets them separately. But that's precisely the problem that relational databases were developed to solve for you. Then there's the whole distributed transaction thing, which is so painful that some people just live with wrong answers instead.
stuff4ben 17y ago
I thought it was a good summary of the problems facing SQL databases as organizations grow. However I would have appreciated some solutions to the problem.
- dflock 17y ago
  I think one of the points made at the end was that there kinda isn't one - people have been trying to come up with the perfect solution to scaling RDBMS's for ever and the current state of the art (pretty much) is as described in the article - i.e, not great. People with really massive MySQL setups (eBay, for example) basically just do a huge amount of the stuff described here, but with lots of message-queue type stuff to glue it together and loads of hard work; i.e. you can make it scale but it's really really hard work because RDBMS's aren't inherent scalable.
  - gaius 17y ago
    the current state of the art (pretty much) is as described in the article - i.e, not great That simply isn't true. For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how RDBMS can't handle their tiny datasets.
    - dflock 17y ago
      What hardware is this running on, roughly? What the author was getting at, I think, was not that you can't do it, just that it's hard/expensive; you need fairly beefy hardware and experienced DBA's to manage horizontal scaleout manually.
      - DanielBMarkham 17y ago
        If that were the case, then I think the concluding sentences should be changed. When hundreds of companies and thousands of the brightest programmers and sysadmins have been trying to solve a problem for twenty years and still haven’t managed to come up with an obvious solution that everyone adopts, that says to me the problem is unsolvable.
Shakescode 17y ago
In the final paragraph he compares himself with Capt Kirk, facing the Kobayashi Maru: "we can only solve this problem by redefining the question." Well, perhaps that's as difficult as solving the "SQL scalability that 20 years of brilliant programmers haven't solved"? ...because he declines to attempt to reframe the question. [ or maybe he's leaving us wanting more: like his next post? ] Anybody up for reformulating the question?
TimothyFitz 17y ago
RAID doesn't meet his definition of "scalable" because it has a central controller. This whole post could be summed up as "ACID doesn't scale", which has been proven. Consistency, Availability or Partition Tolerance; pick two (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.1495 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.1...). Good introduction to non-ACID databases: http://highscalability.com/drop-acid-and-think-about-data http://highscalability.com/drop-acid-and-think-about-data
- Confusion 17y ago
  RAID setups can be outfitted with redundant controllers, in which case it does meet his definition of 'scalable'.
- nettdata 17y ago
  Oh please. The mere fact that he's talking "RAID" instead of SAN speaks volumes. (No pun intended). Any storage engineer worth their salt would be rolling their eyes right now. The doc you linked to was authored in 2002, about the same time that Oracle's RAC was introduced (late 2001). Some of his citations are from the 80's. THE 80'S! DB technology has come a LONG way in 8 years, and that paper is no longer valid, unless you're talking about some basic, "minor"/open source db technologies. If you want to say "open source DB technologies have problems scaling", then go right ahead, and I'll agree. Just don't mind those of us who continue to build large, scalable systems, using the proper DB technologies, that disprove that "sql doesn't scale" generalization.
  - rjurney 17y ago
    SANS are too expensive. They don't scale cost-effectively compared to commodity PC hardware.
    - nettdata 17y ago
      I'd beg to differ. Go take a look at Adam Leventhal's work with the Fishworks stuff. Specifically, go check out the Sun Storage 7310. It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors. I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets. Again, if you're railing against something as ubiquitous as a SAN, I'm not sure there's anything I can say to change your mind. Not that I'm really here to change your mind. Again, the original focus was about SQL not scaling, and you seem to be fixated on the cost of that scaling.
bayareaguy 17y ago
SQL databases are fundamentally non-scalable, and there is no magical pixie dust that we, or anyone, can sprinkle on them to suddenly make them scale. I find this claim laughable. If the databases Mr. Wiggins chooses to work with fail to meet his scalability requirements, perhaps he should consider different databases. Scalable SQL databases have been around for over 3 decades.
- qhoxie 17y ago
  Your comment would hold more weight if you elaborated some more. Are you referring to scaling by his standards? Please explain.
  - bayareaguy 17y ago
    Mr. Wiggins would have us think in terms of "true scalability" which conflates the following concerns: more servers creates more capacity (the classic definition of scalability), the business logic of the app is separated from concerns of scaling server resources (this is not possible past whatever limit you set and so has no classic definition) and no single point of failure (more classically formalized as availability). Companies such as Teradata have long offered SQL systems which meet the classic definitions of scalability and availability. I think TimothyFitz's reply above was accurate. Mr. Wiggins article would be better titled something like "ACID databases have scalability problems, especially cheap ones startups use" but then, like http://news.ycombinator.com/item?id=690653 http://news.ycombinator.com/item?id=690653 , it wouldn't get much response.
    - deleted 17y ago
      [deleted]
pj 17y ago
Okay, see... here's more of the no-SQL agenda at hacker news. This isn't a question of "Does SQL Scale or not?" but "How much do SQL Databases Scale?" I feel like the propaganda here is that because RDBMS doesn't scale to youtube or google scale they suck and that's not true. Like SQL is a waste of time because at some point, you're going to need to shard your database. Look, at that kind of scale, you're going to have problems with any solution to any problem. Handling that kind of scale is going to be expensive no matter what solution you implement, whether it's map/reduce or flat files or some other solution. But deciding to build a system from the beginning on something non-relational because someday you may have to accommodate that kind of scale is an example of premature optimization. The vast majority of features you get with SQL are going to outweigh the limitations of noSQL. I've worked on some pretty high scale systems built on SQL and yes, there are problems, but there's just something irrational going on here and it's off-putting. It's like we are throwing out the baby with the bath water or something.
- qhoxie 17y ago
  Look, at that kind of scale, you're going to have problems with any solution to any problem. Handling that kind of scale is going to be expensive no matter what solution you implement, whether it's map/reduce or flat files or some other solution. It is not a point of whether or not it is expensive. Scaling (nearly) always has expense associated with it. The issue is how much expense, and with some applications it is significantly less with one of the "no-SQL" solutions. But deciding to build a system from the beginning on something non-relational because someday you may have to accommodate that kind of scale is an example of premature optimization. The vast majority of features you get with SQL are going to outweigh the limitations of noSQL. You are making it sound as though a relational database is the correct choice barring any scaling. Perhaps you have not yet thoroughly evaluated some of the alternatives out there, because in many applications, there would be no step back from *SQL.
- gaius 17y ago
  As others have pointed out, the "no SQL" crowd are invariably MySQL users who have run into the limitations of MySQL but for ideological reasons can't state that the problems they encounter are specific to MySQL. DB2, Teradata and Oracle users regularly tackle problems 100x larger than MySQL can handle.
alexgartrell 17y ago
Crazy idea: Write your data-handling code in it's own tight little module. Use good abstractions so that the rest of your app doesn't give a crap HOW it's happening. When it's time to update to millions of clustered servers running bigTable, you rewrite the module, and you're done. Software Engineering saves the day!
- alexgartrell 17y ago
  3 upvotes and 2 downvotes without a single response. As a junior member of Hacker News, I demand an education when I'm downvoted! :)
  - akeefer 17y ago
    I wasn't one of the ones who upvoted or downvoted you, but just because you asked . . . while abstractions around your data-handling code are always a good idea, on some level they always leak. (For example, see the classic post http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Compute... describing how ORM, one of the most common approaches to encapsulating data-handling code, tends to fail). Most of your code is data-handling code on some fundamental level, so it's pretty inevitable that it's going to matter to your application how that data is stored and what sorts of operations you can do on it, and that's especially true if you care about performance (which you generally always do after some level). So even if you have a nice abstraction of your query layer, the kinds of questions you can ask the database efficiently depend heavily on the underlying storage mechanism, and so your application logic has to be built so it only asks the right kinds of questions. Even abstracting away the differences between, say, Oracle and MS SQLServer and MySQL is difficult enough, because of the different capabilities and performance characteristics. When you do that, you basically end up coding to the lowest common denominator, which can often limit what your applications does. Trying to come up with an abstraction that can encapsulate the difference between a row- versus column-oriented database, or between a relational database and some other kind of storage, is pretty much a losing proposition: they're too fundamentally different in terms of what kind of data you can store, how you can store it, how you can query it, what kinds of transactional guarantees you get, and what operations are fast and which ones are slow. So you really do kind of have to take your best shot at it, choose an approach, and if you choose wrong and have to change, it's just going to hurt. A lot. Good encapsulation and abstraction will ease some of the pain, but it's more like drinking whiskey before your leg gets sawed off than it is like general anesthesia.
ojbyrne 17y ago
At a management level there are larger problems. SQL was a solution to the "letting programmers run the show" problem. I.e. a commoditized language that executives could mostly understand and budget for. See pretty well everything Philip Greenspun has written.
antidaily 17y ago
Maybe I am getting old, but I no longer find arguments over semantics as interesting as I once did.
- jmtulloss 17y ago
  I agree, but I'm not sure this is purely a semantic argument. Care to elaborate?
Confusion 17y ago
So while sharding is a form of horizontal scaling, it fails point #2: it is not transparent to the business logic of the application. I do not believe this is usually true, for two reasons. One may be nitpicking, but 'where to get the data' is not part of the business logic: it's pure application logic, dependent on your solution of the problem. In that sense, his argument is wrong. The other reason definitely isn't nitpicking: you can solve the problem by adding a layer between your DAO's and the databases, that handles the 'where to get the data' question. So yes, it requires some programming, but it is still transparant to the business logic. It does not require an invasive change in your application and I think he is grossly exagerating this point.
- prodigal_erik 17y ago
  I don't see a good reason for "which server has my data" to be in-your-face application logic to code, when "which sector of which disk has my data" has long since been delegated to the platform and forgotten.
  - Confusion 17y ago
    If you are bound to SQL databases, either for historical reasons or because the no-SQL database make other problems harder, then 'scaling the application' is a pretty good reason for writing that logic. There is no silver bullet.
nettdata 17y ago
I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of large, scalable database systems. I have, over the past 4 years alone, built Oracle-based, fully scalable databases that handle over 25 million users daily. Go read up on their RAC architecture, and the "shared everything" implementation. Sure, it's expensive, but it works great. For instance, if you play sports games from the world's largest video game corporation, all of your online transactions (achievements, etc) go through exactly such a system that I spent a year architecting and implementing. If you do any online banking in Canada, or with some of the larger banks in the US, that, too, is on my resume. I find that this type of FUD comes about from people that aren't good at designing and implementing large databases, or can't afford the technology that can pull it off, so they slam the technology rather than accept that they, themselves, are the ones lacking. Most of them tend to come from the typical LAMP/SlashDot crowd that only have experience with the minor technologies. Those of us that do it for a living, using the right technologies, seem to have no problems whatsoever scaling SQL. Just saying.
- staunch 17y ago
  Yeah, and if Facebook used Oracle instead of MySQL and memcached, they'd be looking at a $50 million/year+ bill no doubt. Those silly kids with their minor technologies should learn to use a real DB. If those 100 million users that login every day only knew...
  - 4buot 17y ago
    For what its worth, Facebook does have a big Oracle install. I find it strange that everyone trotted out their preformed opinion for this piece, but no one comments on the cassandra article which actually proposes a solution...although it requires you to learn something, how horrible.
    - 17y ago
tommy_chheng 17y ago
Isn't Facebook using MySQL?
trezor 17y ago
Summary of an article about how "SQL Databases Don't Scale": 1. Mentions RAID, not SAN 2. Mentions MySQL and only MySQL (with the exception of PostgreSQL once). 3. Mentions Master-slave replication as a killer scalability feature. I suggest he rename the piece to "A $500 server and MySQL Don't Scale" and then we can all agree and get along.
- khaless 17y ago
  Yeah, there certainly are a few more tricks with regards to scaling RDBMS than the author covered. Depending on what your demands are there are different techniques and protocols which best suite you and can go very far to solving your problem. But with that in mind there may be better ways to solve your problem, and we should not forget about those. If You start sharding you may get write gains, but you have to work very hard to keep things consistent (depending) and you may have to duplicate shards to make them highly available ($$$). Oh - and later down the track your schema might change in ways which your sharding scheme is just not flexible enough to deal with and depending on who you are that may be too much of a risk. Besides the issue is really with availability, consistency and performance. It is very hard to scale all three of these together and even your cashcow solutions will hit their limits (although some of their limits are quite high :))
- TheXenocide 17y ago
  Most of the things he mentions apply to most DB engines, MySQL, MS-SQL, Postgres, Firebird and many editions of Oracle (all of which I've used in data-intensive applications) suffer the same woes. Ultimately it would appear that you went into it reading the article with a narrow minded view of "Yes they do!" instead of "Why don't they?" Also, the disk analogy was for relating information not discussing storage technology or to imply that was what he was using when trying to scale. The portion of vertical scalability clearly presents the option of (and pitfalls therein) using hardware to account for data scalability...
JulianMorrison 17y ago
Scale like what? If you aren't IBM, Facebook or Google, why do you even care?
- jacquesm 17y ago
  ouch, I'm not IBM, Facebook or Google but SQL scalability issues are a good part of my daily workload. (the other part is made up from file system scalability issues). Once you you get past a certain level these are non-trivial problems and anybody out there that is busy solving them has my interest.
tybris 17y ago
Can anyone give me the context? I don't believe in absolutes. I only believe in "it depends on the application/budget/knowledge/etc"
antirez 17y ago
I'm the author of Redis so I should be biased the other way around, but the Redis and KV experience taught me that to be exposed to the KV parading was for me a similar experience as learning the Scheme language, I started to write code in imperative languages in a new way. In the same way once you start thinking at scalability of your data in a new way, from the point of view of partitioning the data, organizing this data in away that is easy to access for your usage pattern, make judicious use of serialization, SQL databases can be a good pick. Simply you need to abandon the paradigm of let's design our nice tables and run our multiple joins and group by against. But of course designing DBs in this new way makes most of the SQL features not needed in most scenarios... and you still have the overhead. And there are a number of other limits now that complex designs are not a good idea, for instance there is no way to get back the data in the natural ordering (the order you pushed this data, or the reverse)... This is where KV stores start to be interesting as real world alternative, not just in order to learn the paradigm of scalability. But again, if you don't trust KV stores it is truly possible to use an SQL database in a more conservative and scalable way.

100 points by sant0sk1 17y ago

This article is about relational databases that you read from and write to. It isn't SQL-specific. It's a good article, not another "SQL is obsolete" as you might think from the title.

wildwood 17y ago

"Sharding kills most of the value of a relational database." I can appreciate the point the author makes about partitioning schemes requiring heavy integration with the business logic, but I disagree with the claim that sharding doesn't work. It's probably more accurate to say that sharding only works well if you design your database very carefully, or just get lucky about how your data model maps to sharding schemes. A bad sharded database can probably hobble your app, but a good one does get you remarkably close to true horizontal scaling. In my experience, at least.

psadauskas 17y ago
If you data model maps well to sharding, it maps well to distributed databases, and you're not gaining much by being on an RDBMS. I think the point they're trying to make is if you're going through the contortions to shard a relational database, you're better off just doing it in a distributed database in the first place.
- TheXenocide 17y ago
  There's more than one way to skin a mongoose and in this case the more common shard solutions are fundamentally different from distributed DB in that a distributed DB often times contains large amounts of a a subset of data that has to be merged with data on other DBs whereas a shard can have entire schema duplication on different shards where you can use relational technology just fine and even if the query is distributed you can just add rows to the end of a dataset rather than joining the data columns manually (implementing your own relational joins). In many cases you partition your sharded data in such a way that the majority of your queries are not distributed so that they can continue to leverage the relational model you had before, only performing a distributed query for more complex actions like reporting and such. Also going back to the first place isn't really an option, shards usually come of a system that has grown, not ground up design. Ultimately I still support the distributed model myself, but a shard model does support relational data much more so than a distributed DB (at least out of the box).
prodigal_erik 17y ago
Sharding means you can't do "select * from a join b using (k)" unless you can prove the a record and the b record will always be hosted on the same shard. So of course you often have to write application code that knows where they are and goes and gets them separately. But that's precisely the problem that relational databases were developed to solve for you. Then there's the whole distributed transaction thing, which is so painful that some people just live with wrong answers instead.

stuff4ben 17y ago

I thought it was a good summary of the problems facing SQL databases as organizations grow. However I would have appreciated some solutions to the problem.

dflock 17y ago
I think one of the points made at the end was that there kinda isn't one - people have been trying to come up with the perfect solution to scaling RDBMS's for ever and the current state of the art (pretty much) is as described in the article - i.e, not great. People with really massive MySQL setups (eBay, for example) basically just do a huge amount of the stuff described here, but with lots of message-queue type stuff to glue it together and loads of hard work; i.e. you can make it scale but it's really really hard work because RDBMS's aren't inherent scalable.
- gaius 17y ago
  the current state of the art (pretty much) is as described in the article - i.e, not great That simply isn't true. For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how RDBMS can't handle their tiny datasets.
  - dflock 17y ago
    What hardware is this running on, roughly? What the author was getting at, I think, was not that you can't do it, just that it's hard/expensive; you need fairly beefy hardware and experienced DBA's to manage horizontal scaleout manually.
    - DanielBMarkham 17y ago
      If that were the case, then I think the concluding sentences should be changed. When hundreds of companies and thousands of the brightest programmers and sysadmins have been trying to solve a problem for twenty years and still haven’t managed to come up with an obvious solution that everyone adopts, that says to me the problem is unsolvable.

Shakescode 17y ago

In the final paragraph he compares himself with Capt Kirk, facing the Kobayashi Maru: "we can only solve this problem by redefining the question." Well, perhaps that's as difficult as solving the "SQL scalability that 20 years of brilliant programmers haven't solved"? ...because he declines to attempt to reframe the question. [ or maybe he's leaving us wanting more: like his next post? ] Anybody up for reformulating the question?

TimothyFitz 17y ago

RAID doesn't meet his definition of "scalable" because it has a central controller. This whole post could be summed up as "ACID doesn't scale", which has been proven. Consistency, Availability or Partition Tolerance; pick two (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.1495 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.1...). Good introduction to non-ACID databases: http://highscalability.com/drop-acid-and-think-about-data http://highscalability.com/drop-acid-and-think-about-data

Confusion 17y ago
RAID setups can be outfitted with redundant controllers, in which case it does meet his definition of 'scalable'.
nettdata 17y ago
Oh please. The mere fact that he's talking "RAID" instead of SAN speaks volumes. (No pun intended). Any storage engineer worth their salt would be rolling their eyes right now. The doc you linked to was authored in 2002, about the same time that Oracle's RAC was introduced (late 2001). Some of his citations are from the 80's. THE 80'S! DB technology has come a LONG way in 8 years, and that paper is no longer valid, unless you're talking about some basic, "minor"/open source db technologies. If you want to say "open source DB technologies have problems scaling", then go right ahead, and I'll agree. Just don't mind those of us who continue to build large, scalable systems, using the proper DB technologies, that disprove that "sql doesn't scale" generalization.
- rjurney 17y ago
  SANS are too expensive. They don't scale cost-effectively compared to commodity PC hardware.
  - nettdata 17y ago
    I'd beg to differ. Go take a look at Adam Leventhal's work with the Fishworks stuff. Specifically, go check out the Sun Storage 7310. It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors. I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets. Again, if you're railing against something as ubiquitous as a SAN, I'm not sure there's anything I can say to change your mind. Not that I'm really here to change your mind. Again, the original focus was about SQL not scaling, and you seem to be fixated on the cost of that scaling.

bayareaguy 17y ago

SQL databases are fundamentally non-scalable, and there is no magical pixie dust that we, or anyone, can sprinkle on them to suddenly make them scale. I find this claim laughable. If the databases Mr. Wiggins chooses to work with fail to meet his scalability requirements, perhaps he should consider different databases. Scalable SQL databases have been around for over 3 decades.

qhoxie 17y ago
Your comment would hold more weight if you elaborated some more. Are you referring to scaling by his standards? Please explain.
- bayareaguy 17y ago
  Mr. Wiggins would have us think in terms of "true scalability" which conflates the following concerns: more servers creates more capacity (the classic definition of scalability), the business logic of the app is separated from concerns of scaling server resources (this is not possible past whatever limit you set and so has no classic definition) and no single point of failure (more classically formalized as availability). Companies such as Teradata have long offered SQL systems which meet the classic definitions of scalability and availability. I think TimothyFitz's reply above was accurate. Mr. Wiggins article would be better titled something like "ACID databases have scalability problems, especially cheap ones startups use" but then, like http://news.ycombinator.com/item?id=690653 http://news.ycombinator.com/item?id=690653 , it wouldn't get much response.
  - deleted 17y ago
    [deleted]

pj 17y ago

Okay, see... here's more of the no-SQL agenda at hacker news. This isn't a question of "Does SQL Scale or not?" but "How much do SQL Databases Scale?" I feel like the propaganda here is that because RDBMS doesn't scale to youtube or google scale they suck and that's not true. Like SQL is a waste of time because at some point, you're going to need to shard your database. Look, at that kind of scale, you're going to have problems with any solution to any problem. Handling that kind of scale is going to be expensive no matter what solution you implement, whether it's map/reduce or flat files or some other solution. But deciding to build a system from the beginning on something non-relational because someday you may have to accommodate that kind of scale is an example of premature optimization. The vast majority of features you get with SQL are going to outweigh the limitations of noSQL. I've worked on some pretty high scale systems built on SQL and yes, there are problems, but there's just something irrational going on here and it's off-putting. It's like we are throwing out the baby with the bath water or something.

qhoxie 17y ago
Look, at that kind of scale, you're going to have problems with any solution to any problem. Handling that kind of scale is going to be expensive no matter what solution you implement, whether it's map/reduce or flat files or some other solution. It is not a point of whether or not it is expensive. Scaling (nearly) always has expense associated with it. The issue is how much expense, and with some applications it is significantly less with one of the "no-SQL" solutions. But deciding to build a system from the beginning on something non-relational because someday you may have to accommodate that kind of scale is an example of premature optimization. The vast majority of features you get with SQL are going to outweigh the limitations of noSQL. You are making it sound as though a relational database is the correct choice barring any scaling. Perhaps you have not yet thoroughly evaluated some of the alternatives out there, because in many applications, there would be no step back from *SQL.
gaius 17y ago
As others have pointed out, the "no SQL" crowd are invariably MySQL users who have run into the limitations of MySQL but for ideological reasons can't state that the problems they encounter are specific to MySQL. DB2, Teradata and Oracle users regularly tackle problems 100x larger than MySQL can handle.

alexgartrell 17y ago

Crazy idea: Write your data-handling code in it's own tight little module. Use good abstractions so that the rest of your app doesn't give a crap HOW it's happening. When it's time to update to millions of clustered servers running bigTable, you rewrite the module, and you're done. Software Engineering saves the day!

alexgartrell 17y ago
3 upvotes and 2 downvotes without a single response. As a junior member of Hacker News, I demand an education when I'm downvoted! :)
- akeefer 17y ago
  I wasn't one of the ones who upvoted or downvoted you, but just because you asked . . . while abstractions around your data-handling code are always a good idea, on some level they always leak. (For example, see the classic post http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Compute... describing how ORM, one of the most common approaches to encapsulating data-handling code, tends to fail). Most of your code is data-handling code on some fundamental level, so it's pretty inevitable that it's going to matter to your application how that data is stored and what sorts of operations you can do on it, and that's especially true if you care about performance (which you generally always do after some level). So even if you have a nice abstraction of your query layer, the kinds of questions you can ask the database efficiently depend heavily on the underlying storage mechanism, and so your application logic has to be built so it only asks the right kinds of questions. Even abstracting away the differences between, say, Oracle and MS SQLServer and MySQL is difficult enough, because of the different capabilities and performance characteristics. When you do that, you basically end up coding to the lowest common denominator, which can often limit what your applications does. Trying to come up with an abstraction that can encapsulate the difference between a row- versus column-oriented database, or between a relational database and some other kind of storage, is pretty much a losing proposition: they're too fundamentally different in terms of what kind of data you can store, how you can store it, how you can query it, what kinds of transactional guarantees you get, and what operations are fast and which ones are slow. So you really do kind of have to take your best shot at it, choose an approach, and if you choose wrong and have to change, it's just going to hurt. A lot. Good encapsulation and abstraction will ease some of the pain, but it's more like drinking whiskey before your leg gets sawed off than it is like general anesthesia.

ojbyrne 17y ago

At a management level there are larger problems. SQL was a solution to the "letting programmers run the show" problem. I.e. a commoditized language that executives could mostly understand and budget for. See pretty well everything Philip Greenspun has written.

antidaily 17y ago

Maybe I am getting old, but I no longer find arguments over semantics as interesting as I once did.

jmtulloss 17y ago
I agree, but I'm not sure this is purely a semantic argument. Care to elaborate?

Confusion 17y ago

So while sharding is a form of horizontal scaling, it fails point #2: it is not transparent to the business logic of the application. I do not believe this is usually true, for two reasons. One may be nitpicking, but 'where to get the data' is not part of the business logic: it's pure application logic, dependent on your solution of the problem. In that sense, his argument is wrong. The other reason definitely isn't nitpicking: you can solve the problem by adding a layer between your DAO's and the databases, that handles the 'where to get the data' question. So yes, it requires some programming, but it is still transparant to the business logic. It does not require an invasive change in your application and I think he is grossly exagerating this point.

prodigal_erik 17y ago
I don't see a good reason for "which server has my data" to be in-your-face application logic to code, when "which sector of which disk has my data" has long since been delegated to the platform and forgotten.
- Confusion 17y ago
  If you are bound to SQL databases, either for historical reasons or because the no-SQL database make other problems harder, then 'scaling the application' is a pretty good reason for writing that logic. There is no silver bullet.

nettdata 17y ago

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of large, scalable database systems. I have, over the past 4 years alone, built Oracle-based, fully scalable databases that handle over 25 million users daily. Go read up on their RAC architecture, and the "shared everything" implementation. Sure, it's expensive, but it works great. For instance, if you play sports games from the world's largest video game corporation, all of your online transactions (achievements, etc) go through exactly such a system that I spent a year architecting and implementing. If you do any online banking in Canada, or with some of the larger banks in the US, that, too, is on my resume. I find that this type of FUD comes about from people that aren't good at designing and implementing large databases, or can't afford the technology that can pull it off, so they slam the technology rather than accept that they, themselves, are the ones lacking. Most of them tend to come from the typical LAMP/SlashDot crowd that only have experience with the minor technologies. Those of us that do it for a living, using the right technologies, seem to have no problems whatsoever scaling SQL. Just saying.

staunch 17y ago
Yeah, and if Facebook used Oracle instead of MySQL and memcached, they'd be looking at a $50 million/year+ bill no doubt. Those silly kids with their minor technologies should learn to use a real DB. If those 100 million users that login every day only knew...
- 4buot 17y ago
  For what its worth, Facebook does have a big Oracle install. I find it strange that everyone trotted out their preformed opinion for this piece, but no one comments on the cassandra article which actually proposes a solution...although it requires you to learn something, how horrible.
  - 17y ago

tommy_chheng 17y ago

Isn't Facebook using MySQL?

trezor 17y ago

Summary of an article about how "SQL Databases Don't Scale": 1. Mentions RAID, not SAN 2. Mentions MySQL and only MySQL (with the exception of PostgreSQL once). 3. Mentions Master-slave replication as a killer scalability feature. I suggest he rename the piece to "A $500 server and MySQL Don't Scale" and then we can all agree and get along.

khaless 17y ago
Yeah, there certainly are a few more tricks with regards to scaling RDBMS than the author covered. Depending on what your demands are there are different techniques and protocols which best suite you and can go very far to solving your problem. But with that in mind there may be better ways to solve your problem, and we should not forget about those. If You start sharding you may get write gains, but you have to work very hard to keep things consistent (depending) and you may have to duplicate shards to make them highly available ($$$). Oh - and later down the track your schema might change in ways which your sharding scheme is just not flexible enough to deal with and depending on who you are that may be too much of a risk. Besides the issue is really with availability, consistency and performance. It is very hard to scale all three of these together and even your cashcow solutions will hit their limits (although some of their limits are quite high :))
TheXenocide 17y ago
Most of the things he mentions apply to most DB engines, MySQL, MS-SQL, Postgres, Firebird and many editions of Oracle (all of which I've used in data-intensive applications) suffer the same woes. Ultimately it would appear that you went into it reading the article with a narrow minded view of "Yes they do!" instead of "Why don't they?" Also, the disk analogy was for relating information not discussing storage technology or to imply that was what he was using when trying to scale. The portion of vertical scalability clearly presents the option of (and pitfalls therein) using hardware to account for data scalability...

JulianMorrison 17y ago

Scale like what? If you aren't IBM, Facebook or Google, why do you even care?

jacquesm 17y ago
ouch, I'm not IBM, Facebook or Google but SQL scalability issues are a good part of my daily workload. (the other part is made up from file system scalability issues). Once you you get past a certain level these are non-trivial problems and anybody out there that is busy solving them has my interest.

tybris 17y ago

Can anyone give me the context? I don't believe in absolutes. I only believe in "it depends on the application/budget/knowledge/etc"

antirez 17y ago

I'm the author of Redis so I should be biased the other way around, but the Redis and KV experience taught me that to be exposed to the KV parading was for me a similar experience as learning the Scheme language, I started to write code in imperative languages in a new way. In the same way once you start thinking at scalability of your data in a new way, from the point of view of partitioning the data, organizing this data in away that is easy to access for your usage pattern, make judicious use of serialization, SQL databases can be a good pick. Simply you need to abandon the paradigm of let's design our nice tables and run our multiple joins and group by against. But of course designing DBs in this new way makes most of the SQL features not needed in most scenarios... and you still have the overhead. And there are a number of other limits now that complex designs are not a good idea, for instance there is no way to get back the data in the natural ordering (the order you pushed this data, or the reverse)... This is where KV stores start to be interesting as real world alternative, not just in order to learn the paradigm of scalability. But again, if you don't trust KV stores it is truly possible to use an SQL database in a more conservative and scalable way.