5 ms·

MySQL and Memcached: End of an Era?

33 points by alrex021 17y ago

patio11 17y ago
I think there is a huge, huge gap between TwiDiBook's needs and the needs of the overwhelming majority of sites on the Internet. There are plenty of applications which do real things for real people which will never have scaling issues associated with them. The vast majority of the remainder of applications will have issues which are tractable by fairly simple solutions. And then we get to the pathbreakers who have to invent new methods of engineering to keep up with their growth curves. The party is just getting started for "Hey, check it: we can put a cache in front of our database" for that broad middle of the curve. (This includes applications where relational databases are a great fit but for certain performance requirements in small bits of the program. For example, you really don't want to have to implement a university backend system in a key/value store. Just trust me on this one. It is virtually made for SQL, but while 99% of the use of the system is less than performance intensive, having resources in place to support that last 1% costs the university hundreds of thousands of dollars. I think we could slash that with creative caching.)
- jbellis 17y ago
  Yes, some parts of the market don't need the kind of scale talked about here (you'd think the domain name would tip you off). And plenty of people get by with MS Access, still. I don't really think either point adds to the discussion.
  - henrikschroder 17y ago
    Of course it adds to the discussion, because the point was that the overwhelming majority of the market does not need this kind of scale. "End of an era" signals a major shift across the market, not that the tiny bleeding edge has moved on to something different.
- rbranson 17y ago
  Better even is that PostgreSQL 9.0 will have solid master/slave async replication. PostgreSQL already supports better write concurrency and multi-core scaling than InnoDB or MyISAM, and the replication will bring web-style read scaling into the picture. PostgreSQL has a "free-er" license, a more unified community, and isn't burdened by possibly hostile ownership.
  - pedalpete 17y ago
    I don't think we can say PostgreSQL is truly 'better' than the non-relational databases which are beginning to permeate the web. I believe the decision is more of a mySql/postgreSQL vs. non-relational databases. Sure, PostgreSQL may have better implementations of certain features than MySQL, but for most web developers, mysql is good enough. How would put PostgreSQL against non-relational dbs?
    - itgoon 17y ago
      I'm only just getting into my first nosql (Cassandra), so I've got no real-world data on it, but I think it would perform in the "well enough" category in performance. Just implement the tables with two columns: a key and a value, and index the crap out of them. It would also have the benefit of better tools and the option to use a JOIN or a transaction where it made sense. Of course, you could do the same with MySql, too.
    - rbranson 17y ago
      I'm not advocating PostgreSQL over a NoSQL. It certainly won't scale like Cassandra. I'm advocating it over MySQL. With the 9.0 release, it will be to a point where there won't be any reason to chose MySQL. PostgreSQL is easier to deploy and simpler to configure, has more features, is less likely to screw up your data, and has more scalable performance.
- petewarden 17y ago
  I'd agree with all that, but add a disclaimer. The rise of cheap and easy solutions for handling Big Data makes a lot of new applications feasible. All those cool correlations that Amazon does to suggest new books? You can do that at home now. Got a problem that would benefit from running an analysis of hundreds of millions of pages? You don't have to be Google to implement that. It is a bit maddening to see people applying the NoSQL hammer to problems that don't need scalability, but it's partly driven by the realization that they open up opportunities to build entirely new applications.
  - mcav 17y ago
    I know there's Hadoop, etc for map/reduce; what is available for correlations like Amazon?
    - bad_user 17y ago
      Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Personally I don't trust NoSql solutions because they aren't mature enough. I've had little problems using MySql/Postgres and never had loss of data ... and I can always come up with a non-relational scheme latter built on top ... when I really need it. The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. I've also used MySql instead of Memcached successfully for a website with tens of millions of hits per day, preferring it because Memcached crashes on me a lot and there are big differences between versions I don't want to worry about ... and yeah, I know async reads are problematic with Mysql/Postgres, but I just threw a web service in front of it, and when using the Memory storage engine it really kicks ass.
      - strlen 17y ago
        > Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Many of the algorithms for recommender systems are especially known for being parallelizable and would be extremely difficult to reliably and quickly implement on top of MySQL (or Postgres or Oracle). I can't even think of a way to do so without making a fairly complex ETL cycle with a separate DWH and a main serving DB (with points at which the site becomes degraded and possibly unavailable). As I've mentioned earlier, with a system like Hadoop, you could start off by implementing these algorithms in Perl or Python (making use extensive libraries already available in the language). Recommendations may not be real time, but there's a huge time difference between doing an even fairly simple "market basket"-type analysis for a data set orders of magnitude difference smaller than Amazon's in MySQL and doing this on top of Hadoop + HBase/Voldemort/Cassandra (i.e., compute in Hadoop, serve from a NoSQL db which offers Hadoop integration). You could even start by doing the computation in Hadoop and loading the data into MySQL (while, perhaps, serving the data from a slave db). > Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Your start-up doesn't need to have tons of users to have tons of data. This is especially true if you're building a start-up to make sense of the data that's already out there (some YC examples are Flightcaster, Directed Edge). For a consumer start-up, ability to have cheap and scalable storage means you can record data much more frequently. > The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. Isn't this the point of separating your data model from your presentation layer? Begin with MySQL (or even simpler, with BerkeleyDB) -- change it to whatever else suits your needs.
tszming 17y ago
... I’m just suggesting do not just grab advice from the Internet or friends tip and do not complicate beyond the need. ...(Peter) http://www.mysqlperformanceblog.com/2009/03/01/kiss-kiss-kiss/ http://www.mysqlperformanceblog.com/2009/03/01/kiss-kiss-kis...
JulianMorrison 17y ago
The Digg link said it best: "Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores" In other words, to get MySQL to superduper scale, you have to turn it into NoSQL anyway. At which point, the only reason to be using MySQL at all is inertia, because it's pretty clunky compared to the other alternatives. Back in the day before Cassandra and etc, there wasn't a concept for NoSQL, and sharding MySQL looked like an exotic way of using a relational DB. Actually, no. It's a way of crudely emulating something that isn't a relational DB.

33 points by alrex021 17y ago

I think there is a huge, huge gap between TwiDiBook's needs and the needs of the overwhelming majority of sites on the Internet. There are plenty of applications which do real things for real people which will never have scaling issues associated with them. The vast majority of the remainder of applications will have issues which are tractable by fairly simple solutions. And then we get to the pathbreakers who have to invent new methods of engineering to keep up with their growth curves. The party is just getting started for "Hey, check it: we can put a cache in front of our database" for that broad middle of the curve. (This includes applications where relational databases are a great fit but for certain performance requirements in small bits of the program. For example, you really don't want to have to implement a university backend system in a key/value store. Just trust me on this one. It is virtually made for SQL, but while 99% of the use of the system is less than performance intensive, having resources in place to support that last 1% costs the university hundreds of thousands of dollars. I think we could slash that with creative caching.)

jbellis 17y ago
Yes, some parts of the market don't need the kind of scale talked about here (you'd think the domain name would tip you off). And plenty of people get by with MS Access, still. I don't really think either point adds to the discussion.
- henrikschroder 17y ago
  Of course it adds to the discussion, because the point was that the overwhelming majority of the market does not need this kind of scale. "End of an era" signals a major shift across the market, not that the tiny bleeding edge has moved on to something different.
rbranson 17y ago
Better even is that PostgreSQL 9.0 will have solid master/slave async replication. PostgreSQL already supports better write concurrency and multi-core scaling than InnoDB or MyISAM, and the replication will bring web-style read scaling into the picture. PostgreSQL has a "free-er" license, a more unified community, and isn't burdened by possibly hostile ownership.
- pedalpete 17y ago
  I don't think we can say PostgreSQL is truly 'better' than the non-relational databases which are beginning to permeate the web. I believe the decision is more of a mySql/postgreSQL vs. non-relational databases. Sure, PostgreSQL may have better implementations of certain features than MySQL, but for most web developers, mysql is good enough. How would put PostgreSQL against non-relational dbs?
  - itgoon 17y ago
    I'm only just getting into my first nosql (Cassandra), so I've got no real-world data on it, but I think it would perform in the "well enough" category in performance. Just implement the tables with two columns: a key and a value, and index the crap out of them. It would also have the benefit of better tools and the option to use a JOIN or a transaction where it made sense. Of course, you could do the same with MySql, too.
  - rbranson 17y ago
    I'm not advocating PostgreSQL over a NoSQL. It certainly won't scale like Cassandra. I'm advocating it over MySQL. With the 9.0 release, it will be to a point where there won't be any reason to chose MySQL. PostgreSQL is easier to deploy and simpler to configure, has more features, is less likely to screw up your data, and has more scalable performance.
petewarden 17y ago
I'd agree with all that, but add a disclaimer. The rise of cheap and easy solutions for handling Big Data makes a lot of new applications feasible. All those cool correlations that Amazon does to suggest new books? You can do that at home now. Got a problem that would benefit from running an analysis of hundreds of millions of pages? You don't have to be Google to implement that. It is a bit maddening to see people applying the NoSQL hammer to problems that don't need scalability, but it's partly driven by the realization that they open up opportunities to build entirely new applications.
- mcav 17y ago
  I know there's Hadoop, etc for map/reduce; what is available for correlations like Amazon?
  - bad_user 17y ago
    Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Personally I don't trust NoSql solutions because they aren't mature enough. I've had little problems using MySql/Postgres and never had loss of data ... and I can always come up with a non-relational scheme latter built on top ... when I really need it. The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. I've also used MySql instead of Memcached successfully for a website with tens of millions of hits per day, preferring it because Memcached crashes on me a lot and there are big differences between versions I don't want to worry about ... and yeah, I know async reads are problematic with Mysql/Postgres, but I just threw a web service in front of it, and when using the Memory storage engine it really kicks ass.
    - strlen 17y ago
      > Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Many of the algorithms for recommender systems are especially known for being parallelizable and would be extremely difficult to reliably and quickly implement on top of MySQL (or Postgres or Oracle). I can't even think of a way to do so without making a fairly complex ETL cycle with a separate DWH and a main serving DB (with points at which the site becomes degraded and possibly unavailable). As I've mentioned earlier, with a system like Hadoop, you could start off by implementing these algorithms in Perl or Python (making use extensive libraries already available in the language). Recommendations may not be real time, but there's a huge time difference between doing an even fairly simple "market basket"-type analysis for a data set orders of magnitude difference smaller than Amazon's in MySQL and doing this on top of Hadoop + HBase/Voldemort/Cassandra (i.e., compute in Hadoop, serve from a NoSQL db which offers Hadoop integration). You could even start by doing the computation in Hadoop and loading the data into MySQL (while, perhaps, serving the data from a slave db). > Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Your start-up doesn't need to have tons of users to have tons of data. This is especially true if you're building a start-up to make sense of the data that's already out there (some YC examples are Flightcaster, Directed Edge). For a consumer start-up, ability to have cheap and scalable storage means you can record data much more frequently. > The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. Isn't this the point of separating your data model from your presentation layer? Begin with MySQL (or even simpler, with BerkeleyDB) -- change it to whatever else suits your needs.

tszming 17y ago

... I’m just suggesting do not just grab advice from the Internet or friends tip and do not complicate beyond the need. ...(Peter) http://www.mysqlperformanceblog.com/2009/03/01/kiss-kiss-kiss/ http://www.mysqlperformanceblog.com/2009/03/01/kiss-kiss-kis...

JulianMorrison 17y ago

The Digg link said it best: "Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores" In other words, to get MySQL to superduper scale, you have to turn it into NoSQL anyway. At which point, the only reason to be using MySQL at all is inertia, because it's pretty clunky compared to the other alternatives. Back in the day before Cassandra and etc, there wasn't a concept for NoSQL, and sharding MySQL looked like an exotic way of using a relational DB. Actually, no. It's a way of crudely emulating something that isn't a relational DB.