6 ms·

Wikipedia (English) starting to migrate to MariaDB

139 points by neya 14y ago

JackWebbHeller 14y ago
Which database system Wikipedia is currently using?
- AdamGibbins 14y ago
  MySQL
  - tommorris 14y ago
    Well, MySQL plus a text store. MediaWiki, the open source software that Wikipedia uses, now officially supports MariaDB.
    - neilk 14y ago
      We're also using OpenStack Swift for media storage these days.
      - mh- 14y ago
        I couldn't see a way to aggregate it in the ganglia linked.. could you share how many mysql (maria) queries per second you guys are averaging?
        neilk 14y ago
        I should clarify: I used to work there but I sometimes still say "we". I'm not a ganglia wizard - maybe this comes close? Although it's just all selects. I think. http://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mysql_com_select+across+all+db*&vl=stmts&x=&n=&hreg%5B%5D=db&mreg%5B%5D=mysql_com_select&gtype=stack&glegend=show&aggregate=1 http://ganglia.wikimedia.org/latest/graph_all_periods.php?ti... You might try asking on wikitech-l, the same mailing list that the original post comes from.
    - whalesalad 14y ago
      I use MariaDB on my Mac for most things that require MySQL. It's a drop in replacement for most things. Is there a special edge case with media wiki they had to solve or is it now just formally announced as being supported?
      - tommorris 14y ago
        So, it's pretty much a drop-in replacement. But as with everything, time m ust be spent testing and so on before rolling it out to production. I believe a transition to MariaDB has been planned for a while now.
      - malinens 14y ago
        I remember I had problems with insert statements when previously with mysql inserts without default values worked but MariaDB was strict in this case and these inserts failed...
- plasma 14y ago
  MariaDB is a fork of MySQL by Monty (original MySQL author)
jamespitts 14y ago
Hopefully this will get more people to adopt mariadb and be free of the legal and upgrade uncertainties of mysql. I installed it a few weeks ago and the result is -exactly- like mysql. http://blog.mariadb.org/mariadb-foundation-to-safeguard-leading-open-source-database/ http://blog.mariadb.org/mariadb-foundation-to-safeguard-lead...
sonier 14y ago
How does MariaDB compare to Percona?
- mjallday 14y ago
  It's a drop in replacement for MySQL, you can use the _storage engines_ that Percona provide such as ExtraDB so maybe it shouldn't differ at all?
revscat 14y ago
Wikipedia seems like the perfect system for a document store like CouchDB. I would be curious to know if they examined it and, if so, found it lacking.
- stingraycharles 14y ago
  My guess is that they have been using MySQL for a long, long time (possibly since the beginning)m, and see no need to migrate to something different. MariaDB is API-compatible with MySQL, so this migration will not involve much changes at the client side; it's mostly thorough testing that needs to be done.
- notatoad 14y ago
  MariaDB is a (almost) drop-in replacement for MySQL, CouchDB is a totally different concept that would require nearly a complete re-write of their backend. I doubt they considered it.
- neilk 14y ago
  MediaWiki uses a fairly typical mostly-normalized relational design. You couldn't easily port it to a document store. http://upload.wikimedia.org/wikipedia/commons/3/33/MediaWiki_database_schema_1-19_%28r102798%29.svg http://upload.wikimedia.org/wikipedia/commons/3/33/MediaWiki... Also, most MediaWiki installs use plugins, and those plugins often create their own database tables. So RDBMS is pretty much baked into the MediaWiki culture. However, MediaWiki is fairly well abstracted from any particular RDBMS implementation. Everything from creating tables to making a join is done through an API.
- tommorris 14y ago
  So, there's some legacy here. 10 years ago, there wasn't anything really like CouchDB.
  - nikcub 14y ago
    Document DB's aren't an invention of the past 10 years. SAS was invented in the 60's, Lotus Notes in the 80's. XML Databases were all the rage during the dotcom bubble. The difference today is that open source projects have made document stores and 'nosql' technologies more accessible.
minimax 14y ago
Several months ago I noticed Wikipedia's Ganglia (cluster performance monitoring software) is publicly accessible. Check out db59 (mentioned in the note) amongst the other servers: http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=MySQL%2520pmtpa&tab=m&vn= http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&... If you want to see a bunch of other cool public Ganglia instances you can do a search something like: https://www.google.com/search?q=%22show+only+nodes+matching%22 https://www.google.com/search?q=%22show+only+nodes+matching%...
- zanny 14y ago
  Any reason why the load on db59 is so high? Are they trying to stress test it?
  - JaggedJax 14y ago
    I couldn't say the reason, but the I/O for db59 [http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=db59.pmtpa.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1355622088&g=network_report&z=large&c=MySQL%20pmtpa http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=...] is much higher than the other servers as well. Also, despite the switch on db59 happening a few days ago, there doesn't seem to be a significant change in overall CPU usage: http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&h=db59.pmtpa.wmnet&m=cpu_report&s=by+name&mc=2&g=cpu_report&c=MySQL+pmtpa http://ganglia.wikimedia.org/latest/graph.php?r=month&z=...
FiloSottile 14y ago
The main goal of migrating to MariaDB is not performance driven. More so, I think it's in WMF's and the open source communities interest to coalesce around the MariaDB Foundation as the best route to ensuring a truly open and well supported future for mysql derived database technology. Performance gains along the way are icing on the cake.
ck2 14y ago
Note that unless your mysql server has more than 4 (physical) cores, you aren't going to see any performance improvements from any variation/fork of mysql. 8 cores is where things start to diverge under load. This website has some benchmarks to prove it: http://vbtechsupport.com/606/ http://vbtechsupport.com/606/ http://vbtechsupport.com/657/2/ http://vbtechsupport.com/657/2/ Also http://www.mysqlperformanceblog.com/2012/02/18/mariadb-5-3-4-benchmarks/ http://www.mysqlperformanceblog.com/2012/02/18/mariadb-5-3-4... But there are other reasons to look at mariadb/percona.
- malinens 14y ago
  What about MariaDB5.5 and MariaDB10.0 performance compared to 5.3?
saosebastiao 14y ago
Why do they care about whether the technology is derived from MySQL or not? Postgres makes way more sense for pretty much every application out there now. What am I missing?
- rubinelli 14y ago
  They aren't starting from scratch. Migrating your database is a complex and risky project, and not the kind of thing you do unless you have a very clear, very pressing reason, like horrible performance and stability issues, or saving tens of millions of dollars in licenses. Migrating to MariaDB is more like upgrading a minor version, with minimal risk and zero downtime. And they have a lot of experience in MySQL, almost all of it transferable to MariaDB.
yuvadam 14y ago
TIL MariaDB is to MySQL what Jenkins is to Hudson.
- altcognito 14y ago
  How long until there is something similar for Java? Let's play name the products that need to be saved from Oracle.
  - tesmar2 14y ago
    open jdk?
  - jzelinskie 14y ago
    Arguably the other JVM-hosted languages are exactly this.
    - sturadnidge 14y ago
      Agreed, but I think s/he meant 'JVM' rather than 'Java'.
      - altcognito 14y ago
        I didn't. But I should have! Thank you. I so often forget those "little" details.
  - jacquesm 14y ago
    How about 'elCaro' ?
damncabbage 14y ago
In case anyone else was wondering about InnoDB like I was: http://en.wikibooks.org/wiki/MariaDB/Storage_Engines http://en.wikibooks.org/wiki/MariaDB/Storage_Engines MariaDB has XtraDB, which is a drop-in replacement for InnoDB; I'm not sure if it's encumbered or not, given it's Percona's fork of InnoDB. (Does anyone have any experience using XtraDB that they can relate?)
- RenierZA 14y ago
  Which storage engine is Wikipedia using (MyISAM, XtraDB or Aria)?

139 points by neya 14y ago

JackWebbHeller 14y ago

Which database system Wikipedia is currently using?

AdamGibbins 14y ago
MySQL
- tommorris 14y ago
  Well, MySQL plus a text store. MediaWiki, the open source software that Wikipedia uses, now officially supports MariaDB.
  - neilk 14y ago
    We're also using OpenStack Swift for media storage these days.
    - mh- 14y ago
      I couldn't see a way to aggregate it in the ganglia linked.. could you share how many mysql (maria) queries per second you guys are averaging?
      - neilk 14y ago
        I should clarify: I used to work there but I sometimes still say "we". I'm not a ganglia wizard - maybe this comes close? Although it's just all selects. I think. http://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mysql_com_select+across+all+db*&vl=stmts&x=&n=&hreg%5B%5D=db&mreg%5B%5D=mysql_com_select&gtype=stack&glegend=show&aggregate=1 http://ganglia.wikimedia.org/latest/graph_all_periods.php?ti... You might try asking on wikitech-l, the same mailing list that the original post comes from.
  - whalesalad 14y ago
    I use MariaDB on my Mac for most things that require MySQL. It's a drop in replacement for most things. Is there a special edge case with media wiki they had to solve or is it now just formally announced as being supported?
    - tommorris 14y ago
      So, it's pretty much a drop-in replacement. But as with everything, time m ust be spent testing and so on before rolling it out to production. I believe a transition to MariaDB has been planned for a while now.
    - malinens 14y ago
      I remember I had problems with insert statements when previously with mysql inserts without default values worked but MariaDB was strict in this case and these inserts failed...
plasma 14y ago
MariaDB is a fork of MySQL by Monty (original MySQL author)

jamespitts 14y ago

Hopefully this will get more people to adopt mariadb and be free of the legal and upgrade uncertainties of mysql. I installed it a few weeks ago and the result is -exactly- like mysql. http://blog.mariadb.org/mariadb-foundation-to-safeguard-leading-open-source-database/ http://blog.mariadb.org/mariadb-foundation-to-safeguard-lead...

sonier 14y ago

How does MariaDB compare to Percona?

mjallday 14y ago
It's a drop in replacement for MySQL, you can use the _storage engines_ that Percona provide such as ExtraDB so maybe it shouldn't differ at all?

revscat 14y ago

Wikipedia seems like the perfect system for a document store like CouchDB. I would be curious to know if they examined it and, if so, found it lacking.

stingraycharles 14y ago
My guess is that they have been using MySQL for a long, long time (possibly since the beginning)m, and see no need to migrate to something different. MariaDB is API-compatible with MySQL, so this migration will not involve much changes at the client side; it's mostly thorough testing that needs to be done.
notatoad 14y ago
MariaDB is a (almost) drop-in replacement for MySQL, CouchDB is a totally different concept that would require nearly a complete re-write of their backend. I doubt they considered it.
neilk 14y ago
MediaWiki uses a fairly typical mostly-normalized relational design. You couldn't easily port it to a document store. http://upload.wikimedia.org/wikipedia/commons/3/33/MediaWiki_database_schema_1-19_%28r102798%29.svg http://upload.wikimedia.org/wikipedia/commons/3/33/MediaWiki... Also, most MediaWiki installs use plugins, and those plugins often create their own database tables. So RDBMS is pretty much baked into the MediaWiki culture. However, MediaWiki is fairly well abstracted from any particular RDBMS implementation. Everything from creating tables to making a join is done through an API.
tommorris 14y ago
So, there's some legacy here. 10 years ago, there wasn't anything really like CouchDB.
- nikcub 14y ago
  Document DB's aren't an invention of the past 10 years. SAS was invented in the 60's, Lotus Notes in the 80's. XML Databases were all the rage during the dotcom bubble. The difference today is that open source projects have made document stores and 'nosql' technologies more accessible.

minimax 14y ago

Several months ago I noticed Wikipedia's Ganglia (cluster performance monitoring software) is publicly accessible. Check out db59 (mentioned in the note) amongst the other servers: http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=MySQL%2520pmtpa&tab=m&vn= http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&... If you want to see a bunch of other cool public Ganglia instances you can do a search something like: https://www.google.com/search?q=%22show+only+nodes+matching%22 https://www.google.com/search?q=%22show+only+nodes+matching%...

zanny 14y ago
Any reason why the load on db59 is so high? Are they trying to stress test it?
- JaggedJax 14y ago
  I couldn't say the reason, but the I/O for db59 [http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=db59.pmtpa.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1355622088&g=network_report&z=large&c=MySQL%20pmtpa http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=...] is much higher than the other servers as well. Also, despite the switch on db59 happening a few days ago, there doesn't seem to be a significant change in overall CPU usage: http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&h=db59.pmtpa.wmnet&m=cpu_report&s=by+name&mc=2&g=cpu_report&c=MySQL+pmtpa http://ganglia.wikimedia.org/latest/graph.php?r=month&z=...

FiloSottile 14y ago

The main goal of migrating to MariaDB is not performance driven. More so, I think it's in WMF's and the open source communities interest to coalesce around the MariaDB Foundation as the best route to ensuring a truly open and well supported future for mysql derived database technology. Performance gains along the way are icing on the cake.

ck2 14y ago

Note that unless your mysql server has more than 4 (physical) cores, you aren't going to see any performance improvements from any variation/fork of mysql. 8 cores is where things start to diverge under load. This website has some benchmarks to prove it: http://vbtechsupport.com/606/ http://vbtechsupport.com/606/ http://vbtechsupport.com/657/2/ http://vbtechsupport.com/657/2/ Also http://www.mysqlperformanceblog.com/2012/02/18/mariadb-5-3-4-benchmarks/ http://www.mysqlperformanceblog.com/2012/02/18/mariadb-5-3-4... But there are other reasons to look at mariadb/percona.

malinens 14y ago
What about MariaDB5.5 and MariaDB10.0 performance compared to 5.3?

saosebastiao 14y ago

Why do they care about whether the technology is derived from MySQL or not? Postgres makes way more sense for pretty much every application out there now. What am I missing?

rubinelli 14y ago
They aren't starting from scratch. Migrating your database is a complex and risky project, and not the kind of thing you do unless you have a very clear, very pressing reason, like horrible performance and stability issues, or saving tens of millions of dollars in licenses. Migrating to MariaDB is more like upgrading a minor version, with minimal risk and zero downtime. And they have a lot of experience in MySQL, almost all of it transferable to MariaDB.

yuvadam 14y ago

TIL MariaDB is to MySQL what Jenkins is to Hudson.

altcognito 14y ago
How long until there is something similar for Java? Let's play name the products that need to be saved from Oracle.
- tesmar2 14y ago
  open jdk?
- jzelinskie 14y ago
  Arguably the other JVM-hosted languages are exactly this.
  - sturadnidge 14y ago
    Agreed, but I think s/he meant 'JVM' rather than 'Java'.
    - altcognito 14y ago
      I didn't. But I should have! Thank you. I so often forget those "little" details.
- jacquesm 14y ago
  How about 'elCaro' ?

damncabbage 14y ago

In case anyone else was wondering about InnoDB like I was: http://en.wikibooks.org/wiki/MariaDB/Storage_Engines http://en.wikibooks.org/wiki/MariaDB/Storage_Engines MariaDB has XtraDB, which is a drop-in replacement for InnoDB; I'm not sure if it's encumbered or not, given it's Percona's fork of InnoDB. (Does anyone have any experience using XtraDB that they can relate?)

RenierZA 14y ago
Which storage engine is Wikipedia using (MyISAM, XtraDB or Aria)?