6 ms·

According to this article [1], you plan to move Tumblr's backend onto WordPress. Considering that Tumblr's infra stores over 1 trillion distinct product objects

According to this article [1], you plan to move Tumblr's backend onto WordPress. Considering that Tumblr's infra stores over 1 trillion distinct product objects, this would be one of the most technically ambitious migrations in history. Can you share any thoughts of how it will be approached? Will you be pruning/purging old content or inactive users? [1] https://poststatus.com/automattic-has-purchased-tumblr/ https://poststatus.com/automattic-has-purchased-tumblr/

joshmn 7y ago
I'm curious: What approach, if any, would you take? I'm sure you have an interesting prospective based on your experience working at MySQL-friendly shops, in addition to being an engineer at Tumblr once upon a time.
- evanelias 7y ago
  Thank you for asking, but I probably shouldn't get into that :) I'm inherently conflicted / biased due to designing a solid chunk of Tumblr's backend, and also previously worked for another Automattic competitor (Six Apart, the defunct company behind Movable Type). Edit to avoid anyone misconstruing, I'm not trying to imply one thing or another, just that I can't approach this impartially. And in any case, I wish everyone well on both sides of this acquisition. I'm just genuinely curious how they plan to proceed from a technical standpoint, as it's a really interesting challenge.
  - mikeschinkel 7y ago
    Just curious about your just curious; what is the architecture of the current Tumblr back end? This appears to be 6 years old. Is it still relevant? http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur...
    - evanelias 7y ago
      Ehhh parts of that article were never accurate, especially the stuff about having an hbase-powered dashboard feed. Primarily the product backend is monolithic PHP (custom framework) + services in various languages + sharded MySQL + Memcached + Gearman. Lots of other technologies in use though too, but I'll defer to current employees if they want to answer.
      - mikeschinkel 7y ago
        Cool, thanks for the answer.
      - jsjohnst 7y ago
        You’re being generous. Most of that article was a pipe dream that never came to fruition.
      - JulianMorrison 7y ago
        Fantasy big data: let's use Hadoop and Kafka! Reality big data: Let's shard it across Mysql.
        evanelias 7y ago
        Not exactly. Tumblr has a pretty huge Hadoop fleet and decently large Kafka setup too. It's just a question of OLTP vs OLAP use-cases being powered by different tech stacks. My answer above was limited to the product backend, i.e. technologies used in serving user requests in real-time. And even then I missed a bunch of large technologies in use there, especially around search and algorithmic ranking.
        lugg 7y ago
        That's kind of the point though. Everyone has a Hadoop/Kafka, but when it comes to actually getting things done, good ole MySQL to the rescue. I honestly don't see the draw for Kafka. And by all means I get it, I just don't buy it. Maybe I'm just holding it wrong or something.
georgestephanis 7y ago
I have no idea precisely what we're going to do or how, but if I were spitballing, I'd think something like ... Currently I think Tumblr stores all posts across all sites in one big table? WP.com does different tables per site. I also think Tumblr's post ids are often above php's int max for 32-bit systems ( 2,147,483,647 ) -- I know I've seen some issues trying to parse tumblr's post ids to integers rather than strings on some old servers years ago. For an overview of how our systems are run here's Barry, our head sysadmin, talking about six years ago on how the wpcom infastructure is structured: https://www.youtube.com/watch?v=57EJ8KDDBH0 https://www.youtube.com/watch?v=57EJ8KDDBH0 It's changed somewhat, but not much conceptually. It's a really fascinating talk and I'd encourage anyone curious on massive scale data to give it a look, and see precisely what can be managed with determination and mysql and coffee.
- raverbashing 7y ago
  Something about its heydays http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur...
- progforlyfe 7y ago
  Close; For the most part, to the application the Posts appear to be from one table since the lookup interface is the same (externally), but in reality at this massive scale, it just wouldn't be possible in a single MySQL table and database, even if you had a huge number of replicas. Tumblr's posts are spread across MANY "shards" which are actually different servers, each running a chunk of posts, shared by the blog owner. e.g. Blogs 1 - 10,000,000 on Shard 1, 10,000,001 to 20,000,000 on Shard 2, etc. More in depth talk here http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur... and http://assets.en.oreilly.com/1/event/74/Massively%20Sharded%20MySQL%20at%20Tumblr%20Presentation.pdf http://assets.en.oreilly.com/1/event/74/Massively%20Sharded%... though it's from 2011/2012, but the overall ideas still hold. At the time of that post (7 years ago) there were already 200 Database servers. Thankfully nothing is 32-bit so no worries about integer overflows. That would cause huge headaches everywhere on the PHP side. In MySQL, a regular unsigned INT column does have that limitation (roughly max 4.5 billion for unsigned, 2.2 signed), so BIGINT must be used there (Twitter had to do the same). Where it gets interesting is PHP doesn't support unsigned integers, so with 64 bit your max integer in PHP is 9,223,372,036,854,775,807, whereas in MySQL an unsigned 64 bit int is double that. I think it's safe to say though that neither Tumblr nor WordPress, even combined, would ever have more posts than atoms on Earth =)
photomatt 7y ago
That's an excellent question! I don't want to be so presumptuous as to define an exact approach before the technical exploration has started, besides saying it'll be done incrementally and in an easily revertable way to be invisible to users, just like the big datacenter migration Automattic just completed a few weeks ago. At the point when we start this the Tumblr team will have been part of Automattic for the better part of a year if not more, so there will be a lot of learning and evolution of the products on both sides to make any migration easier. I promise we'll write about it afterward for anyone who is curious.
- kill_nate_kill 7y ago
  You should almost make it a documentary.
  - duncanawoods 7y ago
    Yes! I’d pay more for real world business documentaries than anything on netflix.
    - echelon 7y ago
      Please actually do this. Find a way to get Verizon to sign off on this, and then get in touch with an established documentary maker. Pair them with an engineer and follow the story of the migration efforts. It will take time, and it'll certainly have a narrative. Nothing like this has been done before. I struggle with making what I do relatable to people, but having a technical or semi-technical documentary following this large project would be eye-opening. We'll even crowd fund this if you give us the chance. I'm not kidding. Please, please, please make this migration a documentary film.
      - artpi 7y ago
        From the documentaries I've seen, its lots of people walking to a meeting, meetings themselves, etc. For that kind of documentary, probably people walking into servers rooms, or having heated discussions. In Automattic, we basically evolved to remove all that :) There would be basically zoom calls and slack discussions. The most ambitious project I worked on in Automattic were just me, looking at the code and trying to understand why something is happening. Or looking up Stripe documentation. We get to sit in front of our laptops in nice places though :)
anentropic 7y ago
Wow, that sounds insane
nthot 7y ago
The fact that they are already planning on making changes seems to run counter to the idea that they would be taking a "Berkshire Hathaway" approach. In my understanding Warren Buffett's current philosophy is to buy companies that need his capital, potentially his name, but definitely not his intervention.

joshmn 7y ago

I'm curious: What approach, if any, would you take? I'm sure you have an interesting prospective based on your experience working at MySQL-friendly shops, in addition to being an engineer at Tumblr once upon a time.

evanelias 7y ago
Thank you for asking, but I probably shouldn't get into that :) I'm inherently conflicted / biased due to designing a solid chunk of Tumblr's backend, and also previously worked for another Automattic competitor (Six Apart, the defunct company behind Movable Type). Edit to avoid anyone misconstruing, I'm not trying to imply one thing or another, just that I can't approach this impartially. And in any case, I wish everyone well on both sides of this acquisition. I'm just genuinely curious how they plan to proceed from a technical standpoint, as it's a really interesting challenge.
- mikeschinkel 7y ago
  Just curious about your just curious; what is the architecture of the current Tumblr back end? This appears to be 6 years old. Is it still relevant? http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur...
  - evanelias 7y ago
    Ehhh parts of that article were never accurate, especially the stuff about having an hbase-powered dashboard feed. Primarily the product backend is monolithic PHP (custom framework) + services in various languages + sharded MySQL + Memcached + Gearman. Lots of other technologies in use though too, but I'll defer to current employees if they want to answer.
    - mikeschinkel 7y ago
      Cool, thanks for the answer.
    - jsjohnst 7y ago
      You’re being generous. Most of that article was a pipe dream that never came to fruition.
    - JulianMorrison 7y ago
      Fantasy big data: let's use Hadoop and Kafka! Reality big data: Let's shard it across Mysql.
      - evanelias 7y ago
        Not exactly. Tumblr has a pretty huge Hadoop fleet and decently large Kafka setup too. It's just a question of OLTP vs OLAP use-cases being powered by different tech stacks. My answer above was limited to the product backend, i.e. technologies used in serving user requests in real-time. And even then I missed a bunch of large technologies in use there, especially around search and algorithmic ranking.
        lugg 7y ago
        That's kind of the point though. Everyone has a Hadoop/Kafka, but when it comes to actually getting things done, good ole MySQL to the rescue. I honestly don't see the draw for Kafka. And by all means I get it, I just don't buy it. Maybe I'm just holding it wrong or something.

georgestephanis 7y ago

I have no idea precisely what we're going to do or how, but if I were spitballing, I'd think something like ... Currently I think Tumblr stores all posts across all sites in one big table? WP.com does different tables per site. I also think Tumblr's post ids are often above php's int max for 32-bit systems ( 2,147,483,647 ) -- I know I've seen some issues trying to parse tumblr's post ids to integers rather than strings on some old servers years ago. For an overview of how our systems are run here's Barry, our head sysadmin, talking about six years ago on how the wpcom infastructure is structured: https://www.youtube.com/watch?v=57EJ8KDDBH0 https://www.youtube.com/watch?v=57EJ8KDDBH0 It's changed somewhat, but not much conceptually. It's a really fascinating talk and I'd encourage anyone curious on massive scale data to give it a look, and see precisely what can be managed with determination and mysql and coffee.

raverbashing 7y ago
Something about its heydays http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur...
progforlyfe 7y ago
Close; For the most part, to the application the Posts appear to be from one table since the lookup interface is the same (externally), but in reality at this massive scale, it just wouldn't be possible in a single MySQL table and database, even if you had a huge number of replicas. Tumblr's posts are spread across MANY "shards" which are actually different servers, each running a chunk of posts, shared by the blog owner. e.g. Blogs 1 - 10,000,000 on Shard 1, 10,000,001 to 20,000,000 on Shard 2, etc. More in depth talk here http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur... and http://assets.en.oreilly.com/1/event/74/Massively%20Sharded%20MySQL%20at%20Tumblr%20Presentation.pdf http://assets.en.oreilly.com/1/event/74/Massively%20Sharded%... though it's from 2011/2012, but the overall ideas still hold. At the time of that post (7 years ago) there were already 200 Database servers. Thankfully nothing is 32-bit so no worries about integer overflows. That would cause huge headaches everywhere on the PHP side. In MySQL, a regular unsigned INT column does have that limitation (roughly max 4.5 billion for unsigned, 2.2 signed), so BIGINT must be used there (Twitter had to do the same). Where it gets interesting is PHP doesn't support unsigned integers, so with 64 bit your max integer in PHP is 9,223,372,036,854,775,807, whereas in MySQL an unsigned 64 bit int is double that. I think it's safe to say though that neither Tumblr nor WordPress, even combined, would ever have more posts than atoms on Earth =)

photomatt 7y ago

That's an excellent question! I don't want to be so presumptuous as to define an exact approach before the technical exploration has started, besides saying it'll be done incrementally and in an easily revertable way to be invisible to users, just like the big datacenter migration Automattic just completed a few weeks ago. At the point when we start this the Tumblr team will have been part of Automattic for the better part of a year if not more, so there will be a lot of learning and evolution of the products on both sides to make any migration easier. I promise we'll write about it afterward for anyone who is curious.

kill_nate_kill 7y ago
You should almost make it a documentary.
- duncanawoods 7y ago
  Yes! I’d pay more for real world business documentaries than anything on netflix.
  - echelon 7y ago
    Please actually do this. Find a way to get Verizon to sign off on this, and then get in touch with an established documentary maker. Pair them with an engineer and follow the story of the migration efforts. It will take time, and it'll certainly have a narrative. Nothing like this has been done before. I struggle with making what I do relatable to people, but having a technical or semi-technical documentary following this large project would be eye-opening. We'll even crowd fund this if you give us the chance. I'm not kidding. Please, please, please make this migration a documentary film.
    - artpi 7y ago
      From the documentaries I've seen, its lots of people walking to a meeting, meetings themselves, etc. For that kind of documentary, probably people walking into servers rooms, or having heated discussions. In Automattic, we basically evolved to remove all that :) There would be basically zoom calls and slack discussions. The most ambitious project I worked on in Automattic were just me, looking at the code and trying to understand why something is happening. Or looking up Stripe documentation. We get to sit in front of our laptops in nice places though :)

anentropic 7y ago

Wow, that sounds insane

nthot 7y ago

The fact that they are already planning on making changes seems to run counter to the idea that they would be taking a "Berkshire Hathaway" approach. In my understanding Warren Buffett's current philosophy is to buy companies that need his capital, potentially his name, but definitely not his intervention.