5 ms·

Migrating Messenger storage to optimize performance

104 points by tagx 8y ago

philip1209 8y ago
Great, now they can serve video ads in Messenger more easily.
- aylmao 8y ago
  This has nothing to do with video though.
tomnipotent 8y ago
TL;DR - Moved from HBase to MyRocks engine on MySQL sitting on top of NVMe storage via their Lightning Server [1], which is a JBOF (just a bunch of flash) setup using x16 PCIe. [1] https://code.facebook.com/posts/989638804458007/introducing-lightning-a-flexible-nvme-jbof/ https://code.facebook.com/posts/989638804458007/introducing-...
- godelmachine 8y ago
  Have they published a paper on this on Facebook Research?
  - midom 8y ago
    doesn't really qualify much as research - just lots of engineering to cover all corners
  - jensvdh 8y ago
    Social media apps have "Research" now?
    - tomnipotent 8y ago
      You are aware that Facebook is the largest website on the planet?
    - alexbeloi 8y ago
      Yes they do/have research. Facebook had 12 papers at the last computer vision conference I attended (CVPR '17) and won the awards for best paper and best student paper.
    - wenc 8y ago
      They have a very strong research arm covering a significant number of research areas. https://research.fb.com/ https://research.fb.com/
- midom 8y ago
  + app server / schema rework
- wenc 8y ago
  Anyone know why Facebook moved off HBase? (article doesn't address this) I get that MyRocks is truly amazing, but I'm wondering what issues they were facing with HBase. I heard HBase was picked over Cassandra (developed at FB) because it was strongly consistent vs Cassandra's eventual consistency.
  - tomnipotent 8y ago
    My guess is operational experience, though the article mentions it was necessary to take advantage of Lightning. I can't think of another business on the planet that has pushed MySQL (and RDBMS' in general) further than FB.
  - ryanobjc 8y ago
    at the time they picked HBase because it was easier to repair missing replicas. Hdfs namenode juat schedules block replication. Cassandra (at least the version they were considering) had to work a lot harder at replacing a missing replica. Remember they had a version that didn’t have workable merkel trees. I also heard they had an issue with network flaps causing unrepairable inconsistency. I never got the full scoop on this. I had worked with the engineers directly responsible for the HBase choice back when. All water under the bridge now it seems.
oerpli 8y ago
They say that they brought some new search features to mobile, though I would prefer if they fixed their search on desktop (or in general) first, as it only returns results from recent messages. Until sometime in 2016/2017 it was possible to look for specific things at whatever time (e.g. 2011) and now even things from <3y ago don't return anything.
- spyspy 8y ago
  I don't see the ability to search within the app. Have they not rolled it out yet?
  - kalleboo 8y ago
    On iOS I see a search bar at the top of the home screen. Type in it and it searches contacts, but there's a "search in messages" item you can tap and it shows matching chats and messages.
- fao_ 8y ago
  One solution is to download an archive and grep the JSON returned
bigato 8y ago
What about giving me an easy option to export my messages while you are at it?
- mtgx 8y ago
  Unfortunately, the feature to export your messages is only available to Facebook's data partners.
  - djohnston 8y ago
    your ignorance adds nothing of value to this conversation. facebook data partners do not have a download your data button. that's not how any of this works.
    - midom 8y ago
      based on his previous submissions, he did not come here to discuss database technology or be factually correct.
  - dang 8y ago
    This sort of shallow dismissal does nobody on any side of any argument any good. We've asked you many times to stop posting unsubstantive comments to HN. If you can't or won't stop, we will ban you. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- tagx 8y ago
  You can use the download your information tool to download all your data on FB, including all your messages https://www.facebook.com/help/212802592074644 https://www.facebook.com/help/212802592074644 A Brooklyn based artist did this recently and made a book of his conversation history (and covered an apartment with some of the pages). You can find a bit more information and pictures at https://www.halfspace.uk/messages-2011-2017 https://www.halfspace.uk/messages-2011-2017
  - bigato 8y ago
Zaheer 8y ago
During the dual-write phase, what happens if one request succeeds while the other doesn't?
- tagx 8y ago
  Iris retries the failed request
  - Zaheer 8y ago
    Can't you potentially land into an inconsistent state if the failed request keeps on failing while the other db request succeeds? Is the dual-write on the synchronous request path? I'm assuming there's a timeout or some fixed number of retries that occurs and once that is exhausted what would happen?
nemothekid 8y ago
Facebook has been using RocksDB more and more (now messenger, but also Instragram’s flavor of Cassandra). I wonder if Google has a similar penetration of LevelDB (and we just don’t hear about it because Google’s infrastructure work isn’t open source)
- asfasgasg 8y ago
  No, it does not. Generally speaking, Google "converges" a layer up (BigTable, Spanner, Colossus). There is no layer like LevelDB that is common to all of these.
- rakoo 8y ago
  There seems to be a pattern with Google: they have internal infrastructure that every ex-Googler seem to miss when they leave the company, because it's so good that it felt like being 10 years in the future. As far as I understand, Google doesn't have a bunch of tools that merely work together, they have one huge system with different bits that _live_ together, so much that separating and open-sourcing them is cool but won't give you the same thing as being from inside: - They use Blaze, a build system that integrates directly with the object store. They open-sourced Bazel as a kind of equivalent, but the build system won't shine unless you have an integrated object store and an integrated vcs client - They have open-sourced Kubernetes, a successor of what they were using for they were using internally for cloud management - They have open-sourced LevelDB, a successor of the fundamental brick they are using for BigTable So in a way LevelDB isn't used as-is inside Google, but its spirit is in use at a fundamental level by pretty much everyone
dingo_bat 8y ago
Whatever you may think of social networking, Facebook takes huge amount of pains to ensure reliability. They moved all the data from spinning disks to flash, also migrating to a new db. No downtime at all. I'm a bit unclear as to how this process resulted in 90% decreased storage use.
- martincmartin 8y ago
  "The simplified data schema directly reduced the size of data on disk. We saved additional space in MyRocks by applying Zstandard, a state-of-the-art lossless data compression algorithm developed by Facebook. We were able to reduce the replication factor from six to three, thanks to differences between the HBase and MyRocks architectures. In total, we reduced storage consumption by 90 percent without data loss, thereby making it practical to use flash storage."

104 points by tagx 8y ago

philip1209 8y ago

Great, now they can serve video ads in Messenger more easily.

aylmao 8y ago
This has nothing to do with video though.

tomnipotent 8y ago

TL;DR - Moved from HBase to MyRocks engine on MySQL sitting on top of NVMe storage via their Lightning Server [1], which is a JBOF (just a bunch of flash) setup using x16 PCIe. [1] https://code.facebook.com/posts/989638804458007/introducing-lightning-a-flexible-nvme-jbof/ https://code.facebook.com/posts/989638804458007/introducing-...

godelmachine 8y ago
Have they published a paper on this on Facebook Research?
- midom 8y ago
  doesn't really qualify much as research - just lots of engineering to cover all corners
- jensvdh 8y ago
  Social media apps have "Research" now?
  - tomnipotent 8y ago
    You are aware that Facebook is the largest website on the planet?
  - alexbeloi 8y ago
    Yes they do/have research. Facebook had 12 papers at the last computer vision conference I attended (CVPR '17) and won the awards for best paper and best student paper.
  - wenc 8y ago
    They have a very strong research arm covering a significant number of research areas. https://research.fb.com/ https://research.fb.com/
midom 8y ago
+ app server / schema rework
wenc 8y ago
Anyone know why Facebook moved off HBase? (article doesn't address this) I get that MyRocks is truly amazing, but I'm wondering what issues they were facing with HBase. I heard HBase was picked over Cassandra (developed at FB) because it was strongly consistent vs Cassandra's eventual consistency.
- tomnipotent 8y ago
  My guess is operational experience, though the article mentions it was necessary to take advantage of Lightning. I can't think of another business on the planet that has pushed MySQL (and RDBMS' in general) further than FB.
- ryanobjc 8y ago
  at the time they picked HBase because it was easier to repair missing replicas. Hdfs namenode juat schedules block replication. Cassandra (at least the version they were considering) had to work a lot harder at replacing a missing replica. Remember they had a version that didn’t have workable merkel trees. I also heard they had an issue with network flaps causing unrepairable inconsistency. I never got the full scoop on this. I had worked with the engineers directly responsible for the HBase choice back when. All water under the bridge now it seems.

oerpli 8y ago

They say that they brought some new search features to mobile, though I would prefer if they fixed their search on desktop (or in general) first, as it only returns results from recent messages. Until sometime in 2016/2017 it was possible to look for specific things at whatever time (e.g. 2011) and now even things from <3y ago don't return anything.

spyspy 8y ago
I don't see the ability to search within the app. Have they not rolled it out yet?
- kalleboo 8y ago
  On iOS I see a search bar at the top of the home screen. Type in it and it searches contacts, but there's a "search in messages" item you can tap and it shows matching chats and messages.
fao_ 8y ago
One solution is to download an archive and grep the JSON returned

bigato 8y ago

What about giving me an easy option to export my messages while you are at it?

mtgx 8y ago
Unfortunately, the feature to export your messages is only available to Facebook's data partners.
- djohnston 8y ago
  your ignorance adds nothing of value to this conversation. facebook data partners do not have a download your data button. that's not how any of this works.
  - midom 8y ago
    based on his previous submissions, he did not come here to discuss database technology or be factually correct.
- dang 8y ago
  This sort of shallow dismissal does nobody on any side of any argument any good. We've asked you many times to stop posting unsubstantive comments to HN. If you can't or won't stop, we will ban you. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
tagx 8y ago
You can use the download your information tool to download all your data on FB, including all your messages https://www.facebook.com/help/212802592074644 https://www.facebook.com/help/212802592074644 A Brooklyn based artist did this recently and made a book of his conversation history (and covered an apartment with some of the pages). You can find a bit more information and pictures at https://www.halfspace.uk/messages-2011-2017 https://www.halfspace.uk/messages-2011-2017
- bigato 8y ago

Zaheer 8y ago

During the dual-write phase, what happens if one request succeeds while the other doesn't?

tagx 8y ago
Iris retries the failed request
- Zaheer 8y ago
  Can't you potentially land into an inconsistent state if the failed request keeps on failing while the other db request succeeds? Is the dual-write on the synchronous request path? I'm assuming there's a timeout or some fixed number of retries that occurs and once that is exhausted what would happen?

nemothekid 8y ago

Facebook has been using RocksDB more and more (now messenger, but also Instragram’s flavor of Cassandra). I wonder if Google has a similar penetration of LevelDB (and we just don’t hear about it because Google’s infrastructure work isn’t open source)

asfasgasg 8y ago
No, it does not. Generally speaking, Google "converges" a layer up (BigTable, Spanner, Colossus). There is no layer like LevelDB that is common to all of these.
rakoo 8y ago
There seems to be a pattern with Google: they have internal infrastructure that every ex-Googler seem to miss when they leave the company, because it's so good that it felt like being 10 years in the future. As far as I understand, Google doesn't have a bunch of tools that merely work together, they have one huge system with different bits that _live_ together, so much that separating and open-sourcing them is cool but won't give you the same thing as being from inside: - They use Blaze, a build system that integrates directly with the object store. They open-sourced Bazel as a kind of equivalent, but the build system won't shine unless you have an integrated object store and an integrated vcs client - They have open-sourced Kubernetes, a successor of what they were using for they were using internally for cloud management - They have open-sourced LevelDB, a successor of the fundamental brick they are using for BigTable So in a way LevelDB isn't used as-is inside Google, but its spirit is in use at a fundamental level by pretty much everyone

dingo_bat 8y ago

Whatever you may think of social networking, Facebook takes huge amount of pains to ensure reliability. They moved all the data from spinning disks to flash, also migrating to a new db. No downtime at all. I'm a bit unclear as to how this process resulted in 90% decreased storage use.

martincmartin 8y ago
"The simplified data schema directly reduced the size of data on disk. We saved additional space in MyRocks by applying Zstandard, a state-of-the-art lossless data compression algorithm developed by Facebook. We were able to reduce the replication factor from six to three, thanks to differences between the HBase and MyRocks architectures. In total, we reduced storage consumption by 90 percent without data loss, thereby making it practical to use flash storage."