9 ms·

An incomplete guide to Facebook thrift

54 points by rajeevk 12y ago

zapf 12y ago
Good to see this. I was sold on Thrift but I think the popularity of REST made people take some weird decisions. Internal services talking to each other using REST seemed like a bit too much, a place, where I though services like Thrift would have worked so well.
- thlt 12y ago
  please excuse my ignorance but may I ask why REST is too much compared to Thrift?
- Xorlev 12y ago
  I think you mean HTTP+JSON. REST as a philosophy while applied mostly to HTTP isn't really tied to it. We personally decided to stick with HTTP (even though Thrift would be "superior") since tools are numerous and simple. It's hard to beat curl and a browser. We also found that services end up used in ways that are hard to predict, especially in bash-written tools and the like. Thrift is a barrier to entry. Additionally, you could do HTTP+Thrift. It's possible, it works, but you lose some of the overhead benefits of Thrift at that point (IMO). In my experience, fulfilling a Thrift request with a Thrift server has tighter latencies at much lower CPU usage for similar request loads.
- sbilstein 12y ago
  It's a tough tradeoff between performance flexibility. Relying on HTTP + JSON is very nice when you have lots of teams, codebases in several languages, and want to preserve the ability to talk quickly to any service over HTTP. I like working HTTP but certainly tools like thrift have there. LinkedIn is all about http://rest.li/ http://rest.li/ for our services though I've been working with DropWizard on a side project and really love it.
m0skit0 12y ago
Nice introduction. I selected Thrift as middleware for a business project for the company I'm working on one year ago, so I'm pretty familiar with it. It has some issues but for the most part it is reliable, fast, and extremely flexible. Problem with Thrift is mainly the lack of documentation and examples, so the more, the better.
- tfigment 12y ago
  I have similar experiences and generally like it. We had to fix one ways in Java at one point (now fixed properly) and not all languages have equal support. We wanted SSL support on a .NET service but nothing official there last I checked. FB updated and standardized SSL on C++ but left out other languages by and large.
  - electrum 12y ago
    There is a native version of Thrift for Java: https://github.com/facebook/swift https://github.com/facebook/swift
  - Xorlev 12y ago
    And a Scala one (https://github.com/twitter/scrooge https://github.com/twitter/scrooge, idomatic Scala generated models/parsers) when paired with Finagle/Thrift is pretty great.
wwarren 12y ago
We use Thrift a lot at my company, not just for internal communication, but for our external facing API. Larger companies like Evernote (https://dev.evernote.com/doc/reference/ https://dev.evernote.com/doc/reference/) have shown that Thrift can be operated successfully at an enormous scale. Facebook also re-forked re-open-sourced Apache Thrift (originally a Facebook project) as fbthrift: https://github.com/facebook/fbthrift https://github.com/facebook/fbthrift
- Xorlev 12y ago
  The unfortunate part of public Thrift APIs is that you must either provide the IDL and hope your consumers are smart enough to work with Thrift, or you have to take the Evernote route and provide SDKs for all the languages you care to support. Admittedly, making SDKs for languages is a lot easier when codegen takes care of 80% of the work.
  - lmm 12y ago
    Most options for an API offer you the same choice, no? If you're offering XML either you offer a wsdl and hope consumers are smart enough, or you provide SDKs for all the languages. If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL.
    - Xorlev 12y ago
      > If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL. I disagree. It's best to, but if someone wants to hack together something with python/requests it's not hard to use the returned dictionary. Even with Java, we'll frequently do some requests to get representative payloads, generate models, clean them up and we're on our way. Or you could just use Map<String, Object>.
billowqiu 12y ago
this is an interested introduction.
Xorlev 12y ago
I found out the other day that Packetbeat supports decoding Thrift structs: http://packetbeat.com/docs/configuration.html#thrift-configuration http://packetbeat.com/docs/configuration.html#thrift-configu... Which makes debugging Thrift-based architectures significantly easier.
chuckcode 12y ago
Does anyone know of a data serialization framework that will do the usual structs and arbitrary size matrices? I'd love to use something like thrift for serializing matrices or other larger data over a wire or onto disk but last time I looked thrift, protocol buffers and avro were all struct oriented and there wasn't a good way to parcel up a matrix or large vector. Currently I've just been using hdf5 as the default way to store reasonably large data matrices, cubes, vectors, etc.
- robrenaud 12y ago
  What's wrong with using a repeated float to represent a vector/matrix, perhaps with a repeated int for indices if you are using a sparse representation?
  - chuckcode 12y ago
    Yes, I could take some time and write it myself. Take a look at hdf5[1] and the features they have around meta information, data chunking, page level compression for efficient reading of sub areas of data spaces. It is a non-trivial amount of functionality but allows some pretty amazing performance, for example see what pytables [2] has built on top of it. Now if they had just made it easier to attach it to a wire as well as a file. [1] http://www.hdfgroup.org/HDF5/ http://www.hdfgroup.org/HDF5/ [2] http://www.pytables.org/moin http://www.pytables.org/moin
- jjtheblunt 12y ago
  Lisp (Clojure, ...) reader and pr would do this, no?
wahsd 12y ago
"Facebook re-open sourced the thrift..." cool, let me get right on that before it's re-re-closed sourced.
GauntletWizard 12y ago
This is honestly a better guide to Thrift than we have inside Facebook, though I wish that said more.
- rajeevk 12y ago
  Thanks you.. What more you expected in this?

54 points by rajeevk 12y ago

zapf 12y ago

Good to see this. I was sold on Thrift but I think the popularity of REST made people take some weird decisions. Internal services talking to each other using REST seemed like a bit too much, a place, where I though services like Thrift would have worked so well.

thlt 12y ago
please excuse my ignorance but may I ask why REST is too much compared to Thrift?
Xorlev 12y ago
I think you mean HTTP+JSON. REST as a philosophy while applied mostly to HTTP isn't really tied to it. We personally decided to stick with HTTP (even though Thrift would be "superior") since tools are numerous and simple. It's hard to beat curl and a browser. We also found that services end up used in ways that are hard to predict, especially in bash-written tools and the like. Thrift is a barrier to entry. Additionally, you could do HTTP+Thrift. It's possible, it works, but you lose some of the overhead benefits of Thrift at that point (IMO). In my experience, fulfilling a Thrift request with a Thrift server has tighter latencies at much lower CPU usage for similar request loads.
sbilstein 12y ago
It's a tough tradeoff between performance flexibility. Relying on HTTP + JSON is very nice when you have lots of teams, codebases in several languages, and want to preserve the ability to talk quickly to any service over HTTP. I like working HTTP but certainly tools like thrift have there. LinkedIn is all about http://rest.li/ http://rest.li/ for our services though I've been working with DropWizard on a side project and really love it.

m0skit0 12y ago

Nice introduction. I selected Thrift as middleware for a business project for the company I'm working on one year ago, so I'm pretty familiar with it. It has some issues but for the most part it is reliable, fast, and extremely flexible. Problem with Thrift is mainly the lack of documentation and examples, so the more, the better.

tfigment 12y ago
I have similar experiences and generally like it. We had to fix one ways in Java at one point (now fixed properly) and not all languages have equal support. We wanted SSL support on a .NET service but nothing official there last I checked. FB updated and standardized SSL on C++ but left out other languages by and large.
- electrum 12y ago
  There is a native version of Thrift for Java: https://github.com/facebook/swift https://github.com/facebook/swift
- Xorlev 12y ago
  And a Scala one (https://github.com/twitter/scrooge https://github.com/twitter/scrooge, idomatic Scala generated models/parsers) when paired with Finagle/Thrift is pretty great.

wwarren 12y ago

We use Thrift a lot at my company, not just for internal communication, but for our external facing API. Larger companies like Evernote (https://dev.evernote.com/doc/reference/ https://dev.evernote.com/doc/reference/) have shown that Thrift can be operated successfully at an enormous scale. Facebook also re-forked re-open-sourced Apache Thrift (originally a Facebook project) as fbthrift: https://github.com/facebook/fbthrift https://github.com/facebook/fbthrift

Xorlev 12y ago
The unfortunate part of public Thrift APIs is that you must either provide the IDL and hope your consumers are smart enough to work with Thrift, or you have to take the Evernote route and provide SDKs for all the languages you care to support. Admittedly, making SDKs for languages is a lot easier when codegen takes care of 80% of the work.
- lmm 12y ago
  Most options for an API offer you the same choice, no? If you're offering XML either you offer a wsdl and hope consumers are smart enough, or you provide SDKs for all the languages. If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL.
  - Xorlev 12y ago
    > If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL. I disagree. It's best to, but if someone wants to hack together something with python/requests it's not hard to use the returned dictionary. Even with Java, we'll frequently do some requests to get representative payloads, generate models, clean them up and we're on our way. Or you could just use Map<String, Object>.

billowqiu 12y ago

this is an interested introduction.

Xorlev 12y ago

I found out the other day that Packetbeat supports decoding Thrift structs: http://packetbeat.com/docs/configuration.html#thrift-configuration http://packetbeat.com/docs/configuration.html#thrift-configu... Which makes debugging Thrift-based architectures significantly easier.

chuckcode 12y ago

Does anyone know of a data serialization framework that will do the usual structs and arbitrary size matrices? I'd love to use something like thrift for serializing matrices or other larger data over a wire or onto disk but last time I looked thrift, protocol buffers and avro were all struct oriented and there wasn't a good way to parcel up a matrix or large vector. Currently I've just been using hdf5 as the default way to store reasonably large data matrices, cubes, vectors, etc.

robrenaud 12y ago
What's wrong with using a repeated float to represent a vector/matrix, perhaps with a repeated int for indices if you are using a sparse representation?
- chuckcode 12y ago
  Yes, I could take some time and write it myself. Take a look at hdf5[1] and the features they have around meta information, data chunking, page level compression for efficient reading of sub areas of data spaces. It is a non-trivial amount of functionality but allows some pretty amazing performance, for example see what pytables [2] has built on top of it. Now if they had just made it easier to attach it to a wire as well as a file. [1] http://www.hdfgroup.org/HDF5/ http://www.hdfgroup.org/HDF5/ [2] http://www.pytables.org/moin http://www.pytables.org/moin
jjtheblunt 12y ago
Lisp (Clojure, ...) reader and pr would do this, no?

wahsd 12y ago

"Facebook re-open sourced the thrift..." cool, let me get right on that before it's re-re-closed sourced.

GauntletWizard 12y ago

This is honestly a better guide to Thrift than we have inside Facebook, though I wish that said more.

rajeevk 12y ago
Thanks you.. What more you expected in this?