13 ms·
Facebook open sources Haxl
- radnam 12y agoCool ! Are there any alternatives out there, especially to deal with fault tolerance especially when if we want to establish connection with 100's of varied databases?
- nbm 12y agoThe release blog post is here - https://code.facebook.com/posts/302060973291128/open-sourcing-haxl-a-library-for-haskell/ https://code.facebook.com/posts/302060973291128/open-sourcin... It contains a lot more information about the problem it was originally created to solve, and potential other use cases.
- AaronFriel 12y agoThis is a fascinating project. Haxl is the brainchild of former Glasgow Haskell Compiler lead* Simon Marlow. The tl;dr of Haxl: what if you could describe accessing a data store (a la SQL) and have the compiler and library work together to "figure out" the most efficient way to perform queries, including performing multiple queries in parallel? That's what Haxl does, it allows you to specify the "shape" of your query, the type checker verifies its correctness, and the library executes it in parallel for you, without the developer having to know about synchronizing access or anything. Here's a link to their paper (PDF): http://www.haskell.org/wikiupload/c/cf/The_Haxl_Project_at_Facebook.pdf http://www.haskell.org/wikiupload/c/cf/The_Haxl_Project_at_F... * - I am not sure if he's still committing, or if he's only doing application development. His accomplishments in Haskell land though, are many. Edited: I removed my comment about GitHub issues, seems it's a known problem. :)
- carlio 12y agoGitHub is currently having problems - https://twitter.com/githubstatus https://twitter.com/githubstatus
- simonmar 12y agoHe is still committing, but not quite so often :)
- vamega 12y agoHe also wrote a book called Parallel and Concurrent Programming in Haskell[1], which I've heard is an excellent read[2]. [1] http://chimera.labs.oreilly.com/books/1230000000929 http://chimera.labs.oreilly.com/books/1230000000929 [2] http://www.serpentine.com/blog/2014/03/18/book-review-parallel-and-concurrent-programming-in-haskell/ http://www.serpentine.com/blog/2014/03/18/book-review-parall...
- freyrs3 12y agoIt's a great read indeed. If anyone is interested in more concrete applications of Haskell then a read through this should be enough to convince anyone that we can do some really amazing parallel programming on top of Haskell's RTS.
- pjc50 12y agoIsn't that what an SQL query planner does?
- AaronFriel 12y agoYes, but this is for non-SQL systems, and it does quite a bit more. It's more like an ORM in this respect, because it weaves together the query planning and processing part which handles retrieving data, and the execution of operating on that data.
- thisisdave 12y agoI feel like I'm missing something. I haven't used databases much, but don't most SQL implementations already "figure out" the most efficient way to perform queries? Can't most implementations already perform queries in parallel?
- AaronFriel 12y agoYes, but this is designed for non-SQL datastores, and arbitrary application code connecting portions of "data acquisition" and "data operation". Imagine if instead of an ORM, you had a single system that weaved together your application code and the database queries, and ensured that where they could execute in parallel, they did.
- thisisdave 12y agoThat makes sense. Thanks
- emmanueloga_ 12y agoreminds me of http://ql.io/ http://ql.io/ (which does not seem to have taken off... last commit: Apr 24, 2013).
- lbrandy 12y agoHi. I'm one of the engineers who has worked on this so if anyone has any specific questions I can help answer and/or get someone to answer. As said by @nbm, we also have a blog post up: https://code.facebook.com/posts/302060973291128/open-sourcing-haxl-a-library-for-haskell/ https://code.facebook.com/posts/302060973291128/open-sourcin....
- Igglyboo 12y agoI had no idea you guys used Haskell at FB. Other than Haxl, what are your main use cases for Haskell?
- simonmar 12y agoHaxl is the only major project at FB using Haskell right now, but who knows where this will lead?
- evmar 12y agoHave you replaced all use of FXL with Haxl? Or are both languages supported? In the latter case, what is the relative proportion of each in the live codebase? (I appreciate that migrating code that already works to a new language often just introduces bugs for no gain, so please don't take my questions as trying to dig up dirt or anything. I'm genuinely just curious.)
- evincarofautumn 12y agoWe are still in the process of migrating from FXL to Haxl, and there is too much FXL code to translate manually, so at the moment we are treating FXL as the source of truth, compiling the FXL codebase to Haskell, and running both concurrently to verify correctness.
- deleted 12y ago[deleted]
- awda 12y agoWhy Haskell?
- JonCoens 12y agoI'm an engineer on the Haxl project and am really excited to launch this today. Ask me anything!
- ciferkey 12y agoHow large has the haskell team at Facebook gotten (unless there isn't an official group and its on a project by project bases)?
- JonCoens 12y agoHaxl is the only team using Haskell in prod, so the team itself isn't all that large. With the traction we gain, though, we could grow.
- Touche 12y agoI'm confused, Haxl is an open source library, right? So what are you using Haxl for at facebook?
- nbm 12y agoThe blog post - https://code.facebook.com/posts/302060973291128/open-sourcing-haxl-a-library-for-haskell/ https://code.facebook.com/posts/302060973291128/open-sourcin... - describes the initial usage - to assist the Sigma system in answering questions like "Is this content spam?". Some background on that system, including the FXL component that Haxl replaces, is here: http://research.microsoft.com/en-us/projects/ldg/a10-stein.pdf http://research.microsoft.com/en-us/projects/ldg/a10-stein.p...
- SkyMarshal 12y agoIs Bryan O'Sullivan and the team from his Haskell-based startup Facebook acquired in 2011 still there? I sat in on a class of his a while back and remember him ruefully laughing about having to use PHP now.
- 12y ago
- EGreg 12y agohttp://platform.qbix.com/guide/patterns http://platform.qbix.com/guide/patterns ;-)
- ixmatus 12y agoYou obviously don't understand what Haxl is for.
- EGreg 12y agoWhy is that obvious to you? Do you care to explain your cryptic comment?
- ixmatus 12y agoWell, that project, Q just doesn't even share the same thesis of Haxl. At all. Two completely different projects, Q couldn't be used for what Haxl is intended. If you had posted the link asking for a comparison, that would've been different, but your ignorance seemed obvious to me because of the haughty wink at the end of your posted link implying we were being let in on some kind of secret.
- EGreg 12y agoHow do you guys batch requests in PHP? You don't, right? So this is an intermediate layer basically, and it sends requests every few millisecods and waits to batch things in between?
- nbm 12y agoThe systems described here isn't directly involved with any PHP code. Like all Facebook services, they are communicated with over the Thrift RPC system, and may have PHP (or any other language) clients, and may talk to other services using Thrift (or occasionally other protocols), some of which may use PHP. If you're asking a general question about batching requests in PHP, http://docs.hhvm.com/manual/en/hack.async.php http://docs.hhvm.com/manual/en/hack.async.php may be informative.
- EGreg 12y agoCool so Hack supports async while php doesn't.
- fnord_murks 12y agoIf you want to have similar query parallelization magic for PHP+SQL have a look at the standalone SQLTap service written by the guys from DaWanda.com (https://github.com/paulasmuth/sqltap https://github.com/paulasmuth/sqltap).
- simonmar 12y agoHere's our paper about the ideas behind Haxl: http://community.haskell.org/~simonmar/papers/haxl-icfp14.pdf http://community.haskell.org/~simonmar/papers/haxl-icfp14.pd...
- seanmcdirmid 12y agoSomehow I think this should have been upvoted to the top.
- untothebreach 12y agoSimon Marlow and Sean McDirmid in the same thread? Be still my heart!
- vdijkbas 12y agoAlso see the slides of Simon's talk at ZuriHac 2014 last weekend: http://www.haskell.org/haskellwiki/ZuriHac2014#Talk_by_Simon_Marlow http://www.haskell.org/haskellwiki/ZuriHac2014#Talk_by_Simon...
- edofic 12y agoSlides from Marlow from 9 months ago about Haxl and it's workings https://github.com/meiersi/HaskellerZ/blob/master/meetups/20130829-FPAfternoon_The_Haxl_Project_at_Facebook/The%20Haxl%20Project%20at%20Facebook.pdf?raw=true https://github.com/meiersi/HaskellerZ/blob/master/meetups/20...
- skyahead 12y agoare there similar things for iOS?
- fiatjaf 12y agohttp://hackage.haskell.org/package/haxl http://hackage.haskell.org/package/haxl Why do Haskell libraries on Hackage doesn't come even with a single example, getting started, how to use, quick start, nothing, really, just function declarations? This scares Haskell newbies.
- m0nastic 12y agoThe "documentation" on Hackage is almost universally just the haddock-generated files (which is why it's mostly just function declarations and type signatures). Most libraries list a "Home Page" that more often than not includes more useful documentation (Haxl's, for example, has the things you've mentioned). I concur, that most of the time, the documentation on Hackage isn't really sufficient, but I've found that for the most part I just use it to find the homepage, and then go there to read the actual documentation. I agree that it would be nice if everything was all in one place.
- dllthomas 12y ago"I agree that it would be nice if everything was all in one place." I actually find "distilled reference with links to source" a fantastically valuable view. I've no objection to providing some sort of combined view, but let's not lose what we have in a quest for consolidation. I've no idea if that's what you meant or not, and don't mean to put words in your mouth of course, just expressing a concern.
- m0nastic 12y agoI agree it's hard to be all things to all people. I think fundamentally, Hackage is meant to be a centralized package repository. If you look at other similar projects, there seems to be no real consensus as to whether that should just be a launching-off point to the actual project page, or more inclusive. When I originally wrote this, I was going to say "It's akin to CPAN", but then to make sure I wasn't misremembering, I looked at a bunch of CPAN packages and saw that they were all actually fully-documented (with examples and whatnot). I think the advantage to having consolidation is that you can then at least try to enforce documentation standards (whether you should, is arguable). What's super frustrating is going to a Hackage page, finding the link to the project home page (frequently on GitHub), going to the GitHub page and then just seeing a barren directory listing of files. I feel slightly uncomfortable making statements about how Hackage should be set up however, as it's like going to a soup kitchen and then complaining about the specific soup they've decided to give you. "Oh, you mean this community resource we've set up which allows anyone to contribute a package and have it globally available doesn't provide exactly the functionality you'd like? Please tell me more about how the community can respond to your whims." Also, for what it's worth, all the Haskell libraries that I make frequent use of tend to have very good documentation (you could argue that's HOW they end up becoming the ones I make frequent use of).
- polskibus 12y agoHow does the functionality of Haxl differ from a mature ORM system? I'm thinking about .NET Entity Framework + LINQ in particular since it not only does the mapping but also assists in query generation, scheduling.
- mkesper 12y agoIt would be cool if you could transform those tables into HTML tables. They would look prettier (not nice JPG noise) and would also be more accessible.