10 ms·
Fighting spam with Haskell
- seddona 11y agoThanks for the overview Simon, great to hear about the use of Haskell at scale. At CircuitHub we use Haskell to build our entire web app, Haskell is great for most tasks these days.
- Lewisham 11y agoDid you use a particular framework? I've been greatly enjoying my 4th attempt at learning Haskell (I think only now am I seasoned enough to really get it), and my general proof that I know something and can use it usefully is to develop a web app (as all apps are now anyways). However, I didn't find anything which looked like it had majority community buy-in unlike Rails/Sinatra/Flask et al.
- seddona 11y agoWe use Yesod
- wz1000 11y agoYesod is the most actively developed and popular web framework for Haskell.
- haxeny1 11y agoFor my side project, I use yesod/websockets for the server and haxe for the client. I feel ok with that choice instead using hpaste. Though eventually I would like to see a complete haskell stack. @Lewissham: about the fourth attempt, it just so happens that this was my third attempt at getting something that was right for me as well as maintaining the typesafety. I do realize that code is still not as seamless as I would like. It still lets me understand the overall roadmap that the code needs to take to be usable/maintainable etc.
- psibi 11y agoCan you expand upon what you mean by complete Haskell stack ? What do you feel are currently missing from Yesod ?
- haxeny1 11y agoYesod is awesome I can't see myself going back to less typesafe database schema tools. The choice I made seemed like a safer path, perhaps not ideal. The application that I am trying to put together is a realtime websockets application. Therefore, I am using yesod/websockets for the server. For the client, I chose haxe simply because of the targets available making it simpler to write plugins. On one of my irc chats or somewhere, I did hear good things about haste, though it is primarily a javascript library, therefore I plan to evaluate it later. About the client: it is a single page app to prevent any page refreshes and I am a bit sold on the frp (using promHx). I guess I should have elaborated a bit more about this in my earlier comment, because my choices are not indicative of anything amiss in yesod.
- e12e 11y agoHave you looked at Elm[1]? Is that "too much web" and too far from Haskell for you? https://github.com/elm-lang/elm-platform#elm-platform https://github.com/elm-lang/elm-platform#elm-platform
- haxeny1 11y agoElm was my first choice for a different side project. I went away because then elm did not have, or it was my lack of knowledge,support for crud widgets. I miss the elm signal approach in haxe for one thing. Here is the link to my repo so things are a bit concrete https://github.com/asm-products/ccar-websockets.git https://github.com/asm-products/ccar-websockets.git . Warning: this is still work in progress and there are way too many issues before I can even release this. The elm repo is here: https://github.com/dservgun/elm_projects https://github.com/dservgun/elm_projects . Edit: added some repos to set a context. The original comment about 4th attempt hit home as the latest repo is my 4th(I am going to quickly lose count) attempt to achieve at a model that might work/scale. There are still some issues that I would like to solve: whether to use code generators or not. Code gens will save me some typing, but I found myself tweaking printfs or equivalents to fit it into a narrow model. Edit II: Elm, iirc, is being supported by prezi that also happens to be using haxe quite actively. So I felt a bit more reassured about using haxe.
- psibi 11y agoAlso the nice thing about Yesod, is the freely available online book[0]. [0] http://www.yesodweb.com/book http://www.yesodweb.com/book
- reagency 11y agoHas the book caught up with the software? Last I checked, the book was written in terms of obsolete APIs that didnt match the yesod-geneterated app boilerplate, and Snoyman said that he didn't have time yet to update the book.
- bjterry 11y agoEven though it was at one point a physical book, I think, it's really a regularly updated documentation for the library. Sometimes it's not fully updated because it's written by one guy (Michael Snoyman, who also leads Yesod), but it always gets fully updated for the new version. It's not like a print book where you have to wait for the 2nd edition to come out.
- haxeny1 11y agoI have been using this site for (yesod 1.4) [http://www.yesodweb.com/book/introduction http://www.yesodweb.com/book/introduction] and I found the documentation current. Is there something that doesn't compile?
- psibi 11y agoI have the book print version and that certainly is outdated. But the online version isn't outdated (mostly). The current version of Yesod[0] is 1.4 and according to the book site it targets 1.4. That being said contributing to any of Michael Snoyman's (main yesod developer) project (including the book) is easy and fun. He is very responsive and friendly. :) [0]https://hackage.haskell.org/package/yesod https://hackage.haskell.org/package/yesod
- wyager 11y agoHappstack, Snap, and Yesod are popular. I'm partial to Happstack because it doesn't use any wonky extensions like template Haskell. It's just a very straightforward, very fast webserver. It's also very agnostic of how you actually render your HTML, whether it's static serving or dynamic generation with Blaze-builder or something.
- kyllo 11y agoTemplate Haskell and QuasiQuoters are optional in Yesod too. The project scaffold you get when you run `yesod init` uses TH and QQ by default, but you don't have to use the scaffold: http://www.yesodweb.com/blog/2012/10/yesod-pure http://www.yesodweb.com/blog/2012/10/yesod-pure
- nlake44 11y agoFacebook censors so much and swaps out links. I can't even post a link to https://scientificamerica.com https://scientificamerica.com. Their SSL cert is weak and should not be trusted. It is swapped out for phishing scams.
- TazeTSchnitzel 11y agoFacebook swaps out links to hide the referrer. It's for privacy.
- reagency 11y agoThat URL is a typosquatting domain-parking site, so....
- fokz 11y agoI am under the impression that a large part of engineering effort at established companies go into porting existing components to a deemed to be more appropriate language for that task. Is it plain impossible to pick the best fit language without implementing a solution in the first place and fleshing out the requirements and challenges that specific to the problem space? Or do the problems evolve fast enough that no matter how well you design the system, it will need to be deprecated once in a few years?
- liviu- 11y agoCould be simply that time uncovers better solutions to the same problems, which triggers the need of porting.
- taeric 11y agoThe problem with this is just how often the story seems to involve porting of solutions. Not necessarily refinement or advancement, but pure porting. With lots of debate/fighting on the syntactic properties of the solutions.
- Lewisham 11y agoBit of Column A, bit of Column B. All software tends to unmaintainability, even if you don't touch it (libraries change/deprecate etc). Given enough time, the problems that you are experiencing, say, memory allocation in C, get solved by something else, such as garbage collectors, and the amount of time it takes to solve the headache in what you've got is longer than just rewriting into the new language.
- lmm 11y agoIt's rarely "just" a port. It's usually because the new language has better characteristics. These days porting can be a very gradual, as-needed process - partly because of thrift, another Facebook tool! One of the big parts of becoming a professional for me was accepting that code has a lifecycle; code is written to make the business money at the time, but it's entirely normal for it to change and die as time goes on. (That said, you should just write everything in Haskell and then you won't have these problems. When was the last time you saw a company port code away from Haskell?)
- VeejayRampay 11y agoAnd two GHC bugs fixed along the way. Well done everyone.
- wz1000 11y agoHow does the hot-swapping work? The only way I had seen of making this happen is what xmonad does. I'm assuming this is radically different from that.
- mmarx 11y agoWhat xmonad does actually isn't hotswapping, but exec'ing the new code and simply passing the current state along. Instead they are using GHC's runtime linker to load/unload modules dynamically, in the same fashion as ghci does.
- agentgt 11y agoI was wondering that myself. Maybe they will release code examples on how that works. I'm only slightly familiar with the Haskell ecosystem but I have also wondered if it has a plugin like system (e.g. OSGI for Java).
- xnyhps 11y agoI've had some mild success with https://hackage.haskell.org/package/dynamic-loader https://hackage.haskell.org/package/dynamic-loader (which is basically just a Haskell wrapper around the C-functions in the RTS), but compared to writing normal Haskell, this is playing with fire as you loose all type-checking guarantees.
- deleted 11y ago[deleted]
- HugoDaniel 11y agoThis is an amazing effort, implementing ApplicativeDo and using Haxl for automatic batching and concurrency, doing code hot-swap in a compiled language, developing per-request automatic memoization, finding a aeson performance bug, translating C++ from/to haskell to do partial marshalling of data, implementing allocation limits to ghc threads, creating a shared c++ library to bundle the c++ dependencies in ghci for interactive coding, killing two ghc bugs, and more... and in the end producing a reliable scalable solution. ouch!
- spopejoy 11y agoIt's definitely not just a "We're really happy with Haskell here at FB" post. This probably explains the trollish comments elsewhere to the tune of "Haskell without Simon+bos /= scalable". It is a little intimidating, though. I think the perception is "Haskell is hard enough, AND I have to be a C++/GHC internals expert too?" The hard truth is you have to be this level of expert at something to achieve robustness at industrial scale. Simon's comment in this thread hints at some of the power of Haskell alone: The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.[1] This pretty much sums up appdev in Haskell. Engineer (or employ) amazing abstractions with no funny business (hello Spring!) and no loss in safety or expressiveness. I'm really happy this team exists, this is the kind of work that will take Haskell into the mainstream. Kudos! [1] https://news.ycombinator.com/item?id=9787523 https://news.ycombinator.com/item?id=9787523
- j_m_b 11y ago|We implemented automatic memoization of top-level computations using a source-to-source translator. This is particularly beneficial in our use-case where multiple policies can refer to the same shared value, and we want to compute it only once. Note, this is per-request memoization rather than global memoization, which lazy evaluation already provides. I would like to know more about this. What is a request exactly? An API call? If so, when an existing policy is changed, do the memoization tables have to change as well? How are the memoization tables shared? If this is running on a cluster, I would imagine that lookups in a memoization table could be a bottleneck to performance.
- simonmar 11y agoFor example, let's say that one of the things you want to compute is the number of friends of the current user. This value is used all over the codebase, but it only makes sense in the context of the current request (because every request has a different idea of "the current user"). So this is a memoized value, even though in the language it looks like a top-level expression. Memoization only stores results during a request. It starts empty at the beginning of the request and is discarded at the end, and it is not shared with any other requests. It's just a map that's passed around (inside the monad) during a request.
- j_m_b 11y agoThanks for the response. Just trying to expand my brain here =), so I have a followup question. I always thought of memoization as storing the parameters to, and result of, a function call in a memotable. Doing some quick research, I came across this definition of memoization from NIST that sounds more general "Save (memoize) a computed answer for possible later reuse, rather than recomputing the answer." What I understand from what you said is that when a request is processed, it produces a map that is passed around for the duration of the request. Something like: Request -> (some processes) -> memoized map -> Policy Filters How is the memoized map reused?
- simonmar 11y agoThe memo table (map) is a bit of state that is maintained throughout the request's lifetime. When we compute a memoized value, it is inserted into the map, and if we need the value again we can just grab it from the map instead of recomputing it. The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.
- dasmoth 11y agoDamn you FaceBook. I dislike the underlying premise, the adverts, and (especially) the "real names" policy. But... between great bits of Open Source like React, cool infrastructure projects like this, and a technical culture which seems a whole lot more open than many other big companies, it's getting kind-of hard to go on hating. Walk back a bit from the obsession with open plan offices, and I might just cave...
- r0naa 11y ago> But... between great bits of Open Source like React, cool infrastructure projects like this, and a technical culture which seems a whole lot more open than many other big companies, it's getting kind-of hard to go on hating. Walk back a bit from the obsession with open plan offices, and I might just cave... Wholeheartedly agree with you comment, Facebook's engineering blog is truly humbling. There is not a day where I do not try to think of something that would both make the Internet more decentralized, anonymous and secure while being extremely profitable. The latter is crucial because capitalism would make the change viral and being profitable would let me attract talent. In short, make something that people want AND that promotes values I believe in.
- MichaelGG 11y agoFacebook could easily promote more privacy and anonymity while remaining profitable. It could have been a non-profit and aimed to make money without compromising privacy. It wouldn't be as profitable but that's a different story.
- afarrell 11y agoI would be genuinely interested in knowing how they could have scaled organizationally without money from investors who would have pushed them to make a profit.
- Bahamut 11y agoTheir commitment to open source is truly amazing - I am currently in the process of interviewing with them, and am increasingly in awe of the company. Surprisingly, their massive open office is the quietest office I have been in. It is a really nice workplace.
- vezzy-fnord 11y agoLoading and unloading code currently uses GHC's built-in runtime linker, although in principle, we could use the system dynamic linker. That they could use the system dynamic linker makes me think they're using some form of relatively basic dlopen/dlsym/call-method procedure, or something along those lines. That's fine, though the use of "hotswapping" evokes the image of some more elaborate DSU mechanism.
- LukeHoersten 11y agoFor those not involved in the Haskell community, Simon Marlow worked full time on the GHC compiler and specifically run-time system for many years. Along with Simon Peyton-Jones, he's huge in the Haskell world. Marlow also wrote the very excellent "Parallel and Concurrent Programming in Haskell" book. Facebook also employs Bryan O'Sullivan, an epic Haskell library writer (Aeson, Attoparsec, Text, Vector, and on and on http://hackage.haskell.org/user/BryanOSullivan http://hackage.haskell.org/user/BryanOSullivan). Bryan also co-authored the "Real World Haskell" book. So Facebook has hired two prolific Haskellers and probably others I don't know about.
- LukeHoersten 11y agoA side note: The Haskell runtime is written in C so Simon Marlow is also one of the best low level C hackers I've ever met.
- reagency 11y agoMarlow also famously says that the doesnt understand the zygohistomorphic prepromorphisms in Haskell.
- zmanian 11y agoBos manages the internal tooling team at Facebook. It is little surprise that Facebook's internal tooling is acquiring a functional programming flavor.
- LukeHoersten 11y agoSpeaking of functional/Haskell network tooling, this is a really cool project: http://www.pfq.io http://www.pfq.io
- Periodic 11y agoIt really shows what an investment they are willing to make in the language. They aren't just training up internal engineers, they're bringing in some of the best in the community. I'm hugely excited that Facebook is making this investment and giving some of the developments back to the community. There are many smaller companies that would be interested in Haskell but don't have the resources or expertise to tackle some of these complicated problems. Facebook is blazing the trail for production Haskell and the rest of us can follow.
- covi 11y agoIn the throughput graph, why does Haxl perform worse in the 3 most common request types?
- Periodic 11y agoI don't think the graph says which request types are more or less common. The ordering could be arbitrary. It would be interesting to know why some are slower. Perhaps they require very little processing and so the time becomes dominated by FFI transformations? It would be nice to know!
- Filligree 11y agoIn fact, it appears to be ordered by performance. There's no other reason I can think of for that appearance.
- JonCoens 11y agoOne of the cases we found while performance profiling was a tradeoff between memory usage and computation completion. On certain requests FXL would use too much memory and be halted by the equivalent to AllocationLimits, while Haxl would happily plow on using less memory and complete the request. When looking at many of those requests in aggregate, the end result would have more successfully completed requests but with longer response times mixed in. Completing more requests was seen as a win over the apparent decrease in throughput.
- simonmar 11y agoThe graph is sorted by performance, with the worst performing (not necessarily the most common) on the left. We've also done more profiling and optimisation since we took those measurements. FXL employed some tricks that were sometimes beneficial, but often weren't - for example it memoized much more aggressively than we do in Haskell. Mostly that's a loss, but just occasionally it's a win. When a profile shows up one of these cases, we can squash it by fixing the original code. What matters most is overall throughput for the typical workload, and we win comfortably there.
- edwinnathaniel 11y agoI learned a few things from this post outside the usual "technical" explanations: 1. They have CORE Haskell contributors on their payroll to deliver this type of project (what this mean is that no... Haskell isn't any better than other language, it's just that they have people who know Haskell very very deep to the compiler level...) 2. In-house custom language eventually does not scale (the EOL is much much much shorter than other programming languages), plan for that :)
- tome 11y agoAnd Google and Dropbox hired Guido van Rossum ...
- Scarbutt 11y agoWhat you should ask yourself then is, why did they hire core Haskell contributors?
- edwinnathaniel 11y agoPerhaps to avoid the same situation with FB Messenger Erlang to C++ conversion? What do you think?
- outworlder 11y agoLet's see. because they can afford to? Also perhaps because they are Facebook, not your average company? Who should they hire instead? Given the opportunity and the means, wouldn't you hire someone of that caliber for your preferred language/environment?
- dasil003 11y ago> (what this mean is that no... Haskell isn't any better than other language, it's just that they have people who know Haskell very very deep to the compiler level...) This is extremely presumptuous for a parenthetical. If this is what you believe it merits a comment all to itself, because I don't see how this follows in the least. To rephrase what you're claiming: if a language has a flaw than it can not be any better than any other language.
- 11y ago
- giancarlostoro 11y agoIt seems like Facebook seems to be the one company developing a lot of the very interesting and useful tools for developers.
- pekk 11y agoThat is unnecessarily negative toward the efforts of literally everyone else.
- giancarlostoro 11y agoI wrote it poorly my apologies. I guess I could see how my messages is misinterpreted.
- reagency 11y agoThis is impressive, and in line with Haskell's philosophy to "avoid success at all costs". As a mere mortal programmer, who knows a little Haskell, my takeaway is that if you want to run Haskell in at web scale for a a large userbase, you need the language's primarily compiler author to help build the application and to modify the Haskell compiler to make it performant. And you also need your team led by a 20-year-veteran Haskell expert who is one of the language's handful of luminaries whonwrotr a plurality of its main libraries. What are the rest of us to do, who aren't at Facebook?
- e12e 11y ago> web scale for a a large userbase Yes, it might take some effort to scale things across a 1.4 billion users. Now, in what language wouldn't that be a challenge? I suppose if you're one of the "many" other companies that are approaching 2 billion active users/month, you might need to put some thought into your systems too...
- DasIch 11y agoThis is an incredibly stupid takeaway. Facebook can hire the best software engineers and computer scientists in the world, so of course they hire such people and these people build stuff at Facebook. That doesn't mean other people can't use Haskell to build web applications.
- zak_mc_kracken 11y ago> That doesn't mean other people can't use Haskell to build web applications. Very few companies do, though, so OP's point still stands. This article would be a much better ad for Haskell if it weren't written by one of GHC's main author. The article also casts doubt on whether Haskell was picked because it's the best tool for the job or because of the team's familiarity with it.
- PaulAJ 11y agoYou don't hire Simon Marlow to hack PHP.
- saosebastiao 11y agoAnybody know what sort of policy resolution algorithms are used? Is this based on Rete, or home grown?
- mark_l_watson 11y agoI don't know if Rete is appropriate. Rete optimizes for a large number of rules and a small amount of data. I used to hack Rete based OPS5 a lot, including adding support for multiple data worlds. If you like Common Lisp, then the OPS5 code base is fun to hack.
- yellowapple 11y ago> Fighting spam > (facebook.com) Ha.
- deleted 11y ago[deleted]
- themeekforgotpw 11y agoDoes this system also detect propaganda operations?
- jjawssd 11y agoNo, it generates them /sarcasm
- themeekforgotpw 11y agoThe question was serious. Malicious actions could certainly include propaganda operations and indeed there has been a growing investment across social networks to block content specific to certain ideas considered dangerous, disinformation and misinformation. My guess is actually that, yes, this system has rules (and gets live feeds of them) to block certain political content associated with influence operations. But I was hoping to get a more serious, direct and charitable answer from someone who works on the system.
- jjawssd 11y agoI'm also waiting for more commentary to this original question. I would wager that no it is not possible to detect propaganda effectively. Effective propaganda notably presents only one side to the story and uses various other tactics to send that message home but detecting one-sided but factual statements is impossible with today's natural language processing tools, as they struggle enough with being able to make sense of anything. On another note, what happens if Facebook is able to detect "propaganda content" but it only silences propaganda it doesn't like? This is what I originally meant. It's similar to how propaganda works in US media. It's not the news stories themselves, it's all about the news networks which are told from above which stories to run. I can see Facebook operating in a similar fashion today or in the future.
- themeekforgotpw 11y agoDefinitely not all strategic communications could be detected and blocked. However some research has been tasked on the issue. Take "Antisocial Behavior in Online Discussion Communities" (http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf). This paper, though it terms it 'antisocial', studies online influence operations and is mentioned on NATO's propaganda ("strategic communication") centre for excellence Facebook page (https://www.facebook.com/StratComCOE https://www.facebook.com/StratComCOE). The point of having a rule set that can be pushed to like the system described here is that it can be fed information from a centralized task force. Indeed, these systems are used across industries with patterns and rules to detect malware, hacking, and a host of other activity (using formats such as STIX or TAXI). It's unlikely that Facebook itself would be trying to determine whether a post to some content is propaganda - it's more likely that their automated system will apply the rules it receives from information defense agencies. As per your other note, the selective censorship of information is absolutely a form a propaganda of its own. Facebook (and the US government) don't have the best history here and have 100% certifiably censored information to prevent effective organization of (for example) May Day protests. I'm not going to go speak about that at length here - hoping to get some technical clarity from those involved.