3 ms·
I've heard from their engineers that Haskell has been their secret to being web scale while having a sane ops and engineering load. :-) Considering I'm doing m
by carterschonwald 13y ago
I've heard from their engineers that Haskell has been their secret to being web scale while having a sane ops and engineering load. :-)
Considering I'm doing my own Haskell based tech in my business (and I'm somewhat actively in the core community), It pleases me greatly to see Haskell/ghc used in a rich array of interesting businesses!
- mwotton 13y agoThis is true :) been a great tool to write a spider in, fast native code, a strict type system and very cheap threads has made it much faster than it otherwise would have been. ZeroMQ has also been a great help in modularising it. I'll try to get a piece out on the technical architecture later. (I'm the CTO, btw.)
- etherael 13y agoMy favourite bits of the project; Elasticsearch ZMQ Golang & Haskell Unbound DNS And a bit of old Python glue to hold everything together. It's funny how long things can last when you think you would have outgrown them pretty quickly. A tiny round robin domain server that maxes out at around 60k domains per second on just a single core to distributed spider bots though is just about easy with Python + ZMQ. Plenty to keep a swarm of far more complex and well engineered Haskell crawlers humming away and throwing the results to Golang indexers to push into the Elasticsearch core for ... Well, I'll leave the rest of it to mwotton's blog post, suffice to say it's been one of those life changingly awesome projects of doom. I'm sold on the benefits of embracing polyglot.
- mwotton 13y agoyes, I probably should have mentioned the other 90% of the project, hey :)