26 ms·

How we secretly introduced Haskell and got away with it

313 points by philippelh 10y ago

sid-kap 10y ago
The comparison between Stack and Python build tools is striking: > No messing around with virtualenvs and requirement files. Everybody builds with the same dependency versions. No more “works on my machine” and “did you install the latest requirements?”. I wonder why the Python ecosystem, which is much more mature, doesn't provide a build tool as delightful as Stack (which is less than 2 years old).
- Lev1a 10y ago
  The whole "Do you have the dependencies and a Python env installed? Noß Then you can't run this script/program." was one of the main reasons I switched from Python to Rust, where cargo as the (very good) package manager comes with the language and, because Rust is a compiled language, you build all the dependencies into your executable you aren't dependent(heh.) on the user having installed a runtime that maybe or maybe not has all the dependencies at the required versions.
  - nimish 10y ago
    I have pygradle generate a PEX file for all of my command scripts. Not as seamless as cargo or stack, but it works without rewriting everything.
    - Lev1a 10y ago
      I should probably have clarified that I wanted the "seamless" way of making the final executable. IIRC, I tried(really hard) and failed on getting some method of "freezing" for Python to work, which made me weary of the prospect of trying something like that in Python in the future(now past).
      - lima 10y ago
        pyinstaller works great. No issues whatsoever and I'm happily deploying Python 3.6 to machines as far back as RHEL 5. Single binary to deploy, no dependencies except libc, life is great.
  - scriptkiddy 10y ago
    I guess this is true if you don't need to interact with any system libraries.
    - Lev1a 10y ago
      Could you give an example of what "system libraries" would pose a problem in my example of using Rust with cargo? Also note that by "no runtime installed" I mean no runtime as in "no Python runtime", "no JVM" etc. not necessarily "no libc" EDIT: formatting
      - scriptkiddy 10y ago
        >Could you give an example of what "system libraries" would pose a problem in my example of using Rust with cargo? Specifically things like xorg libs, libmpeg, libsdl, and such. Not that Rust would have a problem interfacing with them, just that they would need to be present regardless of whether or not someone was just trying to run a distributed binary. Agreed that you wouldn't need a VM like CPython or the JVM. However, Rust isn't unique in that department. Almost all languages that compile to binary executables have this advantage.
        Lev1a 10y ago
        > Specifically things like xorg libs, libmpeg, libsdl, and such. Not that Rust would have a problem interfacing with them, just that they would need to be present regardless of whether or not someone was just trying to run a distributed binary. That's why stuff like that is AFAIK usually either distributed with the binary or is absolutely required to have present on the system, regardless of the PL, if you want to/can only distribute a "naked" binary. > Agreed that you wouldn't need a VM like CPython or the JVM. However, Rust isn't unique in that department. Almost all languages that compile to binary executables have this advantage. Didn't mean to suggest this is unique to Rust, which is why I wrote > because Rust is a compiled language. EDIT: formatting
        scriptkiddy 10y ago
  - Ruud-v-A 10y ago
    Indeed, Rust + Cargo and Haskell + Stack are very similar in this regard. Both have great package managers, and both produce a shippable executable with only a few dependencies on system libraries. One notable difference is that Stack downloads the compiler, whereas for Rust, every version of the language comes with a compiler and a Cargo. This ensures that you can check out a year-old commit and still build your project with Stack (modulo breaking changes in Stack, which so far I have not encountered), whereas for Rust the compiler version is not pinned.
    - twic 10y ago
      > for Rust the compiler version is not pinned Oh, you just need another layer of abstraction! Install rustup, and then (from memory, might be slightly wrong): rustup install 1.15.1 rustup run 1.15.1 cargo build rustup will take care of getting hold of the right versions of cargo and rustc, and then use them to run the build. I admit that it's not as nice as having the build tool download the right version of everything, but it does work, and you could hide this inside a pretty small shell script or function if you wanted it to be neater.
- chowells 10y ago
  Probably for the same reason I greatly prefer cabal to stack. Stack assumes it knows better than me. Cabal just does what I tell it to do. As a domain expert, I greatly prefer the latter. It does what I want, nothing more, nothing less. Stack is a mysterious "solution" to a problem I don't have that works by doing everything differently than I do. Stack was created because not everyone is a domain expert. A lot of people don't want to be domain experts. They just want something that works without having to know all the details. It was only able (in the business sense) to be created because so many people look at Haskell skeptically anyway, and take any excuse to back away from it. The people behind the development of stack also run a major advocacy initiative trying to get people to use Haskell, so they found it to be an important thing to build. You don't need to try to get people to use Python. It's already broadly accepted. When people run into trouble, they just say it's the price of using Python, and aren't willing to make the exchange of giving up power to get rid of a minor inconvenience. So there's no business incentive in the Python ecosystem for making the tradeoffs stack does in the Haskell ecosystem.
  - baldfat 10y ago
    Cabal made me never use Haskell every again. I work in two different locations and at home. All three locations never worked the same and all had different issues with Cabal. After hours and hours of trying different things I walked away into the wonderland of Racket.
    - arianvanp 10y ago
      They're working really hard on improving it though . Cabal 2.0 will have a nix-style build system, in which multiple verions of the same dependency can be installed globally (so no separate sandbox per project). This will solve most problems of where cabal breaks down. This gives us almost the same usefulness as Stack. However, you will have to make sure that there is actually a feasible build plan, by setting up your version bounds correctly. With stack, other people take care of this for you, and you never touch the version bounds, which is relaxing but also gives you less control.
      - 10y ago
- quantumhobbit 10y ago
  I think newer often tends to easier because it has the benefit of hindsight in this case. Also the old tools tend to get more complicated as time goes on.
- chongli 10y ago
  I wonder why the Python ecosystem, which is much more mature I hope I'm not being too pedantic but Python's ecosystem is much larger than Haskell's, it isn't really more mature. Haskell and Python are very similar in age as languages go.
  - vkou 10y ago
    Maturity comes from: * Millions of person-hours being poured into a language... * ...Over a long enough time period that the language can go through several develop-eval-improve cycles - that take real world use cases (And not one-liner bubble sort implementations) into account. In this sense, it doesn't matter whether or not Haskell was invented in 1890, or 1990. #2 is required for maturity, but so is #1. (I am not a huge Python fan.)
    - ska 10y ago
      It is important to note that person-hours are not remotely fungible, and some contributions are even negative in the sense of "developing ecosystem maturity".
    - chongli 10y ago
      You can't get 9 women together and produce a baby in 1 month. A lot of developments within a programming language ecosystem originate from new ideas discovered outside of it. It doesn't matter how many people you have working on a project if the crucial piece of tech they need hasn't been discovered yet.
      - vkou 10y ago
        No, but you also can't study 1 woman having 9 babies, and conclude yourself an expert on pregnancy. I'll prefer to get my advice from the doctor who studied 9 women, having one baby each. Note what I said about a mature language having to go through several write-eval-improve cycles. Diversity matters. A language that one person tinkered on for thirty years is far less likely to be useful, then one that ten people tinkered on for three years. Or, in the case of Python vs Haskell, a hundred people who tinkered on it for twenty-five years.
- chrisdone 10y ago
  Stack requires package sets (aka "snapshots"), which some kind of CI system (Stackage: http://stackage.org/ http://stackage.org/) has to do a daily build job to see if they all build and pass tests together. That requires some money to keep running, and buy-in from package authors as there is a maintenance overhead each release. It took a few years for Stackage to get enough packages in it to be generally useful, and then we wrote the Stack tool which was able to default to using Stackage snapshots. There was (and still is a little bit) of resistance to the whole idea of Stackage from the community; people liked the idea of build plans magically being figured out on demand, it's an interesting research problem (it can be hard to let go of an interesting problem when a solution side-steps it all together). I believe eventually many people changed their minds after experiencing the simplicity of using Stack and not having build hell be a substantial part of their development cycle. Python would likely have to go through the same process. Although with Stack and Yarn providing frozen builds (and QuickLisp); the idea has some precedence, which makes it an easier idea to sell. I mean, Debian and a bunch of other OSes do it like this, but experience shows programmers don't pay attention to that.
  - runeks 10y ago
    Stack enables application development in Haskell, as opposed to just library development. A proper library doesn't have more than 20-ish dependencies, in my opinion, and manually handling these and their version bounds is not a problem. But when writing applications with hundreds of dependencies, manually figuring out a mutually compatible dependency range for all packages just isn't an option. At least not if you want to spend time prototyping code, rather than think about dependency ranges. hpack solves additional problems with the .cabal format (sane defaults as opposed to build failure), and I highly recommend it, for application development at least. I just discovered it a month ago and now I wouldn't be able to live without it.
guptaneil 10y ago
It looks like jobmachine is a private repo, maybe don't include the url in your blog post :)
- rkrzr 10y ago
  I don't think there is any harm done in putting it in there. All GitHub urls follow a standard schema, so if you wanted to you could anyway guess it. The point of the code snippet was simply to highlight how nice the Haskell build tools are compared to Python's.
  - Operyl 10y ago
    I couldn't help but try to see if they were nice enough to opensource it, though, hah.
    - rkrzr 10y ago
      It's currently specialized for our specific use case and would not be useful for anyone else, I think. I'm sure that we will keep iterating on it though, so perhaps it will become more general-purpose in the future so we might look into open-sourcing it then. But I can't promise anything, sorry!
- yawaramin 10y ago
  If the URL to a private repo is not secure, then a whole bunch of people (including GitHub) have a big problem, and exposing jobmachine is the least of their worries ;-)
  - guptaneil 10y ago
    It's not a security problem, or else my comment would have been much more adamant about removing the link. It's mostly just reader confusion. Seeing the git clone url led me to believe the project was open sourced and was disappointed to find that not to be the case.
    - 10y ago
mybrid 10y ago
In the 1990s I did research on the efficacy claims of object oriented programming versus procedural programming. This article bares a striking resemblance in the claims. Case study after case study showed that object oriented code had less bugs due to compilers catching bugs, etc. However, almost every study was similar to this report: it was a re-write from procedural to object-oriented. There exists strong evidence to suggest that throwing away ones first prototype produces the same results as to benefits comparing object-oriented versus procedural programming. The conclusion of my research was the same as others: unless one is comparing the same project written from scratch without any sharing of design or code then these studies claims are correlation, not causation.
- dsacco 10y ago
  Do you have anything published? I'd be very interested in reading.
  - mybrid 10y ago
    Sorry, this was course work for a graduate class.
- NegatioN 10y ago
  Wouldn't a complete rewrite of a module usually be a lot easier if the program is procedural? I think that is one of the big factors stopping that from actually happening with OO code. A rewrite is a lot harder if you tap into what exists elsewhere in the codebase. I bet you have a lot more insight into this than me though.
  - kbenson 10y ago
    I'm not so sure about that. I think a complete rewrite is easier if you correctly separated concerns and compartmentalized needs. OOP is supposed to enforce/encourage that, but it's fairly easy to not treat your objects as black boxes with APIs, and then you've put constraints on replacing a component. Procedural code doesn't necessarily tout it's ability to encourage that, but well planned and implemented functions can give you the same benefit. In the end, it's all up the the programmer.
osi 10y ago
makes me wonder how long they would have lasted had the initial implementation been done simpler (ie, not in their PyDatalog fork).
eternalban 10y ago
> Our lead developer Robert usually comes in a bit later, so we had about an hour to build a working prototype. I'm guessing he is a Python developer and likely he is no longer the lead.
- rkrzr 10y ago
  Don't worry, I'm still here.
  - deleted 10y ago
    [deleted]
  - KurtMueller 10y ago
    Yes, but are you still the lead? :)
    - rkrzr 10y ago
      I'm still the lead.
      - steelbird 10y ago
        0_0
        slezakattack 10y ago
        Well, he's the co-founder of Channable so... :P
- bbcbasic 10y ago
  What's the reasoning?
arianvanp 10y ago
I'm the author of the post. I'll be happy to answer any of your questions :)
- fajpunk 10y ago
  Thanks for the write up! In the beginning you mention that you ran into some bugs in the Python version that would have been caught by the Haskell type checker. Can you go into more detail about what those bugs were?
  - nerdponx 10y ago
    I'm wondering if integrating MyPy into the build, or even using Cython for performance, could have helped.
    - Ruud-v-A 10y ago
      We do actually use Mypy! I wrote about my experience with it here: https://www.reddit.com/r/programming/comments/5x3hdd/channable_how_we_secretly_introduced_haskell_and/degynke/ https://www.reddit.com/r/programming/comments/5x3hdd/channab...
  - Ruud-v-A 10y ago
    The most serious one is that we were submitting jobs (as json) that were missing a few metadata fields. In Python we passed around dictionaries, and even though we had json schema validation in place, this slipped through. In Haskell, we define a record type and the corresponding serializers. It is more code, but what you get is that invalid data cannot exist at runtime: it simply cannot be represented. Also, a compiler refuses to compile your code if you make a typo in a field name.
- BuuQu9hu 10y ago
  Did you try PyPy? If so, how did it perform? If not, why not? PyPy adoption is at only 1% or so, and it'd be nice to see more PyPy usage.
contravariant 10y ago
> The issue here is that we cannot run runWorkerLoop and runFetchLoop with forkIO, because we don’t have the right transformer stack. Am I understanding correctly that this is because, while you can lift e.g. runFetchLoop to something of type IO m (), it's not possible to convert use forkIO on it since it requires an input of type IO ()? Isn't that just a consequence of the fact that Haskell has no possible way of knowing if your side effects can be contained in the IO monad?
- dllthomas 10y ago
  That sounds about right. At the shallowest level, we can't pass `m ()` to forkIO unless m ~ IO, 'cause the types don't match. But beyond that, there is the question of how that extra context would be passed through. For something like ReaderT this is straightforward. But consider StateT - `set` in one thread can't be visible in the other.
- chowells 10y ago
  It's not about side effects, it's about bookkeeping. When you have a type that indicates that you can do IO, but you're also carrying around bookkeeping data implicitly for things like configurations and data sources, forkIO represents a hard problem. If the implicit configuration is updated, there's no way to communicate that across threads. The same is true with all the other things monadic layering can provide. How do you call a continuation that points to a different thread? That doesn't even make sense. So.. Why lie in your type and pretend that those things all make sense? Why not make the type explicit about what makes sense and what doesn't? That way, when someone wants to do something that has no a priori way of making sense, they're required to define how to handle it, such that it makes sense in their specific use case. And that's what the post says they did. All in all, it's things working as designed. Places where you need to stop and think are set up such that you need to stop and think to use them, instead of barging ahead unaware of the issues.
daxfohl 10y ago
I'm just starting with Haskell and PureScript. So far I'm liking the latter better. It solves a few of their gripes with respect to strings, laziness and records, plus has a more granular/extendable effects system and cleans up the standard typeclass hierarchy. Also `head []` doesn't crash. Of course Haskell is more mature, has support for multithreading and STM, compiles to native, so it's more performant. But PureScript integrates with JS libraries and seems "fast enough" in many cases. I think it's more interesting as a project too: the lack of runtime and laziness means the compiler code is orders of magnitude simpler than Haskell's, so I could see a larger community building around it if it catches on. Given that they were on Python earlier, I wonder if PureScript would have been a better choice.
- chowells 10y ago
  I can't imagine using a haskell-like language without laziness. It's what makes it possible to actually write small reusable functions. Tell me, have you ever used foldr in Purescript? It just doesn't lead to reusable logic there, so I have no idea why you would. But in Haskell, foldr is used everywhere. Laziness means that logic built with it is actually reusable.
  - arianvanp 10y ago
    Folds are used in purescript all the time! Sure a foldr is less useful because it doesn't have the nice lazy preserving properties, but you can have strict left folds, which are tail recursive, and thus run in constant space. Elm, another haskell-like, _strict_ language, models entire applications around the strict left fold over events https://guide.elm-lang.org/architecture/ https://guide.elm-lang.org/architecture/ We actually do the same in Jobmachine. The application is driven by a strict left fold over incoming events and current state.
    - chowells 10y ago
      There's a reason I said foldr. In Haskell, even foldl is implemented with foldr. It's that powerful of a tool. In contrast, foldl is far weaker, and necessarily strict in its input. It loses nothing moving to mandatory strictness. But foldr loses everything.
yawaramin 10y ago
Nice article. Out of curiosity, what does your write with? Vim/ghc-mod? Emacs/Intero? IntelliJ IDEA/plugins? Atom/VSCode etc.?
- arianvanp 10y ago
  I tried spacemacs once but I found it too magic. I decided I should learn proper emacs instead of running a random playbox of plugins ontop of plugin. But haven't had the time to properly learn emacs yet. I have tried various haskell plugins for vim in the past, but they always tended to break so I gave up fixing my config and threw them all away. Now it's just plain vim (with some non-haskell related plugins) Next to it I have a terminal that reruns tests when a file changes : `stack test --file-watch` . It's simple but it always works. I'm not sure if the vim stuff got any better lately, I haven't checked. So if you have any suggestions, please tell :)
  - sadgit 10y ago
    Have you tried running tests from stack repl --test? Its so much faster. Id love if i could use file-watch and the repl together.
  - sjakobi 10y ago
    Have you tried ghcid? In my experience it was a bit quicker than `stack --file-watch`
Twirrim 10y ago
While it's an interesting look at a change you introduced, that blog title might not come across quite as intended. If you're having to "secretly introduce" tech, and "get away with it", that suggests there are unnecessary and unproductive constraints on your work; maybe even suggesting that you'd get in trouble for actually daring to make things better.
- _jal 10y ago
  Not the author, but I think that is exactly what they wanted to convey in the title. The implication is that they're fighting the man and won. There's a tradition of programmers laying claim to subversively Making Things Better in spite of the bean counters. Sometimes, it is even true, as far as it goes.
- arianvanp 10y ago
  The "secretly" part refers to part of the story where we had 1 hour to build up quick prototype to pitch to our boss. The title is a bit tongue-in-cheek. It's not that we built this thing in secret for months and then just deployed it. That's also not what the article says :) We had been planning on replacing Scheduler for a while now, and had already written down some mumblings about what the new design should look like. We were also already discussing whether we would switch away from python back then. I think the exact opposite of what you are saying is true. We got the freedom to experiment with something new, and to actually make things better along the way.
  - Twirrim 10y ago
    >I think the exact opposite of what you are saying is true. We got the freedom to experiment with something new, and to actually make things better along the way. Sure, and from what I read I mostly took it that way. My original point was just that maybe a bit of caution would be good in the choice of title. If I was just skimming through the titles on HN, or skimming the article, it could be easy to get the wrong impression of channable.
slezakattack 10y ago
"It is said that Haskell programmers are a rare species, but actually the majority of developers at Channable had used Haskell before." Could you imagine if this wasn't the case? The hurdle to actually get people excited about a language such as Haskell especially moving from something like Python would potentially be huge. Kudos for already having that problem solved.
- arianvanp 10y ago
  We're based in Utrect, where Haskell is part of the mandatory curriculum at and the language of choice for many master courses. Because of this, almost all our developers are somewhat familiar with it, or have at least had one course in it in the past, which really helps a lot!
- nanxor 10y ago
  In many European countries there are workplaces that only hire people with relevant academic degrees. A course on functional+logic programming is often placed in the 2nd or 3rd year of a typical European 3-3½ year CS degree.
deleted 10y ago
[deleted]
davedx 10y ago
Good luck with hiring!
- dualogy 10y ago
  Shouldn't be hard.. Haskell to me so far feels like an ecosystem with way fewer jobs / freelance gigs than there are eager-to-go-commercial enthusiasts hacking away in their spare time..
  - daxfohl 10y ago
    That is confirmed here: https://stackoverflow.blog/2017/02/07/what-programming-languages-weekends/ https://stackoverflow.blog/2017/02/07/what-programming-langu...
- pka 10y ago
  They won't need luck, plenty of people would love to program in Haskell professionally.
- sjakobi 10y ago
  See my reply to a similar comment: https://news.ycombinator.com/item?id=13786992 https://news.ycombinator.com/item?id=13786992
techman9 10y ago
> No messing around with virtualenvs and requirement files. Everybody builds with the same dependency versions. No more “works on my machine” and “did you install the latest requirements?”. While this is nice, of course, I'm not sure that is outcome is unique to Haskell/Stack. It seems like you could accomplish a similar level of reproducibility by building a Docker image or bundling dependencies in some other way.
- seagreen 10y ago
  My understanding is that Docker only stays reproducible if after every change you kill the container and start a new one. Otherwise a particular change may only be working because of a side-effect of a change you introduced earlier and then deleted. This isn't a huge issue, but still it's nice in declarative systems like Stack and NixOS not to have to worry about that kind of thing.
- Ruud-v-A 10y ago
  We are actually using Docker for generating the virtualenv that we ship and running tests now. The motivation for doing this is being able to control the environment; we can run tests and build a package on CI, and we can build the same package locally when CI is down. We don’t use Docker in production. It is not clear to me how Docker solves the issue of pinning dependencies; I would rather have a file that states the exact version of every package to install, than an opaque blessed container image that has some versions installed, and I do want to have the versions used under source control. Generating the image would not be reproducible (in the sense of having the same Python files inside it) without pinning versions somewhere anyway, right? Or am I missing something obvious?
xyzzy4 10y ago
Haskell is a bad language, in my opinion, because you can't tell what the O(n) run-time is for any operation. Instead you just have to "trust" that it'll be fast enough. More on this: https://www.reddit.com/r/haskell/comments/1f48dc/what_does_the_haskell_runtime_look_like/ https://www.reddit.com/r/haskell/comments/1f48dc/what_does_t... All of the answers seem insufficient. Basically you can't estimate Haskell run-time unless you are very familiar with the internal Haskell engine.
- willtim 10y ago
  The complexity of many Haskell collection APIs is actually documented unlike with many other languages. These complexities are not changed by the runtime. Only constant factors and memory use are affected by optimisations and this is no different to any other high-level language that actually optimises code.
  - deleted 10y ago
    [deleted]
- slezakattack 10y ago
  Completely untrue. I'm not a Haskell evangelist (I appreciate it for what it is) but I thought I would at the very least point out that most of the documentation for basic data structures (i.e. Data.List[1]) not only are well-documented but have a link on the far-right side of the documentation site that directly shows you the source code and it's usually easy to tell what it's doing. Any developer should be able to grok that code and determine the run-time complexity. [1] https://hackage.haskell.org/package/base-4.9.1.0/docs/Data-List.html https://hackage.haskell.org/package/base-4.9.1.0/docs/Data-L...
  - seagreen 10y ago
    I think he's talking about how it can be hard to tell what's already been evaluated and what hasn't.
  - astrobe_ 10y ago
    I'm not a C evangelist, but I would point out that most documentation of standard C function is very detailed and its code is shown in its man page. Any developer should be able to grok that code and determine its safety. Mattaku...
gaius 10y ago
At one company I experimentally wrote OCaml and named the resulting native binaries whatever.py. None ever looked at them. So there is alot of scope for shenanigans.
lima 10y ago
You can deploy Python code as a static binary that includes the interpreter along with all dependencies. I heavily use this in production and life is great - deployment means copying one single binary, reverting means running an older one instead. No external dependencies, no pip upgrades, just libc. https://github.com/pyinstaller/pyinstaller https://github.com/pyinstaller/pyinstaller
kisstheblade 10y ago
Always when I see haskell demonstrations eveyrthing looks like just interface declarations. You can do beautiful interfaces with eg. java also. But where is the meat where anything actually happens? I rarely see that in these posts. Yes I could look up the source but I don't have time to read through it randomly. This looks just so nice and stuff just magically works?: runWorkerLoop :: (MonadBackingStore m, MonadLogger m, MonadIO m) => WorkerState -> m WorkerState And monads to boot! (are monads haskells equivalent of java factories? I kid, I kid :)
- Ruud-v-A 10y ago
  The runWorkerLoop function logs a few lines and sends out an initial job request (by enqueueing an event in Redis). It then calls the nested function `go`, which dequeues one event of a TBQueue (a thread-safe bounded queue), matches on the event, and calls the right function to handle it. If the event was not a "stop" event, `go` calls itself to do the next iteration of the loop. `go` takes a WorkerState as argument, which is how it keeps track of which jobs are running, and whether there is an unanswered job request. In reality the signature is a bit uglier, I simplified it for the post because the point was about effects. In particular we also pass in the configuration, Redis connection details, and a callback to manipulate the TBQueue.
framp 10y ago
Glad to see Haskell used in production. It's kind of funny that build reproducibility (which was a major issue before stack) is one of the strong point. I wonder if, for your project, using cloudhaskell would have been more appropriate. I have a feeling some of the problems you found could have been solved with that.
javajosh 10y ago
In case anyone was wondering, the 'stack' command in the article refers to https://docs.haskellstack.org/en/stable/README/ https://docs.haskellstack.org/en/stable/README/. Which actually looks kind of wonderful.
hota_mazi 10y ago
> but if we could get it done, there would be no going back The naïveté in this simple statement is so cute. The list of concerns is also pretty naïve. The main problem you are going to encounter with this project is hiring. If you want to grow this project or if the main developers leave the company, I bet it will get rewritten in a different language in no time.
- sjakobi 10y ago
  See https://news.ycombinator.com/item?id=13784085 https://news.ycombinator.com/item?id=13784085 for why hiring is unlikely to be a problem for them. I also encourage you to find _any_ experience report that tells of difficulties finding the right candidates for a Haskell job.
bykovich 10y ago
Motion for an indefinite moratorium on articles masturbating about Haskell.
vorotato 10y ago
I'd use haskell if it weren't lazy by default and didn't use Cabal. Aka I'd use OCaml/F#.
ctlaltdefeat 10y ago
There are various mature options for scheduling jobs with dependencies between them. Why did you choose to write your own, regardless of the language?
- Ruud-v-A 10y ago
  Good question! None of the existing options that we investigated supported all of our requirements. In particular, all the arrows that can exist in the dependency graph are known ahead of time, but the nodes are not. This means that a job can depend on a job that does not exist yet. (A user can add extra feeds to download, and the merge job should wait for them, even if it had been submitted already.) Furthermore we have a few specific constraints such as “per project, only one of job type x or y may be running at the same time”.
jwatte 10y ago
Instead of using separate monad transformers, we use a single "World" that knows how to provide Redis, logging, iOS, and other typeclass instances. There is a RealWorld that runs on top of IO and a FakeWorld that runs on top of pure State for unit testing. This means that we have to wrap every single API into our own "SupportsRedis" and similar APIs, but in the end I think it's worth it! Unit tests are super fast and not intermittent at all.
mehaveaccount 10y ago
Actually if you wanted a statically typed compiled functional language of Haskell while keeping the declarative logic paradigm of Prolog, then the Mercury programming language was made exactly for you!

313 points by philippelh 10y ago

The comparison between Stack and Python build tools is striking: > No messing around with virtualenvs and requirement files. Everybody builds with the same dependency versions. No more “works on my machine” and “did you install the latest requirements?”. I wonder why the Python ecosystem, which is much more mature, doesn't provide a build tool as delightful as Stack (which is less than 2 years old).

Lev1a 10y ago
The whole "Do you have the dependencies and a Python env installed? Noß Then you can't run this script/program." was one of the main reasons I switched from Python to Rust, where cargo as the (very good) package manager comes with the language and, because Rust is a compiled language, you build all the dependencies into your executable you aren't dependent(heh.) on the user having installed a runtime that maybe or maybe not has all the dependencies at the required versions.
- nimish 10y ago
  I have pygradle generate a PEX file for all of my command scripts. Not as seamless as cargo or stack, but it works without rewriting everything.
  - Lev1a 10y ago
    I should probably have clarified that I wanted the "seamless" way of making the final executable. IIRC, I tried(really hard) and failed on getting some method of "freezing" for Python to work, which made me weary of the prospect of trying something like that in Python in the future(now past).
    - lima 10y ago
      pyinstaller works great. No issues whatsoever and I'm happily deploying Python 3.6 to machines as far back as RHEL 5. Single binary to deploy, no dependencies except libc, life is great.
- scriptkiddy 10y ago
  I guess this is true if you don't need to interact with any system libraries.
  - Lev1a 10y ago
    Could you give an example of what "system libraries" would pose a problem in my example of using Rust with cargo? Also note that by "no runtime installed" I mean no runtime as in "no Python runtime", "no JVM" etc. not necessarily "no libc" EDIT: formatting
    - scriptkiddy 10y ago
      >Could you give an example of what "system libraries" would pose a problem in my example of using Rust with cargo? Specifically things like xorg libs, libmpeg, libsdl, and such. Not that Rust would have a problem interfacing with them, just that they would need to be present regardless of whether or not someone was just trying to run a distributed binary. Agreed that you wouldn't need a VM like CPython or the JVM. However, Rust isn't unique in that department. Almost all languages that compile to binary executables have this advantage.
      - Lev1a 10y ago
        > Specifically things like xorg libs, libmpeg, libsdl, and such. Not that Rust would have a problem interfacing with them, just that they would need to be present regardless of whether or not someone was just trying to run a distributed binary. That's why stuff like that is AFAIK usually either distributed with the binary or is absolutely required to have present on the system, regardless of the PL, if you want to/can only distribute a "naked" binary. > Agreed that you wouldn't need a VM like CPython or the JVM. However, Rust isn't unique in that department. Almost all languages that compile to binary executables have this advantage. Didn't mean to suggest this is unique to Rust, which is why I wrote > because Rust is a compiled language. EDIT: formatting
        scriptkiddy 10y ago
- Ruud-v-A 10y ago
  Indeed, Rust + Cargo and Haskell + Stack are very similar in this regard. Both have great package managers, and both produce a shippable executable with only a few dependencies on system libraries. One notable difference is that Stack downloads the compiler, whereas for Rust, every version of the language comes with a compiler and a Cargo. This ensures that you can check out a year-old commit and still build your project with Stack (modulo breaking changes in Stack, which so far I have not encountered), whereas for Rust the compiler version is not pinned.
  - twic 10y ago
    > for Rust the compiler version is not pinned Oh, you just need another layer of abstraction! Install rustup, and then (from memory, might be slightly wrong): rustup install 1.15.1 rustup run 1.15.1 cargo build rustup will take care of getting hold of the right versions of cargo and rustc, and then use them to run the build. I admit that it's not as nice as having the build tool download the right version of everything, but it does work, and you could hide this inside a pretty small shell script or function if you wanted it to be neater.
chowells 10y ago
Probably for the same reason I greatly prefer cabal to stack. Stack assumes it knows better than me. Cabal just does what I tell it to do. As a domain expert, I greatly prefer the latter. It does what I want, nothing more, nothing less. Stack is a mysterious "solution" to a problem I don't have that works by doing everything differently than I do. Stack was created because not everyone is a domain expert. A lot of people don't want to be domain experts. They just want something that works without having to know all the details. It was only able (in the business sense) to be created because so many people look at Haskell skeptically anyway, and take any excuse to back away from it. The people behind the development of stack also run a major advocacy initiative trying to get people to use Haskell, so they found it to be an important thing to build. You don't need to try to get people to use Python. It's already broadly accepted. When people run into trouble, they just say it's the price of using Python, and aren't willing to make the exchange of giving up power to get rid of a minor inconvenience. So there's no business incentive in the Python ecosystem for making the tradeoffs stack does in the Haskell ecosystem.
- baldfat 10y ago
  Cabal made me never use Haskell every again. I work in two different locations and at home. All three locations never worked the same and all had different issues with Cabal. After hours and hours of trying different things I walked away into the wonderland of Racket.
  - arianvanp 10y ago
    They're working really hard on improving it though . Cabal 2.0 will have a nix-style build system, in which multiple verions of the same dependency can be installed globally (so no separate sandbox per project). This will solve most problems of where cabal breaks down. This gives us almost the same usefulness as Stack. However, you will have to make sure that there is actually a feasible build plan, by setting up your version bounds correctly. With stack, other people take care of this for you, and you never touch the version bounds, which is relaxing but also gives you less control.
    - 10y ago
quantumhobbit 10y ago
I think newer often tends to easier because it has the benefit of hindsight in this case. Also the old tools tend to get more complicated as time goes on.
chongli 10y ago
I wonder why the Python ecosystem, which is much more mature I hope I'm not being too pedantic but Python's ecosystem is much larger than Haskell's, it isn't really more mature. Haskell and Python are very similar in age as languages go.
- vkou 10y ago
  Maturity comes from: * Millions of person-hours being poured into a language... * ...Over a long enough time period that the language can go through several develop-eval-improve cycles - that take real world use cases (And not one-liner bubble sort implementations) into account. In this sense, it doesn't matter whether or not Haskell was invented in 1890, or 1990. #2 is required for maturity, but so is #1. (I am not a huge Python fan.)
  - ska 10y ago
    It is important to note that person-hours are not remotely fungible, and some contributions are even negative in the sense of "developing ecosystem maturity".
  - chongli 10y ago
    You can't get 9 women together and produce a baby in 1 month. A lot of developments within a programming language ecosystem originate from new ideas discovered outside of it. It doesn't matter how many people you have working on a project if the crucial piece of tech they need hasn't been discovered yet.
    - vkou 10y ago
      No, but you also can't study 1 woman having 9 babies, and conclude yourself an expert on pregnancy. I'll prefer to get my advice from the doctor who studied 9 women, having one baby each. Note what I said about a mature language having to go through several write-eval-improve cycles. Diversity matters. A language that one person tinkered on for thirty years is far less likely to be useful, then one that ten people tinkered on for three years. Or, in the case of Python vs Haskell, a hundred people who tinkered on it for twenty-five years.
chrisdone 10y ago
Stack requires package sets (aka "snapshots"), which some kind of CI system (Stackage: http://stackage.org/ http://stackage.org/) has to do a daily build job to see if they all build and pass tests together. That requires some money to keep running, and buy-in from package authors as there is a maintenance overhead each release. It took a few years for Stackage to get enough packages in it to be generally useful, and then we wrote the Stack tool which was able to default to using Stackage snapshots. There was (and still is a little bit) of resistance to the whole idea of Stackage from the community; people liked the idea of build plans magically being figured out on demand, it's an interesting research problem (it can be hard to let go of an interesting problem when a solution side-steps it all together). I believe eventually many people changed their minds after experiencing the simplicity of using Stack and not having build hell be a substantial part of their development cycle. Python would likely have to go through the same process. Although with Stack and Yarn providing frozen builds (and QuickLisp); the idea has some precedence, which makes it an easier idea to sell. I mean, Debian and a bunch of other OSes do it like this, but experience shows programmers don't pay attention to that.
- runeks 10y ago
  Stack enables application development in Haskell, as opposed to just library development. A proper library doesn't have more than 20-ish dependencies, in my opinion, and manually handling these and their version bounds is not a problem. But when writing applications with hundreds of dependencies, manually figuring out a mutually compatible dependency range for all packages just isn't an option. At least not if you want to spend time prototyping code, rather than think about dependency ranges. hpack solves additional problems with the .cabal format (sane defaults as opposed to build failure), and I highly recommend it, for application development at least. I just discovered it a month ago and now I wouldn't be able to live without it.

guptaneil 10y ago

It looks like jobmachine is a private repo, maybe don't include the url in your blog post :)

rkrzr 10y ago
I don't think there is any harm done in putting it in there. All GitHub urls follow a standard schema, so if you wanted to you could anyway guess it. The point of the code snippet was simply to highlight how nice the Haskell build tools are compared to Python's.
- Operyl 10y ago
  I couldn't help but try to see if they were nice enough to opensource it, though, hah.
  - rkrzr 10y ago
    It's currently specialized for our specific use case and would not be useful for anyone else, I think. I'm sure that we will keep iterating on it though, so perhaps it will become more general-purpose in the future so we might look into open-sourcing it then. But I can't promise anything, sorry!
yawaramin 10y ago
If the URL to a private repo is not secure, then a whole bunch of people (including GitHub) have a big problem, and exposing jobmachine is the least of their worries ;-)
- guptaneil 10y ago
  It's not a security problem, or else my comment would have been much more adamant about removing the link. It's mostly just reader confusion. Seeing the git clone url led me to believe the project was open sourced and was disappointed to find that not to be the case.
  - 10y ago

mybrid 10y ago

In the 1990s I did research on the efficacy claims of object oriented programming versus procedural programming. This article bares a striking resemblance in the claims. Case study after case study showed that object oriented code had less bugs due to compilers catching bugs, etc. However, almost every study was similar to this report: it was a re-write from procedural to object-oriented. There exists strong evidence to suggest that throwing away ones first prototype produces the same results as to benefits comparing object-oriented versus procedural programming. The conclusion of my research was the same as others: unless one is comparing the same project written from scratch without any sharing of design or code then these studies claims are correlation, not causation.

dsacco 10y ago
Do you have anything published? I'd be very interested in reading.
- mybrid 10y ago
  Sorry, this was course work for a graduate class.
NegatioN 10y ago
Wouldn't a complete rewrite of a module usually be a lot easier if the program is procedural? I think that is one of the big factors stopping that from actually happening with OO code. A rewrite is a lot harder if you tap into what exists elsewhere in the codebase. I bet you have a lot more insight into this than me though.
- kbenson 10y ago
  I'm not so sure about that. I think a complete rewrite is easier if you correctly separated concerns and compartmentalized needs. OOP is supposed to enforce/encourage that, but it's fairly easy to not treat your objects as black boxes with APIs, and then you've put constraints on replacing a component. Procedural code doesn't necessarily tout it's ability to encourage that, but well planned and implemented functions can give you the same benefit. In the end, it's all up the the programmer.

osi 10y ago

makes me wonder how long they would have lasted had the initial implementation been done simpler (ie, not in their PyDatalog fork).

eternalban 10y ago

> Our lead developer Robert usually comes in a bit later, so we had about an hour to build a working prototype. I'm guessing he is a Python developer and likely he is no longer the lead.

rkrzr 10y ago
Don't worry, I'm still here.
- deleted 10y ago
  [deleted]
- KurtMueller 10y ago
  Yes, but are you still the lead? :)
  - rkrzr 10y ago
    I'm still the lead.
    - steelbird 10y ago
      0_0
      - slezakattack 10y ago
        Well, he's the co-founder of Channable so... :P
bbcbasic 10y ago
What's the reasoning?

arianvanp 10y ago

I'm the author of the post. I'll be happy to answer any of your questions :)

fajpunk 10y ago
Thanks for the write up! In the beginning you mention that you ran into some bugs in the Python version that would have been caught by the Haskell type checker. Can you go into more detail about what those bugs were?
- nerdponx 10y ago
  I'm wondering if integrating MyPy into the build, or even using Cython for performance, could have helped.
  - Ruud-v-A 10y ago
    We do actually use Mypy! I wrote about my experience with it here: https://www.reddit.com/r/programming/comments/5x3hdd/channable_how_we_secretly_introduced_haskell_and/degynke/ https://www.reddit.com/r/programming/comments/5x3hdd/channab...
- Ruud-v-A 10y ago
  The most serious one is that we were submitting jobs (as json) that were missing a few metadata fields. In Python we passed around dictionaries, and even though we had json schema validation in place, this slipped through. In Haskell, we define a record type and the corresponding serializers. It is more code, but what you get is that invalid data cannot exist at runtime: it simply cannot be represented. Also, a compiler refuses to compile your code if you make a typo in a field name.
BuuQu9hu 10y ago
Did you try PyPy? If so, how did it perform? If not, why not? PyPy adoption is at only 1% or so, and it'd be nice to see more PyPy usage.

contravariant 10y ago

> The issue here is that we cannot run runWorkerLoop and runFetchLoop with forkIO, because we don’t have the right transformer stack. Am I understanding correctly that this is because, while you can lift e.g. runFetchLoop to something of type IO m (), it's not possible to convert use forkIO on it since it requires an input of type IO ()? Isn't that just a consequence of the fact that Haskell has no possible way of knowing if your side effects can be contained in the IO monad?

dllthomas 10y ago
That sounds about right. At the shallowest level, we can't pass `m ()` to forkIO unless m ~ IO, 'cause the types don't match. But beyond that, there is the question of how that extra context would be passed through. For something like ReaderT this is straightforward. But consider StateT - `set` in one thread can't be visible in the other.
chowells 10y ago
It's not about side effects, it's about bookkeeping. When you have a type that indicates that you can do IO, but you're also carrying around bookkeeping data implicitly for things like configurations and data sources, forkIO represents a hard problem. If the implicit configuration is updated, there's no way to communicate that across threads. The same is true with all the other things monadic layering can provide. How do you call a continuation that points to a different thread? That doesn't even make sense. So.. Why lie in your type and pretend that those things all make sense? Why not make the type explicit about what makes sense and what doesn't? That way, when someone wants to do something that has no a priori way of making sense, they're required to define how to handle it, such that it makes sense in their specific use case. And that's what the post says they did. All in all, it's things working as designed. Places where you need to stop and think are set up such that you need to stop and think to use them, instead of barging ahead unaware of the issues.

daxfohl 10y ago

I'm just starting with Haskell and PureScript. So far I'm liking the latter better. It solves a few of their gripes with respect to strings, laziness and records, plus has a more granular/extendable effects system and cleans up the standard typeclass hierarchy. Also `head []` doesn't crash. Of course Haskell is more mature, has support for multithreading and STM, compiles to native, so it's more performant. But PureScript integrates with JS libraries and seems "fast enough" in many cases. I think it's more interesting as a project too: the lack of runtime and laziness means the compiler code is orders of magnitude simpler than Haskell's, so I could see a larger community building around it if it catches on. Given that they were on Python earlier, I wonder if PureScript would have been a better choice.

chowells 10y ago
I can't imagine using a haskell-like language without laziness. It's what makes it possible to actually write small reusable functions. Tell me, have you ever used foldr in Purescript? It just doesn't lead to reusable logic there, so I have no idea why you would. But in Haskell, foldr is used everywhere. Laziness means that logic built with it is actually reusable.
- arianvanp 10y ago
  Folds are used in purescript all the time! Sure a foldr is less useful because it doesn't have the nice lazy preserving properties, but you can have strict left folds, which are tail recursive, and thus run in constant space. Elm, another haskell-like, _strict_ language, models entire applications around the strict left fold over events https://guide.elm-lang.org/architecture/ https://guide.elm-lang.org/architecture/ We actually do the same in Jobmachine. The application is driven by a strict left fold over incoming events and current state.
  - chowells 10y ago
    There's a reason I said foldr. In Haskell, even foldl is implemented with foldr. It's that powerful of a tool. In contrast, foldl is far weaker, and necessarily strict in its input. It loses nothing moving to mandatory strictness. But foldr loses everything.

yawaramin 10y ago

Nice article. Out of curiosity, what does your write with? Vim/ghc-mod? Emacs/Intero? IntelliJ IDEA/plugins? Atom/VSCode etc.?

arianvanp 10y ago
I tried spacemacs once but I found it too magic. I decided I should learn proper emacs instead of running a random playbox of plugins ontop of plugin. But haven't had the time to properly learn emacs yet. I have tried various haskell plugins for vim in the past, but they always tended to break so I gave up fixing my config and threw them all away. Now it's just plain vim (with some non-haskell related plugins) Next to it I have a terminal that reruns tests when a file changes : `stack test --file-watch` . It's simple but it always works. I'm not sure if the vim stuff got any better lately, I haven't checked. So if you have any suggestions, please tell :)
- sadgit 10y ago
  Have you tried running tests from stack repl --test? Its so much faster. Id love if i could use file-watch and the repl together.
- sjakobi 10y ago
  Have you tried ghcid? In my experience it was a bit quicker than `stack --file-watch`

Twirrim 10y ago

While it's an interesting look at a change you introduced, that blog title might not come across quite as intended. If you're having to "secretly introduce" tech, and "get away with it", that suggests there are unnecessary and unproductive constraints on your work; maybe even suggesting that you'd get in trouble for actually daring to make things better.

_jal 10y ago
Not the author, but I think that is exactly what they wanted to convey in the title. The implication is that they're fighting the man and won. There's a tradition of programmers laying claim to subversively Making Things Better in spite of the bean counters. Sometimes, it is even true, as far as it goes.
arianvanp 10y ago
The "secretly" part refers to part of the story where we had 1 hour to build up quick prototype to pitch to our boss. The title is a bit tongue-in-cheek. It's not that we built this thing in secret for months and then just deployed it. That's also not what the article says :) We had been planning on replacing Scheduler for a while now, and had already written down some mumblings about what the new design should look like. We were also already discussing whether we would switch away from python back then. I think the exact opposite of what you are saying is true. We got the freedom to experiment with something new, and to actually make things better along the way.
- Twirrim 10y ago
  >I think the exact opposite of what you are saying is true. We got the freedom to experiment with something new, and to actually make things better along the way. Sure, and from what I read I mostly took it that way. My original point was just that maybe a bit of caution would be good in the choice of title. If I was just skimming through the titles on HN, or skimming the article, it could be easy to get the wrong impression of channable.

slezakattack 10y ago

"It is said that Haskell programmers are a rare species, but actually the majority of developers at Channable had used Haskell before." Could you imagine if this wasn't the case? The hurdle to actually get people excited about a language such as Haskell especially moving from something like Python would potentially be huge. Kudos for already having that problem solved.

arianvanp 10y ago
We're based in Utrect, where Haskell is part of the mandatory curriculum at and the language of choice for many master courses. Because of this, almost all our developers are somewhat familiar with it, or have at least had one course in it in the past, which really helps a lot!
nanxor 10y ago
In many European countries there are workplaces that only hire people with relevant academic degrees. A course on functional+logic programming is often placed in the 2nd or 3rd year of a typical European 3-3½ year CS degree.

deleted 10y ago

[deleted]

davedx 10y ago

Good luck with hiring!

dualogy 10y ago
Shouldn't be hard.. Haskell to me so far feels like an ecosystem with way fewer jobs / freelance gigs than there are eager-to-go-commercial enthusiasts hacking away in their spare time..
- daxfohl 10y ago
  That is confirmed here: https://stackoverflow.blog/2017/02/07/what-programming-languages-weekends/ https://stackoverflow.blog/2017/02/07/what-programming-langu...
pka 10y ago
They won't need luck, plenty of people would love to program in Haskell professionally.
sjakobi 10y ago
See my reply to a similar comment: https://news.ycombinator.com/item?id=13786992 https://news.ycombinator.com/item?id=13786992

techman9 10y ago

> No messing around with virtualenvs and requirement files. Everybody builds with the same dependency versions. No more “works on my machine” and “did you install the latest requirements?”. While this is nice, of course, I'm not sure that is outcome is unique to Haskell/Stack. It seems like you could accomplish a similar level of reproducibility by building a Docker image or bundling dependencies in some other way.

seagreen 10y ago
My understanding is that Docker only stays reproducible if after every change you kill the container and start a new one. Otherwise a particular change may only be working because of a side-effect of a change you introduced earlier and then deleted. This isn't a huge issue, but still it's nice in declarative systems like Stack and NixOS not to have to worry about that kind of thing.
Ruud-v-A 10y ago
We are actually using Docker for generating the virtualenv that we ship and running tests now. The motivation for doing this is being able to control the environment; we can run tests and build a package on CI, and we can build the same package locally when CI is down. We don’t use Docker in production. It is not clear to me how Docker solves the issue of pinning dependencies; I would rather have a file that states the exact version of every package to install, than an opaque blessed container image that has some versions installed, and I do want to have the versions used under source control. Generating the image would not be reproducible (in the sense of having the same Python files inside it) without pinning versions somewhere anyway, right? Or am I missing something obvious?

xyzzy4 10y ago

Haskell is a bad language, in my opinion, because you can't tell what the O(n) run-time is for any operation. Instead you just have to "trust" that it'll be fast enough. More on this: https://www.reddit.com/r/haskell/comments/1f48dc/what_does_the_haskell_runtime_look_like/ https://www.reddit.com/r/haskell/comments/1f48dc/what_does_t... All of the answers seem insufficient. Basically you can't estimate Haskell run-time unless you are very familiar with the internal Haskell engine.

willtim 10y ago
The complexity of many Haskell collection APIs is actually documented unlike with many other languages. These complexities are not changed by the runtime. Only constant factors and memory use are affected by optimisations and this is no different to any other high-level language that actually optimises code.
- deleted 10y ago
  [deleted]
slezakattack 10y ago
Completely untrue. I'm not a Haskell evangelist (I appreciate it for what it is) but I thought I would at the very least point out that most of the documentation for basic data structures (i.e. Data.List[1]) not only are well-documented but have a link on the far-right side of the documentation site that directly shows you the source code and it's usually easy to tell what it's doing. Any developer should be able to grok that code and determine the run-time complexity. [1] https://hackage.haskell.org/package/base-4.9.1.0/docs/Data-List.html https://hackage.haskell.org/package/base-4.9.1.0/docs/Data-L...
- seagreen 10y ago
  I think he's talking about how it can be hard to tell what's already been evaluated and what hasn't.
- astrobe_ 10y ago
  I'm not a C evangelist, but I would point out that most documentation of standard C function is very detailed and its code is shown in its man page. Any developer should be able to grok that code and determine its safety. Mattaku...

gaius 10y ago

At one company I experimentally wrote OCaml and named the resulting native binaries whatever.py. None ever looked at them. So there is alot of scope for shenanigans.

lima 10y ago

You can deploy Python code as a static binary that includes the interpreter along with all dependencies. I heavily use this in production and life is great - deployment means copying one single binary, reverting means running an older one instead. No external dependencies, no pip upgrades, just libc. https://github.com/pyinstaller/pyinstaller https://github.com/pyinstaller/pyinstaller

kisstheblade 10y ago

Always when I see haskell demonstrations eveyrthing looks like just interface declarations. You can do beautiful interfaces with eg. java also. But where is the meat where anything actually happens? I rarely see that in these posts. Yes I could look up the source but I don't have time to read through it randomly. This looks just so nice and stuff just magically works?: runWorkerLoop :: (MonadBackingStore m, MonadLogger m, MonadIO m) => WorkerState -> m WorkerState And monads to boot! (are monads haskells equivalent of java factories? I kid, I kid :)

Ruud-v-A 10y ago
The runWorkerLoop function logs a few lines and sends out an initial job request (by enqueueing an event in Redis). It then calls the nested function `go`, which dequeues one event of a TBQueue (a thread-safe bounded queue), matches on the event, and calls the right function to handle it. If the event was not a "stop" event, `go` calls itself to do the next iteration of the loop. `go` takes a WorkerState as argument, which is how it keeps track of which jobs are running, and whether there is an unanswered job request. In reality the signature is a bit uglier, I simplified it for the post because the point was about effects. In particular we also pass in the configuration, Redis connection details, and a callback to manipulate the TBQueue.

framp 10y ago

Glad to see Haskell used in production. It's kind of funny that build reproducibility (which was a major issue before stack) is one of the strong point. I wonder if, for your project, using cloudhaskell would have been more appropriate. I have a feeling some of the problems you found could have been solved with that.

javajosh 10y ago

In case anyone was wondering, the 'stack' command in the article refers to https://docs.haskellstack.org/en/stable/README/ https://docs.haskellstack.org/en/stable/README/. Which actually looks kind of wonderful.

hota_mazi 10y ago

> but if we could get it done, there would be no going back The naïveté in this simple statement is so cute. The list of concerns is also pretty naïve. The main problem you are going to encounter with this project is hiring. If you want to grow this project or if the main developers leave the company, I bet it will get rewritten in a different language in no time.

sjakobi 10y ago
See https://news.ycombinator.com/item?id=13784085 https://news.ycombinator.com/item?id=13784085 for why hiring is unlikely to be a problem for them. I also encourage you to find _any_ experience report that tells of difficulties finding the right candidates for a Haskell job.

bykovich 10y ago

Motion for an indefinite moratorium on articles masturbating about Haskell.

vorotato 10y ago

I'd use haskell if it weren't lazy by default and didn't use Cabal. Aka I'd use OCaml/F#.

ctlaltdefeat 10y ago

There are various mature options for scheduling jobs with dependencies between them. Why did you choose to write your own, regardless of the language?

Ruud-v-A 10y ago
Good question! None of the existing options that we investigated supported all of our requirements. In particular, all the arrows that can exist in the dependency graph are known ahead of time, but the nodes are not. This means that a job can depend on a job that does not exist yet. (A user can add extra feeds to download, and the merge job should wait for them, even if it had been submitted already.) Furthermore we have a few specific constraints such as “per project, only one of job type x or y may be running at the same time”.

jwatte 10y ago

Instead of using separate monad transformers, we use a single "World" that knows how to provide Redis, logging, iOS, and other typeclass instances. There is a RealWorld that runs on top of IO and a FakeWorld that runs on top of pure State for unit testing. This means that we have to wrap every single API into our own "SupportsRedis" and similar APIs, but in the end I think it's worth it! Unit tests are super fast and not intermittent at all.

mehaveaccount 10y ago

Actually if you wanted a statically typed compiled functional language of Haskell while keeping the declarative logic paradigm of Prolog, then the Mercury programming language was made exactly for you!