16 ms·

I spun up two Haskell teams at work, and now it composes about half of our codebases. Happy to answer questions about the experience.

klysm 3y ago
Any regrets?
- ilikebits 3y ago
  In the broad scheme of things, not really. I think we waited too long to ship our initial prototype to customers, but a big part of that was that we were pretty sloppy in our product management at that stage of the company (I think we had closed the A about a year ago, and did not yet have our first dedicated PM), and our technical rewrite was also partially a feature and UX rewrite. It's been several years since, and we have clawed our way out of that hole, and I think our delivery process is actually in a really good place now. If I were to give advice to people looking at adopting new languages, I would say to get it into production ASAP. I believe the keyword to search for is the "tracer bullet pattern". One of my favorite blog posts on this is https://blog.thepete.net/blog/2019/10/04/hello-production/ https://blog.thepete.net/blog/2019/10/04/hello-production/
  - amelius 3y ago
    Isn't the "tracer bullet" pattern what you get for free in many IDEs when you click "New Project"?
_8j50 3y ago
If I may ask a naive question: why? Is there a specific advantage it gives you, like Ocaml in finance/trading?
- ilikebits 3y ago
  For us, there were a couple advantages. For context, I work at FOSSA (https://fossa.com/ https://fossa.com/). Our core product solves software supply chain needs in enterprises (around licensing and security), and our core technology is around compiler, build, and source code analysis. Off the top of my head, 3 advantages stood out: 1. First, if you're not going that far off the beaten low-level path, Haskell has incredible productivity benefits. Effect tracking has enormous benefits for testability and understandability. If you've ever been down a debugging rabbit hole shaped like "there's no way this logging call is sending that API request", then you might be pleasantly surprised to discover that you can statically guarantee that this doesn't occur in Haskell programs! Pattern matching, algebraic data types (sum types!), and typeclass derivation make it much easier to make it impossible to construct invalid representations of data. Other languages are finally picking this up, but their versions of pattern matching often have caveats for backwards-idiom-compatibility. And monads are a very powerful abstraction. It's like being able to write your own semantics for async-await (I've talked more about this before at https://lobste.rs/s/7cllte/monads_part_six_really_what_is_monad#c_nbusni https://lobste.rs/s/7cllte/monads_part_six_really_what_is_mo...). 2. Haskell was a good domain fit for us. One thing we build is the FOSSA CLI (https://github.com/fossas/fossa-cli/ https://github.com/fossas/fossa-cli/), which runs in customer CI pipelines to analyze their builds. It's a very compiler-shaped problem: shell out to some tools, do a lot of parsing, think very hard, and then spit out a JSON blob to send back to the API. Our first version of this was written in Go. At the time of development, writing correct, testable parsers in Go was like pulling teeth. We have a relatively small headcount-to-product-surface-area ratio, and our team was running up against the overhead of rewriting traverse in Go over and over again (that's a Haskell-flavored joke, but if you've ever been annoyed at writing yet another for-loop in Go, you get it). We decided to hack out a prototype in Haskell, and it turned out to be a good fit. 3. Lastly, the kind of people who wind up working at FOSSA and are interested in the code analysis bits tend to be the same kind of nerds who love Haskell. We had lots of people on our team who were chomping at the bit to try it, so we decided to try it out. I really can't understate how big of a productivity difference it makes when people are working with tools that they actually enjoy rather than are merely forcing themselves to use. It is night and day. If you want to learn more, we also did an interview with Serokell on this topic (https://serokell.io/blog/haskell-in-production-fossa https://serokell.io/blog/haskell-in-production-fossa), and discussed it on an episode of our engineering podcast (https://fossa.com/blog/fossa-podcast-adopting-haskell/ https://fossa.com/blog/fossa-podcast-adopting-haskell/).
- jstx1 3y ago
  OCaml doesn’t give a specific advantage in trading, it’s just that Jane Street like to be pretentious about most things including programming languages.
  - nequo 3y ago
    Do you mean no specific advantage relative to C++? If so, then wouldn’t it be an advantage that the GC lets you be (nearly) certain that you have no memory bugs? Or do you mean relative to Java?
becquerel 3y ago
What industry? Most Haskell use seems to happen in finance or similar fields.
- ilikebits 3y ago
  Developer tools! I think Haskell and OCaml are popular in finance because they're high-level and familiar to math/quant types (who are broadly--and this is a sweeping generalization--more familiar with expressing algorithms in recursion than in procedural sequences). I don't have much experience with OCaml, but I can also say that Haskell in particular has incredible support for building very high-level libraries that are both expressive and have strong compile-time guarantees. I've found it's relatively easier for a programming expert to build such a library and a domain expert to consume the library in Haskell than in many other languages I've seen (perhaps Python comes the closest, but that feels to me more like an ecosystem thing than a language thing).
- pid-1 3y ago
  Which finance companies are using Haskell?
  - sigrlami 3y ago
    You can also check here https://haskellcosm.com/ https://haskellcosm.com/ Click on "Area" column header to sort.
  - tromp 3y ago
    One that is quite visible in Haskell circles is Standard Chartered Bank [1]. [1] https://serokell.io/blog/haskell-in-production-standard-chartered https://serokell.io/blog/haskell-in-production-standard-char...
- sigrlami 3y ago
  Check 2nd chart here(not mobile-ready): https://haskellcosm.com/analysis.html https://haskellcosm.com/analysis.html
  - 3y ago
edejong 3y ago
Really curious: Half of the code base in LoC or in functionality?
- ilikebits 3y ago
  In functionality, roughly. By LoC, I think the majority of our codebase is still JavaScript/TypeScript. Many of the subsystems still in TS are being rewritten in Haskell (not because we have a "rewrite in Haskell" mandate, but because they are due for a rewrite anyway due to scale and requirement changes, and the team that owns that domain now primarily writes in Haskell).
  - golergka 3y ago
    > Many of the subsystems still in TS are being rewritten in Haskell Have you tried fp-ts ecosystem? Code written in it looks pretty similar to Haskell, but it's still the same language, and you can gradually adopt it in the codebase instead of re-writing the whole thing. (I've also sent a question about personal professional development to your email from profile, hope this isn't abuse of your kindness here)
    - CodeAndCuffs 3y ago
      Fp-ts is one of like, 4 things in life I feel the need to shill for. It's docs are a little rough coming into it for the first time, and I think some of gcanti's tutorials are a little to complex. But I've slipped it into 3 or 4 moderate sized projects. Every time someone goes to touch it there's initial confusion, a 5 minute explanation of Either, 5 minutes of Q and A, and then they love it.
      - golergka 3y ago
        The only bad thing about this is, now that I've started writing functional code with it, I don't want to go back. I'm in-between jobs right now, and finding a company which utilizes it is a challenge.
    - 3y ago
anta40 3y ago
What kind of applications your team are working it? How's the overall development process felt? I'd like to know the reason to choose Haskell over other languages.
- ilikebits 3y ago
  Re: applications and language comparison, see my answer over at https://news.ycombinator.com/item?id=36744384 https://news.ycombinator.com/item?id=36744384 Re: development process - it's very similar to development in other languages. Write, compile, yell at compiler, push, complain about how slow CI is, deploy. You know, the usual. I think the most interesting difference is the _onboarding_ curve. Haskell's curve is pretty brutal, although I think most of this is because of bad pedagogy (many monad tutorials are bad, and beginners can't tell) rather than because of intrinsically difficult concepts. Some observations: 1. Empirically, zero to code review is roughly six weeks for a professional industry software engineer. It's not that much longer than other languages we've had to teach. But it _feels_ very brutal because zero to side project is roughly three or four weeks. Contrast this against Go, where zero to side project is about five minutes. 2. Having an experienced Haskell engineer on your team to start with makes a WORLD of a difference. You've gotten a type error - why? Is it because GHC is doing weird backwards type inference stuff again? Or is it because you've misunderstood this fundamental concept? Or is it because you've done a typo, and GHC has inferred a downstream site to be a type error? This sort of thing is very difficult to explain in words and in general, and much easier to pick up through experience and mentorship. If you do not have an experienced Haskeller at your disposal, I would strongly recommend starting with side projects first, and using the Functional Programming Slack (fpslack.com), who are some of the friendliest and most patient folks I've had the pleasure of talking to.
  - Tade0 3y ago
    > (many monad tutorials are bad, and beginners can't tell) It appears monads truly are something you can either understand or explain, but not both. I find it suspicious. I mean, plenty of Haskell devs out there, surely it's not that hard?
agumonkey 3y ago
What were the positive surprises, the negative surprises (even simple stuff). What's your team workflow around designing the code ? Do you follow mainstream ideas or do you have very special tricks (say innovation on top of property based testing.. metaprogramming.. whatever)
- ilikebits 3y ago
  Re: positive and negative - refer to my answer in https://news.ycombinator.com/item?id=36744384 https://news.ycombinator.com/item?id=36744384 and especially the linked Serokell interview, where we dived into this question specifically. Re: design workflow - this is not very different from any other code. You have module interfaces, logical subsystems, separately deployable service entrypoints, etc. Our Haskell code tends to take special care to avoid statefulness and globals, but this is not any more of a design burden than writing idiomatic code in any other language (e.g. planning out your classes in Java). Re: special tricks - we used fused-effects to model our effects, which has been pretty useful (it lets us delay teaching transformers), but this is by no means a secret that we've discovered. I think the biggest trick is to write simple code. I think when you give a junior engineer a sufficiently powerful hammer (like Haskell and its effect tracking), they will attempt to nail absolutely everything to the wall (e.g. build a taxonomy of every possible effect and every possible exception and write your whole program in this framework for Maximum Compile Time Safety). Avoid this. It's just not a productive use of time, and the gains to safety often are not worth the effort and velocity and overhead you paid to build the underlying framework. Focus on the high-ROI piece of the program. Write the effects that you would use for testing, do the rest in IO for now, and come back later if you find that you need more granularity in your effects. The important thing is to ship the product. (I go into much more detail about this in the Serokell interview linked in the other HN comment.)
  - agumonkey 3y ago
    thanks a lot, very much so :)
rr808 3y ago
Have you ever left a job with a Haskell codebase and then the organization regretted it? Usually niche technology becomes difficult to support and often ported out.
- nickpeterson 3y ago
  I once built a data import tool using F# (It was a .net shop). I left for another position and my intention was to help them with consulting-style work, but my next position was more demanding than I anticipated and my former company was forced to deal with it. I consider it one of my larger mistakes. In hindsight the real problem was building the thing in isolation and then dumping it on someone else with minimal training time. I think different tools and languages are fine, but they need a bus factor of 2-3 to prevent this kind of stuff.
- ilikebits 3y ago
  Excellent question, and this was a serious concern for us. And we have experience with this! We had one deep Haskell expert on the team who helped build a bunch of core primitives who has since left to another company. One of the motivations for adoption for us was strong interest on the team. Looking back on our team... 1. We started with me (enthusiast), our expert,and a Go developer with no Haskell experience. 2. Over time, at its peak, the team grew to us 3, Python developer who had dabbled, two Ruby developers who were interested but had no experience, a frontend expert who had dabbled, and another engineer who I didn't work with closely but I think had a background in Java? 3. Our expert left the team before or around the time that the last person (maybe-Java person) joined, I think. 4. On the second Haskell team I spun up, it's me, one person from the old team, a Ruby/JS developer, and a JS developer. We were able to get all of these people productive in Haskell. It got easier once we had more experience teaching folks. There are a handful of _key differences_ between Haskell and ALGOL-family languages (mostly around evaluation model and effect tracking), and once you nail those down, the rest is pretty smooth sailing, and your SWE experience and intuition begins kicking in again. Although we miss our expert very much every day (they were a very cool person! They drove a motorcycle!), their departure has not had an outsized impact on our velocity. A sibling comment recommended a bus factor of 2-3. I think this is roughly correct, although I would think of this as not merely your "don't get hit by a bus" group, but also your core teaching group.
ju7k 3y ago
*comprises
archibaldJ 3y ago
Hi there. I'm working on my first Haskell app in production; Just wondering.. Do you guys use parsec in production? What is the motivation for having Haskell to be half of the codebase? (Are there some open-source domain-specific libraries that are used intensively? Or do you guys prefer to implement things in-house (on top of libs that are more abstract)?) And if parsec is used in production, any tips on how to best design/debug parsec parsers? (Thanks!) Lastly, in your experience, what are some of the best monads / design patterns to improve team productivity? :{ Really curious about it

klysm 3y ago

Any regrets?

ilikebits 3y ago
In the broad scheme of things, not really. I think we waited too long to ship our initial prototype to customers, but a big part of that was that we were pretty sloppy in our product management at that stage of the company (I think we had closed the A about a year ago, and did not yet have our first dedicated PM), and our technical rewrite was also partially a feature and UX rewrite. It's been several years since, and we have clawed our way out of that hole, and I think our delivery process is actually in a really good place now. If I were to give advice to people looking at adopting new languages, I would say to get it into production ASAP. I believe the keyword to search for is the "tracer bullet pattern". One of my favorite blog posts on this is https://blog.thepete.net/blog/2019/10/04/hello-production/ https://blog.thepete.net/blog/2019/10/04/hello-production/
- amelius 3y ago
  Isn't the "tracer bullet" pattern what you get for free in many IDEs when you click "New Project"?

_8j50 3y ago

If I may ask a naive question: why? Is there a specific advantage it gives you, like Ocaml in finance/trading?

ilikebits 3y ago
For us, there were a couple advantages. For context, I work at FOSSA (https://fossa.com/ https://fossa.com/). Our core product solves software supply chain needs in enterprises (around licensing and security), and our core technology is around compiler, build, and source code analysis. Off the top of my head, 3 advantages stood out: 1. First, if you're not going that far off the beaten low-level path, Haskell has incredible productivity benefits. Effect tracking has enormous benefits for testability and understandability. If you've ever been down a debugging rabbit hole shaped like "there's no way this logging call is sending that API request", then you might be pleasantly surprised to discover that you can statically guarantee that this doesn't occur in Haskell programs! Pattern matching, algebraic data types (sum types!), and typeclass derivation make it much easier to make it impossible to construct invalid representations of data. Other languages are finally picking this up, but their versions of pattern matching often have caveats for backwards-idiom-compatibility. And monads are a very powerful abstraction. It's like being able to write your own semantics for async-await (I've talked more about this before at https://lobste.rs/s/7cllte/monads_part_six_really_what_is_monad#c_nbusni https://lobste.rs/s/7cllte/monads_part_six_really_what_is_mo...). 2. Haskell was a good domain fit for us. One thing we build is the FOSSA CLI (https://github.com/fossas/fossa-cli/ https://github.com/fossas/fossa-cli/), which runs in customer CI pipelines to analyze their builds. It's a very compiler-shaped problem: shell out to some tools, do a lot of parsing, think very hard, and then spit out a JSON blob to send back to the API. Our first version of this was written in Go. At the time of development, writing correct, testable parsers in Go was like pulling teeth. We have a relatively small headcount-to-product-surface-area ratio, and our team was running up against the overhead of rewriting traverse in Go over and over again (that's a Haskell-flavored joke, but if you've ever been annoyed at writing yet another for-loop in Go, you get it). We decided to hack out a prototype in Haskell, and it turned out to be a good fit. 3. Lastly, the kind of people who wind up working at FOSSA and are interested in the code analysis bits tend to be the same kind of nerds who love Haskell. We had lots of people on our team who were chomping at the bit to try it, so we decided to try it out. I really can't understate how big of a productivity difference it makes when people are working with tools that they actually enjoy rather than are merely forcing themselves to use. It is night and day. If you want to learn more, we also did an interview with Serokell on this topic (https://serokell.io/blog/haskell-in-production-fossa https://serokell.io/blog/haskell-in-production-fossa), and discussed it on an episode of our engineering podcast (https://fossa.com/blog/fossa-podcast-adopting-haskell/ https://fossa.com/blog/fossa-podcast-adopting-haskell/).
jstx1 3y ago
OCaml doesn’t give a specific advantage in trading, it’s just that Jane Street like to be pretentious about most things including programming languages.
- nequo 3y ago
  Do you mean no specific advantage relative to C++? If so, then wouldn’t it be an advantage that the GC lets you be (nearly) certain that you have no memory bugs? Or do you mean relative to Java?

becquerel 3y ago

What industry? Most Haskell use seems to happen in finance or similar fields.

ilikebits 3y ago
Developer tools! I think Haskell and OCaml are popular in finance because they're high-level and familiar to math/quant types (who are broadly--and this is a sweeping generalization--more familiar with expressing algorithms in recursion than in procedural sequences). I don't have much experience with OCaml, but I can also say that Haskell in particular has incredible support for building very high-level libraries that are both expressive and have strong compile-time guarantees. I've found it's relatively easier for a programming expert to build such a library and a domain expert to consume the library in Haskell than in many other languages I've seen (perhaps Python comes the closest, but that feels to me more like an ecosystem thing than a language thing).
pid-1 3y ago
Which finance companies are using Haskell?
- sigrlami 3y ago
  You can also check here https://haskellcosm.com/ https://haskellcosm.com/ Click on "Area" column header to sort.
- tromp 3y ago
  One that is quite visible in Haskell circles is Standard Chartered Bank [1]. [1] https://serokell.io/blog/haskell-in-production-standard-chartered https://serokell.io/blog/haskell-in-production-standard-char...
sigrlami 3y ago
Check 2nd chart here(not mobile-ready): https://haskellcosm.com/analysis.html https://haskellcosm.com/analysis.html
- 3y ago

edejong 3y ago

Really curious: Half of the code base in LoC or in functionality?

ilikebits 3y ago
In functionality, roughly. By LoC, I think the majority of our codebase is still JavaScript/TypeScript. Many of the subsystems still in TS are being rewritten in Haskell (not because we have a "rewrite in Haskell" mandate, but because they are due for a rewrite anyway due to scale and requirement changes, and the team that owns that domain now primarily writes in Haskell).
- golergka 3y ago
  > Many of the subsystems still in TS are being rewritten in Haskell Have you tried fp-ts ecosystem? Code written in it looks pretty similar to Haskell, but it's still the same language, and you can gradually adopt it in the codebase instead of re-writing the whole thing. (I've also sent a question about personal professional development to your email from profile, hope this isn't abuse of your kindness here)
  - CodeAndCuffs 3y ago
    Fp-ts is one of like, 4 things in life I feel the need to shill for. It's docs are a little rough coming into it for the first time, and I think some of gcanti's tutorials are a little to complex. But I've slipped it into 3 or 4 moderate sized projects. Every time someone goes to touch it there's initial confusion, a 5 minute explanation of Either, 5 minutes of Q and A, and then they love it.
    - golergka 3y ago
      The only bad thing about this is, now that I've started writing functional code with it, I don't want to go back. I'm in-between jobs right now, and finding a company which utilizes it is a challenge.
  - 3y ago

anta40 3y ago

What kind of applications your team are working it? How's the overall development process felt? I'd like to know the reason to choose Haskell over other languages.

ilikebits 3y ago
Re: applications and language comparison, see my answer over at https://news.ycombinator.com/item?id=36744384 https://news.ycombinator.com/item?id=36744384 Re: development process - it's very similar to development in other languages. Write, compile, yell at compiler, push, complain about how slow CI is, deploy. You know, the usual. I think the most interesting difference is the _onboarding_ curve. Haskell's curve is pretty brutal, although I think most of this is because of bad pedagogy (many monad tutorials are bad, and beginners can't tell) rather than because of intrinsically difficult concepts. Some observations: 1. Empirically, zero to code review is roughly six weeks for a professional industry software engineer. It's not that much longer than other languages we've had to teach. But it _feels_ very brutal because zero to side project is roughly three or four weeks. Contrast this against Go, where zero to side project is about five minutes. 2. Having an experienced Haskell engineer on your team to start with makes a WORLD of a difference. You've gotten a type error - why? Is it because GHC is doing weird backwards type inference stuff again? Or is it because you've misunderstood this fundamental concept? Or is it because you've done a typo, and GHC has inferred a downstream site to be a type error? This sort of thing is very difficult to explain in words and in general, and much easier to pick up through experience and mentorship. If you do not have an experienced Haskeller at your disposal, I would strongly recommend starting with side projects first, and using the Functional Programming Slack (fpslack.com), who are some of the friendliest and most patient folks I've had the pleasure of talking to.
- Tade0 3y ago
  > (many monad tutorials are bad, and beginners can't tell) It appears monads truly are something you can either understand or explain, but not both. I find it suspicious. I mean, plenty of Haskell devs out there, surely it's not that hard?

agumonkey 3y ago

What were the positive surprises, the negative surprises (even simple stuff). What's your team workflow around designing the code ? Do you follow mainstream ideas or do you have very special tricks (say innovation on top of property based testing.. metaprogramming.. whatever)

ilikebits 3y ago
Re: positive and negative - refer to my answer in https://news.ycombinator.com/item?id=36744384 https://news.ycombinator.com/item?id=36744384 and especially the linked Serokell interview, where we dived into this question specifically. Re: design workflow - this is not very different from any other code. You have module interfaces, logical subsystems, separately deployable service entrypoints, etc. Our Haskell code tends to take special care to avoid statefulness and globals, but this is not any more of a design burden than writing idiomatic code in any other language (e.g. planning out your classes in Java). Re: special tricks - we used fused-effects to model our effects, which has been pretty useful (it lets us delay teaching transformers), but this is by no means a secret that we've discovered. I think the biggest trick is to write simple code. I think when you give a junior engineer a sufficiently powerful hammer (like Haskell and its effect tracking), they will attempt to nail absolutely everything to the wall (e.g. build a taxonomy of every possible effect and every possible exception and write your whole program in this framework for Maximum Compile Time Safety). Avoid this. It's just not a productive use of time, and the gains to safety often are not worth the effort and velocity and overhead you paid to build the underlying framework. Focus on the high-ROI piece of the program. Write the effects that you would use for testing, do the rest in IO for now, and come back later if you find that you need more granularity in your effects. The important thing is to ship the product. (I go into much more detail about this in the Serokell interview linked in the other HN comment.)
- agumonkey 3y ago
  thanks a lot, very much so :)

rr808 3y ago

Have you ever left a job with a Haskell codebase and then the organization regretted it? Usually niche technology becomes difficult to support and often ported out.

nickpeterson 3y ago
I once built a data import tool using F# (It was a .net shop). I left for another position and my intention was to help them with consulting-style work, but my next position was more demanding than I anticipated and my former company was forced to deal with it. I consider it one of my larger mistakes. In hindsight the real problem was building the thing in isolation and then dumping it on someone else with minimal training time. I think different tools and languages are fine, but they need a bus factor of 2-3 to prevent this kind of stuff.
ilikebits 3y ago
Excellent question, and this was a serious concern for us. And we have experience with this! We had one deep Haskell expert on the team who helped build a bunch of core primitives who has since left to another company. One of the motivations for adoption for us was strong interest on the team. Looking back on our team... 1. We started with me (enthusiast), our expert,and a Go developer with no Haskell experience. 2. Over time, at its peak, the team grew to us 3, Python developer who had dabbled, two Ruby developers who were interested but had no experience, a frontend expert who had dabbled, and another engineer who I didn't work with closely but I think had a background in Java? 3. Our expert left the team before or around the time that the last person (maybe-Java person) joined, I think. 4. On the second Haskell team I spun up, it's me, one person from the old team, a Ruby/JS developer, and a JS developer. We were able to get all of these people productive in Haskell. It got easier once we had more experience teaching folks. There are a handful of _key differences_ between Haskell and ALGOL-family languages (mostly around evaluation model and effect tracking), and once you nail those down, the rest is pretty smooth sailing, and your SWE experience and intuition begins kicking in again. Although we miss our expert very much every day (they were a very cool person! They drove a motorcycle!), their departure has not had an outsized impact on our velocity. A sibling comment recommended a bus factor of 2-3. I think this is roughly correct, although I would think of this as not merely your "don't get hit by a bus" group, but also your core teaching group.

ju7k 3y ago

*comprises

archibaldJ 3y ago

Hi there. I'm working on my first Haskell app in production; Just wondering.. Do you guys use parsec in production? What is the motivation for having Haskell to be half of the codebase? (Are there some open-source domain-specific libraries that are used intensively? Or do you guys prefer to implement things in-house (on top of libs that are more abstract)?) And if parsec is used in production, any tips on how to best design/debug parsec parsers? (Thanks!) Lastly, in your experience, what are some of the best monads / design patterns to improve team productivity? :{ Really curious about it