12 ms·

A couple million lines of Haskell: Production engineering at Mercury

435 points by unignorant 5mo ago

nektro 5mo ago
lots of really great pieces of advice in this
maz1b 5mo ago
I think perhaps contrary to popular belief, Mercury choosing Haskell and their early leadership having such a storied experience in it probably played some non-insignificant role in their success. As a customer of Mercury, it's truly one of the critical companies my toolkit, and I just can't help but feel that their choosing of Haskell made their progress, development and overall journey that much better. I realize that you can make this argument with most languages, and it's not to say that a FP lang like Haskell is a recipe for success, but this intentional decision particularly pre "vibe coding" and the LLM era seems particularly prescient, of course combined with their engineering culture that was detailed in the post.
- 1024bits 5mo ago
  I'd also wager that hiring generalists with no prior experience in the language actually helped them, because they got to instill their culture and style from the ground up with their new hires. Pre vibe-coding, most of those people would'nt have wanted to just jump in and hack away with zero instruction.
- ipnon 5mo ago
  I have noticed that everything in their app Just Works. It's very satisfying coming from other services!
  - jwsteigerwalt 5mo ago
    I feel the same way. I only started using Mercury about 6 months ago and I’m continually impressed that it just makes sense.
- spopejoy 5mo ago
  I would counter that it was probably their startup-oriented fintech focus and execution that led to their success. I love good tech culture as much as the next HNer but I've seen companies with great tech die because of bad biz focus. I might further argue that the startup-y fintech culture led to good tech culture. The fact that they didn't start as a bank (as opposed to say SVB) means that they didn't have to be as conservative, or integrate with some horrific ancient tech stack. I'm pleased they've had such success with Haskell, but much like Jane Street and OCAML, I think the language choice is almost accidental*, as much as the companies would like you to believe otherwise. I would like to know however what they're doing for front-end. I would guess that all of this Haskell is back-end only. *EDIT by "accidental" I mean to the business side. Jane St had some good trades, Mercury had great focus and execution. They also have some good tech :)
dnnddidiej 5mo ago
I think you have to get a Haskell job early in career and stick to Haskell jobs. Breaking in is really hard as you come without experience there will be plenty of others with Haskell experience to compete. And because the jobs are rare if it doesnt work out (company becomes bad to work for or layoff) you can be unstuck (or I guess you would switch to Rust, Scala or F#)
- matt-noonan 5mo ago
  As somebody who has helped hire many Haskell devs, I can say that lots of Haskell experience isn't always a positive. We have to filter carefully to make sure that we end up with developers who want to build real things, not developers who just want to get paid for noodling around with Haskell. As far as I'm concerned, I'd much rather hire somebody with lots of experience building things who ended up coming to Haskell later because they viscerally understand the benefits and risks. Somebody with lots and lots of Haskell experience who never delivered much is a big risk.
  - dnnddidiej 5mo ago
    Interesting, I guess it then depends on the company (or recruiter) then.
  - _qhqq 5mo ago
    haha i've abused this recruiting mindset for a decade it's so easy to scout when a company has this haskell philosophy. either by the interviewers themselves or by the bloggers they hired to guide their team. the trick? i just..lie. "oh yeah i'm super pragmatic. i'm not hardline about haskell. i don't think you should be fancy." see how easy it is? i am suddenly hired and got a fat raise. and if the company moves off haskell? i quit immediately, get another haskell job, and talk to my former coworkers on the way out to embolden them to do the same. it helps that i have the "real world" stuff on my resume. i rode the 2010s job hopping ride as a haskeller doing this. each time a 20-30% raise. and i get to still write haskell. and i am always a top percentile haskeller at the company so i can code however tf i want lolol. suddenly - singletons, Generics, HKD! so here's to earning another million bucks "noodling around with Haskell" :cheers:
threethirtytwo 5mo ago
I really believe in FP and Haskell but I want to examine this objectively. Empirically speaking is what Mercury done successful truly because of Haskell? Do they have metrics that demonstrates clear superiority along some vectored trait like complexity, bug count, etc? >A couple million lines of Haskell, maintained by people who learned the language on the job, at a company that moves huge amounts of money? The conventional wisdom says this should be a disaster, but surprisingly, it isn't. The system we've built has worked well for years, through hypergrowth, through the SVB crisis that sent $2 billion in new deposits our way in five days,1 through regulatory examinations, through all the ordinary and extraordinary things that happen to a financial system at scale. This one is quite telling. Do people have counter examples?
- cfiggers 5mo ago
  Without having run the whole company twice in parallel, once using Haskell and again in some other language, and without having measured both runs exactly the same way, I don't think metrics like you're interested in could possibly have sufficient context to mean anything reliable. Obviously Mercury is successful, and obviously Haskell is how they did it. So it's essential to their success. Would it be instrumental to anyone else's anywhere else doing anything else? Can't possibly know, I don't think.
  - threethirtytwo 5mo ago
    I’m asking for solutions and answers. Yeah. I’m aware of how hard it is to get metrics. You can still compare lines of code and bug rate over the same period of time.
    - IIsi50MHz 5mo ago
      You can, but then "The cake is a lie.", because linecount and bug rate, when concieved as proxies for productivity[1] or quality rarely match up with reality in a way that allows you to make predictions or reason about past outcomes. You can reason about frequency of particular types bugs, such as null pointers or overflow, or whether those bugs can occur at all. [1] https://www.folklore.org/Negative_2000_Lines_Of_Code.html https://www.folklore.org/Negative_2000_Lines_Of_Code.html
le-mark 5mo ago
It’s hard to imagine what two millions lines of Haskell could possibly be doing. I mean that’s a lot of code and I have the impression that Haskell is “tight” meaning a little code can do a lot. Maybe they have a lot of libraries to do things like json serializing/deserializing, rest api frameworks, logging etc?
- imoverclocked 5mo ago
  From TFA: > The problem is that we cannot trust code we cannot instrument. If a third-party binding makes HTTP calls through concrete functions, we have no way to add tracing, no way to inject timeouts tuned to our SLOs, no way to simulate partner outages in testing, and no way to explain the 400ms gap in a trace except by squinting at it and developing theories. So we write our own. More work upfront, but the clients we write are observable by construction, because we built them that way from the start.
  - troupo 5mo ago
    > If a third-party binding makes HTTP calls through concrete functions, we have no way to add tracing, no way to inject timeouts tuned to our SLOs, no way to simulate partner outages in testing, and no way to explain the 400ms gap in a trace Given that tracing etc. is IO, are they just threading IO through the entirety of all their Haskell code?
    - tome 5mo ago
      Tracing doesn’t actually require IO, only emitting the traces does, and those two need not be done at the same point. In any case, anywhere they’re doing HTTP calls they are already threading IO, so they don’t have to pay an additional cost.
- verandaguy 5mo ago
  Nit: the quality of a language that you call "tight" is usually called "expressive." You can use few characters to express a relatively very abstract idea. Some people call this "high-level," too. I will say, though, that 2 million lines of code is much less code than it sounds like at first glance, especially for a company in a highly-regulated space like finance, plus a few years of progress.
bri3d 5mo ago
> Haskell gives you tools to encode these incantations in types so they cannot be forgotten. This is, for my money, the single most valuable thing the language offers a production engineering organization. Haskell is admittedly, probably the most powerful widely (or even somewhat widely) used language for doing this, but this general pattern works really well in Rust and TypeScript too and is one of my very favorite tools for writing better code. I also really like doing things like User -> LoggedInUser -> AccessControlledLoggedInUser to prevent the kind of really obvious AuthZ bugs people make in web applications time and time again. I've found this pattern to be massively underutilized in industry.
- miki123211 5mo ago
  This isn't specific to Rust or Typescript. You can do this in basically any language. Imagine you have to distinguish between unescaped and escaped strings for security purposes. Even with a dynamically typed language, you can keep escaped strings as an Escaped class, with escape(str)->Escaped and dangerouslyAssumeEscaped(str)->Escaped functions (or static methods). There's a performance cost to this, so that's a tradeoff you have to weigh, but it is possible. Another way of doing this is Application Hungarian[1], though that relies on the programmer more than it does on the compiler. [1] https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...
  - wyager 5mo ago
    > You can do this in basically any language. You can do it in Assembly. That doesn't mean it's cost effective.
    - myst 5mo ago
      Costs are a skill issue ;-)
      - intrinsicallee 5mo ago
        You demonstrate well the problem: yes anything that is computable can be than in any computation system. That's not what discussions about tooling are about. If a tool can help enforce some ways of doing things, or if it doesn't constrain people much, that has consequences for the type of work that gets done with them and the systems you encounter running out there that you might be invited or find the need to work with. "I can do it" is exactly the wrong answer. "How can I guarantee that others will do it" is the point being made.
faangguyindia 5mo ago
I use Haskell a lot, but I notice that it's very hard to cross-compile it. If only cross-compilation became easy so that I can develop on my chip Macs and deploy on x64/AMD Linux servers. >statically linking Haskell binaries is quite a challenge >build requirements really slow down the process. I have to use dockers to help cache dependencies and avoid recompiling things that have not changed, but it is still slow and puts out large binaries. Also, the Docker-based deployment takes a lot of time as it needs to recompile each module. While you can cache some part of it, it's still slow. Meanwhile with Go it's painless. And i am not the only one having this issue: https://news.ycombinator.com/item?id=47957624#47972671 https://news.ycombinator.com/item?id=47957624#47972671 Such a shame Haskell is beautiful and performant language still build is slow.
amitbidlan 5mo ago
[flagged]
golem14 5mo ago
I'm not particularly into Haskell or functional languages, but I came here to say it warms my heart to hear that people in finance actually embrace it (also J/APL). It seems like a good choice for banking. The Mercury site also looks way better than most other banks I have ever used (load speed is also very good.) On the danger of seeming like a shill (I'm not), I'm tempted to try them out.
- sillysaurusx 5mo ago
  Do it. You won’t be disappointed. I’ve been using them for about 5 years now.
thot_experiment 5mo ago
My bestie works at this company and looking from the outside they have a good engineering culture. I do think Haskell is the right tool for the job, and they are playing to it's strengths, but part of me wonders if a lot of their success is attributable to the place just being well run in general.
- ironmagma 5mo ago
  That would not run counter to the popular (whether true or not) idea that by using functional programming languages you filter for a higher quality labor pool / applicant pool.
  - runevault 5mo ago
    The version I've always heard is just well designed but less popular languages, but the ones I can think of were all functional (Haskell/F#/OCaml/Clojure/Elm/Erlang)
  - spopejoy 5mo ago
    That wouldn't apply here, since as the article says they hire "generalists, and most of them have never written a line of Haskell before joining." In any case, I think the "Haskell tax" concept (where you can pay well-paid programmers less if you have a Haskell shop) is stale by now. Rust attracted away a lot of FP-ers, plus mainstream langs like C++, Java and even Typescript got smarter. Haskell's biggest problem by far is the tiny labor pool, which Mercury seems to wisely avoid.
    - sn9 5mo ago
      The post explicitly makes the case for the filtering playing a role. Ctrl-F "Python".
- meken 5mo ago
  > but part of me wonders if a lot of their success is attributable to the place just being well run in general That was my sense reading the article - that the author would be running a successful engineering org using any language really.
hmokiguess 5mo ago
> written using the mosaic theory of information and a range of journalistic tools what does that mean?
sillysaurusx 5mo ago
I just wanted to leave a glowing comment about Mercury itself, since this is one of the few times I’ll be able to. I’ve been using Mercury for 5 years. In that time, I’ve been able to wire transfer money without having to worry it might disappear (functionally impossible at certain other banks), created hundreds of virtual debit cards each with their own limit and pulling from different accounts, created dozens of accounts (a “place to put money”) named by function (each of my household utilities gets its own account, with an automatic rule to pull in money whenever it gets paid out), and… well, I think that covers everything. This has given me unprecedented insight into my financial life. I know exactly how much I spend on groceries, on each utility, and on entertainment. I can project ahead and get a burn rate for my household. And my ex wife uses it too, on the same login, which is as easy as “make an account named with her first name” and a corresponding virtual debit card. I’m convinced the only reason people don’t use Mercury is that they don’t know what they’re missing. You have to pay for personal banking (a couple hundred a year iirc), but the business banking is free. If you want to try them out, you can start an LLC for a few dollars (at least in Missouri) and get overnight access to Mercury. All that’s required is your EIN. They’ve been one of the single best products I’ve ever used. The sole wrinkle was when they canceled all their existing virtual cards due to reasons, which threw my recurring billing into chaos. But every great company is allowed at least one mega annoyance, and that one was a blip. If you’re wondering whether to try them out, the answer is yes, and I’m excited for you to discover how cool it is. https://www.mercury.com https://www.mercury.com
- alchemist1e9 5mo ago
  > I’m convinced the only reason people don’t use Mercury is that they don’t know what they’re missing. Very well could be true because I had no idea who or what they are. Do they have strong low level automation support for the customer programmatically even for personal accounts? I use ledger for plaintext accounting for both personal and business and sync of data is slightly annoying, perhaps Mercury’s products solve that trivially?
  - sras-me 5mo ago
    > I use ledger for plaintext accounting for both personal and business and sync of data is slightly annoying.. I made this to solve it https://sras.me/accounts/ https://sras.me/accounts/ Feel free to use it as it stores data on your browser's local storage only. For syncing between devices, you would be able to use Google firebase's free tier and export your accounts (after compressing and encrypting) there and import from another device. Let me know if you want to try it..
Miles_Stone 5mo ago
[flagged]
yufiz 5mo ago
risky move, what is the talent pool for Haskell devs these days?
- wyager 5mo ago
  There are two countervailing effects when you choose a more theoretically advanced programming language. On the one hand, your hiring pool shrinks. On the other hand, the quality of the remaining hiring pool goes way up, which acts as an excellent recruiting filter (for both employer and employee). Jane Street made a similar play with OCaml.
  - reikonomusha 5mo ago
    The problem is that the intersection between your business's interests and the interests of the small pool of available developers is usually very small. Building banking apps? Well, even if it's Haskell, the Haskellers were dreaming of GPU compiler jobs, not banking front ends. So you're probably down to literally 5 qualified people on earth who want your job. But then 3 of those 5 don't want to relocate or have other operational desires that require you to re-think how you run your team, and 2 of those 5 believe so strongly in supply-demand that their salary should be 3x the industry average. Many companies, including Jane Street, come to the same conclusion: If you really want developers of a niche language, you have to be very good at finding smart people who don't know the language and training them.
- jkachmar 5mo ago
  not speaking in any official capacity, but: we great internal training material courtesy of some very thoughtful folks, and ultimately one hopes that most of the code is going to be pretty straightforward wherever possible.
- ufo 5mo ago
  The article has a section covering this. According to them, finding Haskell talent is not difficult, the bigger problem is onboarding them into the company coding style because Haskell developers come with strong opinions.
wyager 5mo ago
Mercury has been awesome, I've been using them for my business account for years and recently started using them for personal as well. I didn't know they used Haskell until well after I started using them, but it definitely tracks. The quality of their exposed software surface is at least a couple stddev above median.
cubefox 5mo ago
I know this is not the point of the article, but I find the anecdote in the beginning about null pointer errors somewhat ironic. Haskell's solution to null pointers are option types (`Maybe x` in Haskell), but these are known to be suboptimal. In languages with option types, if you want to weaken the type requirement for a function parameter, or strengthen the guarantee for a return type, you have to change the code at every call site. E.g, if you have a function which you can improve by changing - a parameter Foo to Option<Foo> or - a return value Option<Bar> to Bar you would have to change the code at all call sites. Which could be anything between annoying and practically impossible. In languages that solve null pointer errors instead with untagged union types (like TypeScript or Scala 3), this problem doesn't occur. So you can change - a parameter Foo to Foo | Null or - a return value Bar | Null to Bar and all call sites of the function can remain unchanged, since the type system knows that weakening the type requirement for a parameter, or strengthening the promise for a return type, is a safe change than can't cause a type error. So yes, option types do avoid null pointer exceptions, but they solve the issue in a very suboptimal way.
- wazHFsRy 5mo ago
  Mostly though if you do anything with the returned value at the call site you need to change that code anyways? If it is not just passing it on, and even then you might need to adapt its signatures. E.g. if you change from String | Null to String you remove the null handling. If you add Null you need to add Null handling?
  - cubefox 5mo ago
    No that's not right. If you were calling a function which might return null (String | Null), you will already have null handling at the call site, but if you now change that function such that it never returns null (String), you still have the (now unnecessary) null handling, but this doesn't hurt and you don't have to change anything at the call site. Likewise, if you were passing a String to a function that doesn't accept null (String), the call site already made sure that the parameter isn't null, and if you change the function so that it does now accept null (String | Null), again nothing needs to be changed at the call site.
djyde 5mo ago
Haskell is like an instrument I can never quite master, yet I find it utterly fascinating and keep trying to learn it without success.
MrBuddyCasino 5mo ago
Just wanted to say that rarely has a text make me want to work for a company as much as this one. This is the way engineering should be done. I have never worked in such an organisation.
xedrac 5mo ago
I loved working in Haskell for a few years. I wasn't actively looking it, but the opportunity just sort of landed in my lap. It was exciting and mentally stimulating. But the unfortunate fact is, I am easily twice as productive in Rust as I am Haskell, even after 3 years of nothing but Haskell. There are more pitfalls in Haskell that you have to just know how to avoid. It can be very difficult to digest as the language can be borderline write-only at times, depending on the author of the code. The tooling is often married to Nix, which is it's own complex beast. And it feels like language extensions are all over the place. Cabal files are not my favorite. And the compiler errors take some time to get get used to.
- django77 5mo ago
  Is the productivity 2x all across the board, or are there some parts that are less productive with Rust? Also, what do you mean by write-only?
  - qsera 5mo ago
    >what do you mean by write-only? I think they meant that in Haskell it is very easy to write externally unreadable code..
- Darmani 5mo ago
  Pretty surprising -- I had much the opposite experience. On our last product, we decided to start switching from Typescript to Rust on the backend because we got tired of crashes. I consider that to be one of the greatest technical mistakes I've made ever, as our productivity slowed massively. I'll just share two time-draining issues that only occur in Rust: (1) Writing higher-order functions (e.g.: a function to open a database connection, do something, and then close it -- yes, I know you can use RAII for this particular example), which is trivial in Haskell and TypeScript and JavaScript and C++ and PHP, turned out to be so impossible in Rust [even after asking Rust-expert friends for help], that I learned to just give up and never try, though it sometimes worked to write a macro instead. (2) It's happened many times that I would attempt a refactoring, spend all day fixing type errors, finally get to the top-level file, get a type error that's actually caused somewhere else by basic parts of the design, and conclude the entire refactoring I had attempted is impossible and need to revert everything. On top of that, Rust is the only modern language I can name where using a value by its interface instead of its concrete type lies somewhere between advanced and impossible, depending on what exactly you're doing. I came away concluding that application code (as opposed to systems or library code) should, to a first approximation, never be written in Rust.
isatty 5mo ago
I don't believe I'm the target market (I'm plenty happy with my small CU), and seeing their billboards makes me want to never use them BUT: seeing this post and their culture and that they use Haskell is kinda changing my mind.
- markdennis 5mo ago
  What’s wrong with their billboards?
  - isatty 5mo ago
    Typical SF garbage billboards with either “AI” or something they think is clever but isn’t. Mercury is pretty tame by comparison I’ll admit.
throw567643u8 5mo ago
[dead]
stardustrosalia 5mo ago
Hell yeah, I made a hyperbolic PDE solver in a bizarre constrained space in haskell! Unboxed vectors and performant algebriac systems out of functional programming is a blast.
tromp 5mo ago
A similar Haskell success story (from Bellroy) is the subject of an upcoming Melbourne Compose meeting: https://luma.com/uhdgct1v https://luma.com/uhdgct1v
KolmogorovComp 5mo ago
> [To lib authors] Nobody is obviously in charge in the way a fast-moving production team would mean "in charge," and that creates understandable hesitation around making breaking changes, even when experience has taught us better ways to design these systems. > This is not a complaint about volunteer maintainers. It is simply one of the ambient risks of building serious systems on a smaller ecosystem. And so instead of paying the lib authors who already have domain expertise and know their codebase, they chose to rewrite it from scratch/fork without contributing back. So classic.
- cinntaile 5mo ago
  Now you can develop the lib in the direction that you need and you have people on payroll that do it, this seems like good risk management.
- iand675 5mo ago
  Author here: I think you are projecting quite a bit. We do in fact hire a lot of people who maintain things, and even pay quite a lot for OSS development on things like the compiler and libraries we care about. But we still have business objectives to achieve, and sometimes it makes more sense to write things that better suit our needs.
shevy-java 5mo ago
A couple million lines of code? Try a better programming language next time, dagnabbit!!! (There will be downvotes I suppose. More lines of code the better?)
shawryadev 5mo ago
[dead]
scotty79 5mo ago
Will AI agents turn Haskell into a commercially viable language?
GRMPZ23 5mo ago
[flagged]
azan_ 5mo ago
Cool article, wish it wasn’t AI written/edited.
JasonHEIN 5mo ago
huh i love haskell soooooo muchhhh
neilv 5mo ago
I worked on a somewhat similar system in a fringe language (Scheme, and later, Racket) that got huge, but that remained manageable and high-velocity over a long period by a small team. We didn't create many bugs, and usually functionality could be added very rapidly (e.g., we were the first to achieve a certain certification for hosting sensitive data on AWS). Though occasionally functionality had to be added more slowly, because we had to write from scratch what would be an off-the-shelf component in a more popular platform. But once we did it, it worked, and we were back to our old velocity, and not slowed by the bloat and complexity of dozens of off-the-shelf frameworks. We could also adapt rapidly because we controlled a manageable platform, which is how we were able to move fast to AWS when there was a need. The system also had some technical bits of architectural secret sauce from the start (for complex data, and Web interaction), which enabled a lot of rapid development of functionality, and also set the tone for later empowering smartness. One difference with our system, from the Haskell fintech, was that our team size was very small (only 2-3 software engineers at a time, and someone who managed all the ops). So we didn't have the challenges of hundreds of people trying to coordinate and have a coherent system while getting their things done. Instead, there was usually one person doing more technical and architectural changes to the code, and a prolific other person doing huge amounts business logic functionality for complex processes. With careful use of current/near-term LLM-ish AI tools, software development might find some related efficiencies of very small and incredibly effective teams. But the model that comes to mind is having a small number of very sharp thinkers keeping things on an empowering and manageable path -- not churning massive bloat to knock off story points and letting sustainability be someone else's problem.
germandiago 5mo ago
I am currently reading Real-World Ocaml and I am really learning more about functional programming, though I was already familiar with a few things. Looks to me like you can build amazingly robust pieces of software with functional programming. However, I am divided. I have a backend that works in NiceGUI for a product. It does the job. The code is reasonable and MVVM. The most important task it does is connecting to a websocket per customer and consume data to present some analytics. I will not have a great deal of customers, maybe in the tens or maximum hundreds visiting the website. I also want REPL and/or hot reload, but I am aware that as I grow features (users admin panels, more analytics, etc) maybe functional programming can do a good job transforming data pipelines. But Haskell or Ocaml are static. I guess if I want something later that grows and scales and is still dynamic Clojure or Elixir should be a good choice. But at the same time I am afraid that if at some point I need to refactor, things will go wrong. Currently I use Python with Mypy. All is written in the backend: the frontend is generated by NiceGUI from the backend.
- spopejoy 5mo ago
  Not sure about Ocaml but with Haskell you can use ghci/`cabal repl` and get blazing fast reload of a web app as you develop. Tbh a lot of haskellers don't take advantage of this IMO.
  - germandiago 5mo ago
    Ocaml seems to have a REPL as well, not sure how it works outside of Emacs (in Emacs with utop looks good what I am trying). Haskell is so so correct that it tends to get a bit on the way and you tend to encode everything in the type system. This is a blessing for correctness and a curse for other stuff (tracing, debugging, adding side-effects). This is the reason why I am looking at Ocaml instead of Haskell: not so pure, more pragmatic and supports imperative programming well. As I said, it is double-edged.
jappgar 5mo ago
It's a double-edged sword. Two million lines is a major feat. It's also represents a significant maintenance burden. The advantages to Haskell are theoretically obvious. The downsides are harder to intuit. The temptation is to model _everything_ as types. The codebase itseld becomes a _business specification_, not an application. Every policy change is a major refactor (some of which are shockingly high-touch thanks to Haskell safety). The lesson is you cannot have your cake and eat it too. Eventually you become trapped by your types. Haskell is really impressive and powerful, perhaps especially at this scale. However it brings its own unique problems. The temptation to model business logic as types leads to rigid structures. And the safety these structures bring can blind you to other classes of risk.
- tylerchilds 5mo ago
  Typescript too: https://www.richard-towers.com/2023/03/11/typescripting-the-technical-interview.html https://www.richard-towers.com/2023/03/11/typescripting-the-... Tbvh the biggest downside of a Turing complete type system is that you can theoretically implement an application that compiles to dust.
- tauroid 5mo ago
  [dead]
- deleted 5mo ago
  [deleted]
- tikhonj 5mo ago
  You can do a great job of navigating that as long as you have some experienced engineers with taste building the core pieces. You can't have it all, but you can have a lot. I interned at Jane Street years ago and they seemed to do a great job of walking that line (in OCaml rather than Haskell, but same difference). They moved remarkably quickly despite working in an area with a lot of inherent complexity and where reliability and correctness are an existential concern to the business. (Which, perhaps surprisingly, is massively more the case for a trading firm than for a Mercury-like neobank...) In hindsight, a key thing Jane Street did was hire some experienced OCaml programmers with great taste (like Stephen Weeks, the author of MLton) and let them build the core libraries and guide the whole codebase from the beginning. Unfortunately, this is one of the things that Mercury didn't do anywhere near as well.
deleted 5mo ago
[deleted]
zerr 5mo ago
I once saw a real world Haskell code (from a huge investment bank) - the abundance of single letter variable names and short cryptic function names was striking.
- chuckadams 5mo ago
  Half the time the function names are operators, so you end up with an explosion of typographical symbols that makes perl look like cobol. Not my favorite aspect of the Haskell ecosystem.
  - lelele 5mo ago
    You forgot the 10 levels of precedence and the 3 associativity options for operators ;)
- twic 5mo ago
  A skilled programmer can write APL in any language.
GuB-42 5mo ago
The problem I have with functional programming is debugging. Or more precisely, I would say it is a strength of imperative programming, especially the procedural kind. In functional/declarative style, you generally describe how things should be, not how things are made, and you let the language piece everything together to get the expected result in the end. It is all well and good (and even better) if you did everything right, but what if you didn't and you don't get the expected result? How do you find the bug? In a language like C, it is relatively straightforward: go line by line, look at the execution state (the RAM, essentially) between each step and if it isn't as expected, something wrong must happen at that line, so you step in and progress like that. Harder to do when the language goes out of its way to hide the state from you, as it is the case for functional programming. It is interesting that the longest section of the article is about this problem: "design for introspection", where the author has to go out of his way to make his code debuggable. A good insight on the often overlooked practical use of Haskell.
- mrkeen 5mo ago
  My trick to debugging is to simply make every nontrivial piece of code return the same output for the same input. (The trivial pieces of code too!) No other (mainstream) language comes close. But what about situations where the code cannot be written in such a form (like shared memory concurrency)? I use transactions for that. No other (mainstream) language comes close. And that's without the low hanging fruit of no nulls, no implicit integer casts, etc. It is absolutely true that debugging Haskell code is harder than debugging other languages. If you took away the bottom 90% of footguns, how could it not be?
  - zelphirkalt 5mo ago
    Same output for same input is implicitly part of FP. (Not for OOP, due to mutation and side-effects.) I would think that when writing Haskell, one naturally always aims for same input same output.
- WorldMaker 5mo ago
  Functional Programming debugging is often "REPL-guided" in a way that imperative programming often is not. This is not unique to functional programming, though. Even the (mostly) imperative languages Python and Javascript you may be more likely to use REPLs of one sort or another (Python shells, browser consoles, Node/Deno/Bun shells, notebooks, etc.) as your first layer of debugging. There are interesting trade-offs in REPL-oriented debugging. One of the big things is that in a language like C you might often start first from whole program debugging and breakpoints to try to hit exactly where you think the problem is. In a REPL-oriented world you often try to build the components of your program in a way that you can test more units of it directly in the REPL. Your module/API/Type boundaries in a REPL world become to mirror your debuggability story. There is sometimes more pressure to get those right and easy to use than in imperative languages like C/C++ because you might want to reach for them directly in a REPL. But yes, a tradeoff versus whole program-first debugging is sometimes it becomes harder to isolate complex integration issues between your units in strange real world scenarios. However, that REPL-first approach is often encouraging of minimizing your integration "surface" to a bare minimum so often FP languages don't exhibit some of the same integration effects you see in imperative languages. > Harder to do when the language goes out of its way to hide the state from you, as it is the case for functional programming. Functional programming languages aren't really hiding any state from you. They also are running on imperative hardware and still dealing with real hardware states. At some point there is a translation between the "worlds" (which also likely aren't as different as you seem to think that they are). You still have those imperative breakpoints and imperative debuggers to fallback on. That's why the term is "REPL-guided" debugging. You can use a REPL to pinpoint the problematic unit (the exact module/API/function) and the problematic input giving you the surprise output. If you can't see the bug in the source as written you can still send it to an imperative debugger and watch nearly the same "line-by-line" experience and hope it provides additional missing context. Even better by that point you probably don't need to choose good "breakpoints" because you've already isolated the problem enough in the REPL to have "natural breakpoints" because the unit you are debugging may be small and narrow enough that stepping just that unit is all you need. > It is interesting that the longest section of the article is about this problem: "design for introspection", where the author has to go out of his way to make his code debuggable. I think you found the wrong message from that section. That section wasn't about debuggability it was about observability. It was about connecting logging/telemetry systems correctly, mocking fakes during testing, adding retries/circuit-breakers at a systematic/app-wide level rather than relying on individual libraries to get it right. In the imperative world these aren't debugging issues either: These are Dependency Injection issues. These are Middleware installing issues. These are factoring concerns like using Abstract Interfaces over Concrete Classes at your public API boundary. The design suggestions are factorings. They don't impact debuggability, they impact how easy it is to install observability middleware to someone else's public API.
parentheses 5mo ago
IMO types are the main lever you can use other than procedural abstraction. I feel that Haskell gives you both in a way that marries them for maximum constraint-building. Constraints that prevent illogical or illegal programs are the bread and butter of reliable software.
wbsun 5mo ago
> We serve over 300,000 businesses. We processed $248 billion in transaction volume in 2025 on $650 million in annualized revenue > The system we've built has worked well for years, through hypergrowth, through the SVB crisis that sent $2 billion in new deposits our way in five day. This is a strange way to describe a transaction processing system with total amount of money it processed. The reliability or scalability is not measured as O(n) dollars. In theory, $248bln or $2bln can be done in one transaction although I know it is not the case in reality. It would be impressive to see the typical system design properties like transactions per second, latency distribution, etc.
sriku 5mo ago
I wish people'd report metrics like "40 type classes, 5000 functions, 300 instances, 20 monads, 14 functors, ..." And such instead of "lines of haskell".
- wbsun 5mo ago
  I wish people'd report metrics like "transactions per second, 99.x% availability with N secs/mins p99 latency, ..." when describing how practically useful and how effective a real-world banking/transaction system is, which further proves the practicality of the programming language.
  - sriku 5mo ago
    Those inseparably mix up programming language, compiler, programmer(s) and application domain as factors.
edmondx 5mo ago
Great article. I like how it presents a realistic view without hype, showing both the benefits and the tradeoffs.

435 points by unignorant 5mo ago

lots of really great pieces of advice in this

maz1b 5mo ago

I think perhaps contrary to popular belief, Mercury choosing Haskell and their early leadership having such a storied experience in it probably played some non-insignificant role in their success. As a customer of Mercury, it's truly one of the critical companies my toolkit, and I just can't help but feel that their choosing of Haskell made their progress, development and overall journey that much better. I realize that you can make this argument with most languages, and it's not to say that a FP lang like Haskell is a recipe for success, but this intentional decision particularly pre "vibe coding" and the LLM era seems particularly prescient, of course combined with their engineering culture that was detailed in the post.

1024bits 5mo ago
I'd also wager that hiring generalists with no prior experience in the language actually helped them, because they got to instill their culture and style from the ground up with their new hires. Pre vibe-coding, most of those people would'nt have wanted to just jump in and hack away with zero instruction.
ipnon 5mo ago
I have noticed that everything in their app Just Works. It's very satisfying coming from other services!
- jwsteigerwalt 5mo ago
  I feel the same way. I only started using Mercury about 6 months ago and I’m continually impressed that it just makes sense.
spopejoy 5mo ago
I would counter that it was probably their startup-oriented fintech focus and execution that led to their success. I love good tech culture as much as the next HNer but I've seen companies with great tech die because of bad biz focus. I might further argue that the startup-y fintech culture led to good tech culture. The fact that they didn't start as a bank (as opposed to say SVB) means that they didn't have to be as conservative, or integrate with some horrific ancient tech stack. I'm pleased they've had such success with Haskell, but much like Jane Street and OCAML, I think the language choice is almost accidental*, as much as the companies would like you to believe otherwise. I would like to know however what they're doing for front-end. I would guess that all of this Haskell is back-end only. *EDIT by "accidental" I mean to the business side. Jane St had some good trades, Mercury had great focus and execution. They also have some good tech :)

dnnddidiej 5mo ago

I think you have to get a Haskell job early in career and stick to Haskell jobs. Breaking in is really hard as you come without experience there will be plenty of others with Haskell experience to compete. And because the jobs are rare if it doesnt work out (company becomes bad to work for or layoff) you can be unstuck (or I guess you would switch to Rust, Scala or F#)

matt-noonan 5mo ago
As somebody who has helped hire many Haskell devs, I can say that lots of Haskell experience isn't always a positive. We have to filter carefully to make sure that we end up with developers who want to build real things, not developers who just want to get paid for noodling around with Haskell. As far as I'm concerned, I'd much rather hire somebody with lots of experience building things who ended up coming to Haskell later because they viscerally understand the benefits and risks. Somebody with lots and lots of Haskell experience who never delivered much is a big risk.
- dnnddidiej 5mo ago
  Interesting, I guess it then depends on the company (or recruiter) then.
- _qhqq 5mo ago
  haha i've abused this recruiting mindset for a decade it's so easy to scout when a company has this haskell philosophy. either by the interviewers themselves or by the bloggers they hired to guide their team. the trick? i just..lie. "oh yeah i'm super pragmatic. i'm not hardline about haskell. i don't think you should be fancy." see how easy it is? i am suddenly hired and got a fat raise. and if the company moves off haskell? i quit immediately, get another haskell job, and talk to my former coworkers on the way out to embolden them to do the same. it helps that i have the "real world" stuff on my resume. i rode the 2010s job hopping ride as a haskeller doing this. each time a 20-30% raise. and i get to still write haskell. and i am always a top percentile haskeller at the company so i can code however tf i want lolol. suddenly - singletons, Generics, HKD! so here's to earning another million bucks "noodling around with Haskell" :cheers:

threethirtytwo 5mo ago

I really believe in FP and Haskell but I want to examine this objectively. Empirically speaking is what Mercury done successful truly because of Haskell? Do they have metrics that demonstrates clear superiority along some vectored trait like complexity, bug count, etc? >A couple million lines of Haskell, maintained by people who learned the language on the job, at a company that moves huge amounts of money? The conventional wisdom says this should be a disaster, but surprisingly, it isn't. The system we've built has worked well for years, through hypergrowth, through the SVB crisis that sent $2 billion in new deposits our way in five days,1 through regulatory examinations, through all the ordinary and extraordinary things that happen to a financial system at scale. This one is quite telling. Do people have counter examples?

cfiggers 5mo ago
Without having run the whole company twice in parallel, once using Haskell and again in some other language, and without having measured both runs exactly the same way, I don't think metrics like you're interested in could possibly have sufficient context to mean anything reliable. Obviously Mercury is successful, and obviously Haskell is how they did it. So it's essential to their success. Would it be instrumental to anyone else's anywhere else doing anything else? Can't possibly know, I don't think.
- threethirtytwo 5mo ago
  I’m asking for solutions and answers. Yeah. I’m aware of how hard it is to get metrics. You can still compare lines of code and bug rate over the same period of time.
  - IIsi50MHz 5mo ago
    You can, but then "The cake is a lie.", because linecount and bug rate, when concieved as proxies for productivity[1] or quality rarely match up with reality in a way that allows you to make predictions or reason about past outcomes. You can reason about frequency of particular types bugs, such as null pointers or overflow, or whether those bugs can occur at all. [1] https://www.folklore.org/Negative_2000_Lines_Of_Code.html https://www.folklore.org/Negative_2000_Lines_Of_Code.html

le-mark 5mo ago

It’s hard to imagine what two millions lines of Haskell could possibly be doing. I mean that’s a lot of code and I have the impression that Haskell is “tight” meaning a little code can do a lot. Maybe they have a lot of libraries to do things like json serializing/deserializing, rest api frameworks, logging etc?

imoverclocked 5mo ago
From TFA: > The problem is that we cannot trust code we cannot instrument. If a third-party binding makes HTTP calls through concrete functions, we have no way to add tracing, no way to inject timeouts tuned to our SLOs, no way to simulate partner outages in testing, and no way to explain the 400ms gap in a trace except by squinting at it and developing theories. So we write our own. More work upfront, but the clients we write are observable by construction, because we built them that way from the start.
- troupo 5mo ago
  > If a third-party binding makes HTTP calls through concrete functions, we have no way to add tracing, no way to inject timeouts tuned to our SLOs, no way to simulate partner outages in testing, and no way to explain the 400ms gap in a trace Given that tracing etc. is IO, are they just threading IO through the entirety of all their Haskell code?
  - tome 5mo ago
    Tracing doesn’t actually require IO, only emitting the traces does, and those two need not be done at the same point. In any case, anywhere they’re doing HTTP calls they are already threading IO, so they don’t have to pay an additional cost.
verandaguy 5mo ago
Nit: the quality of a language that you call "tight" is usually called "expressive." You can use few characters to express a relatively very abstract idea. Some people call this "high-level," too. I will say, though, that 2 million lines of code is much less code than it sounds like at first glance, especially for a company in a highly-regulated space like finance, plus a few years of progress.

bri3d 5mo ago

> Haskell gives you tools to encode these incantations in types so they cannot be forgotten. This is, for my money, the single most valuable thing the language offers a production engineering organization. Haskell is admittedly, probably the most powerful widely (or even somewhat widely) used language for doing this, but this general pattern works really well in Rust and TypeScript too and is one of my very favorite tools for writing better code. I also really like doing things like User -> LoggedInUser -> AccessControlledLoggedInUser to prevent the kind of really obvious AuthZ bugs people make in web applications time and time again. I've found this pattern to be massively underutilized in industry.

miki123211 5mo ago
This isn't specific to Rust or Typescript. You can do this in basically any language. Imagine you have to distinguish between unescaped and escaped strings for security purposes. Even with a dynamically typed language, you can keep escaped strings as an Escaped class, with escape(str)->Escaped and dangerouslyAssumeEscaped(str)->Escaped functions (or static methods). There's a performance cost to this, so that's a tradeoff you have to weigh, but it is possible. Another way of doing this is Application Hungarian[1], though that relies on the programmer more than it does on the compiler. [1] https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...
- wyager 5mo ago
  > You can do this in basically any language. You can do it in Assembly. That doesn't mean it's cost effective.
  - myst 5mo ago
    Costs are a skill issue ;-)
    - intrinsicallee 5mo ago
      You demonstrate well the problem: yes anything that is computable can be than in any computation system. That's not what discussions about tooling are about. If a tool can help enforce some ways of doing things, or if it doesn't constrain people much, that has consequences for the type of work that gets done with them and the systems you encounter running out there that you might be invited or find the need to work with. "I can do it" is exactly the wrong answer. "How can I guarantee that others will do it" is the point being made.

faangguyindia 5mo ago

I use Haskell a lot, but I notice that it's very hard to cross-compile it. If only cross-compilation became easy so that I can develop on my chip Macs and deploy on x64/AMD Linux servers. >statically linking Haskell binaries is quite a challenge >build requirements really slow down the process. I have to use dockers to help cache dependencies and avoid recompiling things that have not changed, but it is still slow and puts out large binaries. Also, the Docker-based deployment takes a lot of time as it needs to recompile each module. While you can cache some part of it, it's still slow. Meanwhile with Go it's painless. And i am not the only one having this issue: https://news.ycombinator.com/item?id=47957624#47972671 https://news.ycombinator.com/item?id=47957624#47972671 Such a shame Haskell is beautiful and performant language still build is slow.

amitbidlan 5mo ago

[flagged]

golem14 5mo ago

I'm not particularly into Haskell or functional languages, but I came here to say it warms my heart to hear that people in finance actually embrace it (also J/APL). It seems like a good choice for banking. The Mercury site also looks way better than most other banks I have ever used (load speed is also very good.) On the danger of seeming like a shill (I'm not), I'm tempted to try them out.

sillysaurusx 5mo ago
Do it. You won’t be disappointed. I’ve been using them for about 5 years now.

thot_experiment 5mo ago

My bestie works at this company and looking from the outside they have a good engineering culture. I do think Haskell is the right tool for the job, and they are playing to it's strengths, but part of me wonders if a lot of their success is attributable to the place just being well run in general.

ironmagma 5mo ago
That would not run counter to the popular (whether true or not) idea that by using functional programming languages you filter for a higher quality labor pool / applicant pool.
- runevault 5mo ago
  The version I've always heard is just well designed but less popular languages, but the ones I can think of were all functional (Haskell/F#/OCaml/Clojure/Elm/Erlang)
- spopejoy 5mo ago
  That wouldn't apply here, since as the article says they hire "generalists, and most of them have never written a line of Haskell before joining." In any case, I think the "Haskell tax" concept (where you can pay well-paid programmers less if you have a Haskell shop) is stale by now. Rust attracted away a lot of FP-ers, plus mainstream langs like C++, Java and even Typescript got smarter. Haskell's biggest problem by far is the tiny labor pool, which Mercury seems to wisely avoid.
  - sn9 5mo ago
    The post explicitly makes the case for the filtering playing a role. Ctrl-F "Python".
meken 5mo ago
> but part of me wonders if a lot of their success is attributable to the place just being well run in general That was my sense reading the article - that the author would be running a successful engineering org using any language really.

hmokiguess 5mo ago

> written using the mosaic theory of information and a range of journalistic tools what does that mean?

sillysaurusx 5mo ago

I just wanted to leave a glowing comment about Mercury itself, since this is one of the few times I’ll be able to. I’ve been using Mercury for 5 years. In that time, I’ve been able to wire transfer money without having to worry it might disappear (functionally impossible at certain other banks), created hundreds of virtual debit cards each with their own limit and pulling from different accounts, created dozens of accounts (a “place to put money”) named by function (each of my household utilities gets its own account, with an automatic rule to pull in money whenever it gets paid out), and… well, I think that covers everything. This has given me unprecedented insight into my financial life. I know exactly how much I spend on groceries, on each utility, and on entertainment. I can project ahead and get a burn rate for my household. And my ex wife uses it too, on the same login, which is as easy as “make an account named with her first name” and a corresponding virtual debit card. I’m convinced the only reason people don’t use Mercury is that they don’t know what they’re missing. You have to pay for personal banking (a couple hundred a year iirc), but the business banking is free. If you want to try them out, you can start an LLC for a few dollars (at least in Missouri) and get overnight access to Mercury. All that’s required is your EIN. They’ve been one of the single best products I’ve ever used. The sole wrinkle was when they canceled all their existing virtual cards due to reasons, which threw my recurring billing into chaos. But every great company is allowed at least one mega annoyance, and that one was a blip. If you’re wondering whether to try them out, the answer is yes, and I’m excited for you to discover how cool it is. https://www.mercury.com https://www.mercury.com

alchemist1e9 5mo ago
> I’m convinced the only reason people don’t use Mercury is that they don’t know what they’re missing. Very well could be true because I had no idea who or what they are. Do they have strong low level automation support for the customer programmatically even for personal accounts? I use ledger for plaintext accounting for both personal and business and sync of data is slightly annoying, perhaps Mercury’s products solve that trivially?
- sras-me 5mo ago
  > I use ledger for plaintext accounting for both personal and business and sync of data is slightly annoying.. I made this to solve it https://sras.me/accounts/ https://sras.me/accounts/ Feel free to use it as it stores data on your browser's local storage only. For syncing between devices, you would be able to use Google firebase's free tier and export your accounts (after compressing and encrypting) there and import from another device. Let me know if you want to try it..

Miles_Stone 5mo ago

[flagged]

yufiz 5mo ago

risky move, what is the talent pool for Haskell devs these days?

wyager 5mo ago
There are two countervailing effects when you choose a more theoretically advanced programming language. On the one hand, your hiring pool shrinks. On the other hand, the quality of the remaining hiring pool goes way up, which acts as an excellent recruiting filter (for both employer and employee). Jane Street made a similar play with OCaml.
- reikonomusha 5mo ago
  The problem is that the intersection between your business's interests and the interests of the small pool of available developers is usually very small. Building banking apps? Well, even if it's Haskell, the Haskellers were dreaming of GPU compiler jobs, not banking front ends. So you're probably down to literally 5 qualified people on earth who want your job. But then 3 of those 5 don't want to relocate or have other operational desires that require you to re-think how you run your team, and 2 of those 5 believe so strongly in supply-demand that their salary should be 3x the industry average. Many companies, including Jane Street, come to the same conclusion: If you really want developers of a niche language, you have to be very good at finding smart people who don't know the language and training them.
jkachmar 5mo ago
not speaking in any official capacity, but: we great internal training material courtesy of some very thoughtful folks, and ultimately one hopes that most of the code is going to be pretty straightforward wherever possible.
ufo 5mo ago
The article has a section covering this. According to them, finding Haskell talent is not difficult, the bigger problem is onboarding them into the company coding style because Haskell developers come with strong opinions.

wyager 5mo ago

Mercury has been awesome, I've been using them for my business account for years and recently started using them for personal as well. I didn't know they used Haskell until well after I started using them, but it definitely tracks. The quality of their exposed software surface is at least a couple stddev above median.

cubefox 5mo ago

I know this is not the point of the article, but I find the anecdote in the beginning about null pointer errors somewhat ironic. Haskell's solution to null pointers are option types (`Maybe x` in Haskell), but these are known to be suboptimal. In languages with option types, if you want to weaken the type requirement for a function parameter, or strengthen the guarantee for a return type, you have to change the code at every call site. E.g, if you have a function which you can improve by changing - a parameter Foo to Option<Foo> or - a return value Option<Bar> to Bar you would have to change the code at all call sites. Which could be anything between annoying and practically impossible. In languages that solve null pointer errors instead with untagged union types (like TypeScript or Scala 3), this problem doesn't occur. So you can change - a parameter Foo to Foo | Null or - a return value Bar | Null to Bar and all call sites of the function can remain unchanged, since the type system knows that weakening the type requirement for a parameter, or strengthening the promise for a return type, is a safe change than can't cause a type error. So yes, option types do avoid null pointer exceptions, but they solve the issue in a very suboptimal way.

wazHFsRy 5mo ago
Mostly though if you do anything with the returned value at the call site you need to change that code anyways? If it is not just passing it on, and even then you might need to adapt its signatures. E.g. if you change from String | Null to String you remove the null handling. If you add Null you need to add Null handling?
- cubefox 5mo ago
  No that's not right. If you were calling a function which might return null (String | Null), you will already have null handling at the call site, but if you now change that function such that it never returns null (String), you still have the (now unnecessary) null handling, but this doesn't hurt and you don't have to change anything at the call site. Likewise, if you were passing a String to a function that doesn't accept null (String), the call site already made sure that the parameter isn't null, and if you change the function so that it does now accept null (String | Null), again nothing needs to be changed at the call site.

djyde 5mo ago

Haskell is like an instrument I can never quite master, yet I find it utterly fascinating and keep trying to learn it without success.

MrBuddyCasino 5mo ago

Just wanted to say that rarely has a text make me want to work for a company as much as this one. This is the way engineering should be done. I have never worked in such an organisation.

xedrac 5mo ago

I loved working in Haskell for a few years. I wasn't actively looking it, but the opportunity just sort of landed in my lap. It was exciting and mentally stimulating. But the unfortunate fact is, I am easily twice as productive in Rust as I am Haskell, even after 3 years of nothing but Haskell. There are more pitfalls in Haskell that you have to just know how to avoid. It can be very difficult to digest as the language can be borderline write-only at times, depending on the author of the code. The tooling is often married to Nix, which is it's own complex beast. And it feels like language extensions are all over the place. Cabal files are not my favorite. And the compiler errors take some time to get get used to.

django77 5mo ago
Is the productivity 2x all across the board, or are there some parts that are less productive with Rust? Also, what do you mean by write-only?
- qsera 5mo ago
  >what do you mean by write-only? I think they meant that in Haskell it is very easy to write externally unreadable code..
Darmani 5mo ago
Pretty surprising -- I had much the opposite experience. On our last product, we decided to start switching from Typescript to Rust on the backend because we got tired of crashes. I consider that to be one of the greatest technical mistakes I've made ever, as our productivity slowed massively. I'll just share two time-draining issues that only occur in Rust: (1) Writing higher-order functions (e.g.: a function to open a database connection, do something, and then close it -- yes, I know you can use RAII for this particular example), which is trivial in Haskell and TypeScript and JavaScript and C++ and PHP, turned out to be so impossible in Rust [even after asking Rust-expert friends for help], that I learned to just give up and never try, though it sometimes worked to write a macro instead. (2) It's happened many times that I would attempt a refactoring, spend all day fixing type errors, finally get to the top-level file, get a type error that's actually caused somewhere else by basic parts of the design, and conclude the entire refactoring I had attempted is impossible and need to revert everything. On top of that, Rust is the only modern language I can name where using a value by its interface instead of its concrete type lies somewhere between advanced and impossible, depending on what exactly you're doing. I came away concluding that application code (as opposed to systems or library code) should, to a first approximation, never be written in Rust.

isatty 5mo ago

I don't believe I'm the target market (I'm plenty happy with my small CU), and seeing their billboards makes me want to never use them BUT: seeing this post and their culture and that they use Haskell is kinda changing my mind.

markdennis 5mo ago
What’s wrong with their billboards?
- isatty 5mo ago
  Typical SF garbage billboards with either “AI” or something they think is clever but isn’t. Mercury is pretty tame by comparison I’ll admit.

throw567643u8 5mo ago

[dead]

stardustrosalia 5mo ago

Hell yeah, I made a hyperbolic PDE solver in a bizarre constrained space in haskell! Unboxed vectors and performant algebriac systems out of functional programming is a blast.

tromp 5mo ago

A similar Haskell success story (from Bellroy) is the subject of an upcoming Melbourne Compose meeting: https://luma.com/uhdgct1v https://luma.com/uhdgct1v

KolmogorovComp 5mo ago

> [To lib authors] Nobody is obviously in charge in the way a fast-moving production team would mean "in charge," and that creates understandable hesitation around making breaking changes, even when experience has taught us better ways to design these systems. > This is not a complaint about volunteer maintainers. It is simply one of the ambient risks of building serious systems on a smaller ecosystem. And so instead of paying the lib authors who already have domain expertise and know their codebase, they chose to rewrite it from scratch/fork without contributing back. So classic.

cinntaile 5mo ago
Now you can develop the lib in the direction that you need and you have people on payroll that do it, this seems like good risk management.
iand675 5mo ago
Author here: I think you are projecting quite a bit. We do in fact hire a lot of people who maintain things, and even pay quite a lot for OSS development on things like the compiler and libraries we care about. But we still have business objectives to achieve, and sometimes it makes more sense to write things that better suit our needs.

shevy-java 5mo ago

A couple million lines of code? Try a better programming language next time, dagnabbit!!! (There will be downvotes I suppose. More lines of code the better?)

shawryadev 5mo ago

[dead]

scotty79 5mo ago

Will AI agents turn Haskell into a commercially viable language?

GRMPZ23 5mo ago

[flagged]

azan_ 5mo ago

Cool article, wish it wasn’t AI written/edited.

JasonHEIN 5mo ago

huh i love haskell soooooo muchhhh

neilv 5mo ago

I worked on a somewhat similar system in a fringe language (Scheme, and later, Racket) that got huge, but that remained manageable and high-velocity over a long period by a small team. We didn't create many bugs, and usually functionality could be added very rapidly (e.g., we were the first to achieve a certain certification for hosting sensitive data on AWS). Though occasionally functionality had to be added more slowly, because we had to write from scratch what would be an off-the-shelf component in a more popular platform. But once we did it, it worked, and we were back to our old velocity, and not slowed by the bloat and complexity of dozens of off-the-shelf frameworks. We could also adapt rapidly because we controlled a manageable platform, which is how we were able to move fast to AWS when there was a need. The system also had some technical bits of architectural secret sauce from the start (for complex data, and Web interaction), which enabled a lot of rapid development of functionality, and also set the tone for later empowering smartness. One difference with our system, from the Haskell fintech, was that our team size was very small (only 2-3 software engineers at a time, and someone who managed all the ops). So we didn't have the challenges of hundreds of people trying to coordinate and have a coherent system while getting their things done. Instead, there was usually one person doing more technical and architectural changes to the code, and a prolific other person doing huge amounts business logic functionality for complex processes. With careful use of current/near-term LLM-ish AI tools, software development might find some related efficiencies of very small and incredibly effective teams. But the model that comes to mind is having a small number of very sharp thinkers keeping things on an empowering and manageable path -- not churning massive bloat to knock off story points and letting sustainability be someone else's problem.

germandiago 5mo ago

I am currently reading Real-World Ocaml and I am really learning more about functional programming, though I was already familiar with a few things. Looks to me like you can build amazingly robust pieces of software with functional programming. However, I am divided. I have a backend that works in NiceGUI for a product. It does the job. The code is reasonable and MVVM. The most important task it does is connecting to a websocket per customer and consume data to present some analytics. I will not have a great deal of customers, maybe in the tens or maximum hundreds visiting the website. I also want REPL and/or hot reload, but I am aware that as I grow features (users admin panels, more analytics, etc) maybe functional programming can do a good job transforming data pipelines. But Haskell or Ocaml are static. I guess if I want something later that grows and scales and is still dynamic Clojure or Elixir should be a good choice. But at the same time I am afraid that if at some point I need to refactor, things will go wrong. Currently I use Python with Mypy. All is written in the backend: the frontend is generated by NiceGUI from the backend.

spopejoy 5mo ago
Not sure about Ocaml but with Haskell you can use ghci/`cabal repl` and get blazing fast reload of a web app as you develop. Tbh a lot of haskellers don't take advantage of this IMO.
- germandiago 5mo ago
  Ocaml seems to have a REPL as well, not sure how it works outside of Emacs (in Emacs with utop looks good what I am trying). Haskell is so so correct that it tends to get a bit on the way and you tend to encode everything in the type system. This is a blessing for correctness and a curse for other stuff (tracing, debugging, adding side-effects). This is the reason why I am looking at Ocaml instead of Haskell: not so pure, more pragmatic and supports imperative programming well. As I said, it is double-edged.

jappgar 5mo ago

It's a double-edged sword. Two million lines is a major feat. It's also represents a significant maintenance burden. The advantages to Haskell are theoretically obvious. The downsides are harder to intuit. The temptation is to model _everything_ as types. The codebase itseld becomes a _business specification_, not an application. Every policy change is a major refactor (some of which are shockingly high-touch thanks to Haskell safety). The lesson is you cannot have your cake and eat it too. Eventually you become trapped by your types. Haskell is really impressive and powerful, perhaps especially at this scale. However it brings its own unique problems. The temptation to model business logic as types leads to rigid structures. And the safety these structures bring can blind you to other classes of risk.

tylerchilds 5mo ago
Typescript too: https://www.richard-towers.com/2023/03/11/typescripting-the-technical-interview.html https://www.richard-towers.com/2023/03/11/typescripting-the-... Tbvh the biggest downside of a Turing complete type system is that you can theoretically implement an application that compiles to dust.
tauroid 5mo ago
[dead]
deleted 5mo ago
[deleted]
tikhonj 5mo ago
You can do a great job of navigating that as long as you have some experienced engineers with taste building the core pieces. You can't have it all, but you can have a lot. I interned at Jane Street years ago and they seemed to do a great job of walking that line (in OCaml rather than Haskell, but same difference). They moved remarkably quickly despite working in an area with a lot of inherent complexity and where reliability and correctness are an existential concern to the business. (Which, perhaps surprisingly, is massively more the case for a trading firm than for a Mercury-like neobank...) In hindsight, a key thing Jane Street did was hire some experienced OCaml programmers with great taste (like Stephen Weeks, the author of MLton) and let them build the core libraries and guide the whole codebase from the beginning. Unfortunately, this is one of the things that Mercury didn't do anywhere near as well.

deleted 5mo ago

[deleted]

zerr 5mo ago

I once saw a real world Haskell code (from a huge investment bank) - the abundance of single letter variable names and short cryptic function names was striking.

chuckadams 5mo ago
Half the time the function names are operators, so you end up with an explosion of typographical symbols that makes perl look like cobol. Not my favorite aspect of the Haskell ecosystem.
- lelele 5mo ago
  You forgot the 10 levels of precedence and the 3 associativity options for operators ;)
twic 5mo ago
A skilled programmer can write APL in any language.

GuB-42 5mo ago

The problem I have with functional programming is debugging. Or more precisely, I would say it is a strength of imperative programming, especially the procedural kind. In functional/declarative style, you generally describe how things should be, not how things are made, and you let the language piece everything together to get the expected result in the end. It is all well and good (and even better) if you did everything right, but what if you didn't and you don't get the expected result? How do you find the bug? In a language like C, it is relatively straightforward: go line by line, look at the execution state (the RAM, essentially) between each step and if it isn't as expected, something wrong must happen at that line, so you step in and progress like that. Harder to do when the language goes out of its way to hide the state from you, as it is the case for functional programming. It is interesting that the longest section of the article is about this problem: "design for introspection", where the author has to go out of his way to make his code debuggable. A good insight on the often overlooked practical use of Haskell.

mrkeen 5mo ago
My trick to debugging is to simply make every nontrivial piece of code return the same output for the same input. (The trivial pieces of code too!) No other (mainstream) language comes close. But what about situations where the code cannot be written in such a form (like shared memory concurrency)? I use transactions for that. No other (mainstream) language comes close. And that's without the low hanging fruit of no nulls, no implicit integer casts, etc. It is absolutely true that debugging Haskell code is harder than debugging other languages. If you took away the bottom 90% of footguns, how could it not be?
- zelphirkalt 5mo ago
  Same output for same input is implicitly part of FP. (Not for OOP, due to mutation and side-effects.) I would think that when writing Haskell, one naturally always aims for same input same output.
WorldMaker 5mo ago
Functional Programming debugging is often "REPL-guided" in a way that imperative programming often is not. This is not unique to functional programming, though. Even the (mostly) imperative languages Python and Javascript you may be more likely to use REPLs of one sort or another (Python shells, browser consoles, Node/Deno/Bun shells, notebooks, etc.) as your first layer of debugging. There are interesting trade-offs in REPL-oriented debugging. One of the big things is that in a language like C you might often start first from whole program debugging and breakpoints to try to hit exactly where you think the problem is. In a REPL-oriented world you often try to build the components of your program in a way that you can test more units of it directly in the REPL. Your module/API/Type boundaries in a REPL world become to mirror your debuggability story. There is sometimes more pressure to get those right and easy to use than in imperative languages like C/C++ because you might want to reach for them directly in a REPL. But yes, a tradeoff versus whole program-first debugging is sometimes it becomes harder to isolate complex integration issues between your units in strange real world scenarios. However, that REPL-first approach is often encouraging of minimizing your integration "surface" to a bare minimum so often FP languages don't exhibit some of the same integration effects you see in imperative languages. > Harder to do when the language goes out of its way to hide the state from you, as it is the case for functional programming. Functional programming languages aren't really hiding any state from you. They also are running on imperative hardware and still dealing with real hardware states. At some point there is a translation between the "worlds" (which also likely aren't as different as you seem to think that they are). You still have those imperative breakpoints and imperative debuggers to fallback on. That's why the term is "REPL-guided" debugging. You can use a REPL to pinpoint the problematic unit (the exact module/API/function) and the problematic input giving you the surprise output. If you can't see the bug in the source as written you can still send it to an imperative debugger and watch nearly the same "line-by-line" experience and hope it provides additional missing context. Even better by that point you probably don't need to choose good "breakpoints" because you've already isolated the problem enough in the REPL to have "natural breakpoints" because the unit you are debugging may be small and narrow enough that stepping just that unit is all you need. > It is interesting that the longest section of the article is about this problem: "design for introspection", where the author has to go out of his way to make his code debuggable. I think you found the wrong message from that section. That section wasn't about debuggability it was about observability. It was about connecting logging/telemetry systems correctly, mocking fakes during testing, adding retries/circuit-breakers at a systematic/app-wide level rather than relying on individual libraries to get it right. In the imperative world these aren't debugging issues either: These are Dependency Injection issues. These are Middleware installing issues. These are factoring concerns like using Abstract Interfaces over Concrete Classes at your public API boundary. The design suggestions are factorings. They don't impact debuggability, they impact how easy it is to install observability middleware to someone else's public API.

parentheses 5mo ago

IMO types are the main lever you can use other than procedural abstraction. I feel that Haskell gives you both in a way that marries them for maximum constraint-building. Constraints that prevent illogical or illegal programs are the bread and butter of reliable software.

wbsun 5mo ago

> We serve over 300,000 businesses. We processed $248 billion in transaction volume in 2025 on $650 million in annualized revenue > The system we've built has worked well for years, through hypergrowth, through the SVB crisis that sent $2 billion in new deposits our way in five day. This is a strange way to describe a transaction processing system with total amount of money it processed. The reliability or scalability is not measured as O(n) dollars. In theory, $248bln or $2bln can be done in one transaction although I know it is not the case in reality. It would be impressive to see the typical system design properties like transactions per second, latency distribution, etc.

sriku 5mo ago

I wish people'd report metrics like "40 type classes, 5000 functions, 300 instances, 20 monads, 14 functors, ..." And such instead of "lines of haskell".

wbsun 5mo ago
I wish people'd report metrics like "transactions per second, 99.x% availability with N secs/mins p99 latency, ..." when describing how practically useful and how effective a real-world banking/transaction system is, which further proves the practicality of the programming language.
- sriku 5mo ago
  Those inseparably mix up programming language, compiler, programmer(s) and application domain as factors.

edmondx 5mo ago

Great article. I like how it presents a realistic view without hype, showing both the benefits and the tradeoffs.