27 ms·

Haskell in Production: Standard Chartered

186 points by NaeosPsy 3y ago

YetAnotherNick 3y ago
> Mu can easily take several minutes to compile a single large application. We will circumvent those problems by switching our front-end to use GHC directly What does he mean by this?
- Vosporos 3y ago
  From what I understand: GHC takes care of the parsing of the source file, error reporting, etc. Then the AST (or an intermediate representation) is passed to the of the Mu compiler, which performs whole-program optimisations.
  - nequo 3y ago
    Gergo Erdi gave a talk on the GHC efforts of Standard Chartered last year: https://www.youtube.com/watch?v=fZ66Pz7015Q https://www.youtube.com/watch?v=fZ66Pz7015Q
- wathef 3y ago
  I may be wrong, but if I remember correctly Standard Charter uses their own “dialect” called Mu[1]. 1 - https://icfp21.sigplan.org/details/hiw-2021-papers/14/Haskell-reinterpreted-large-scale-real-world-experience-with-the-Mu-compiler-in-Fin https://icfp21.sigplan.org/details/hiw-2021-papers/14/Haskel...
  - deleted 3y ago
    [deleted]
- grumpyprole 3y ago
  He probably means that GHC will parse and type check faster. Switching the front-end doesn't sound like it will circumvent the whole program nature of their compilation. But I'm just guessing.
- misja111 3y ago
  I'm also interested. Just guessing, but I guess it means that while they use their own (slow compiling) 'Mu' dialect for most applications, they have switched to using 'standard' GHC for their front-end apps.
  - tuukkah 3y ago
    Frontend here refers to the first stage of the three-stage structure of compilers, ie. the functionalities such as parsing and type checking that come before the optimization and target code generation phases: https://en.wikipedia.org/wiki/Compiler#Three-stage_compiler_structure https://en.wikipedia.org/wiki/Compiler#Three-stage_compiler_...
    - misja111 3y ago
      Didn't know that, thanks! But then I'm wondering how they can do parsing and type checking with GHC when they are using their own dialect, "Mu". How come GHC can parse that?
      - nequo 3y ago
        The differences between Mu and Haskell are not that large. They use GHC as a library in their front end. Gergo Erdi gave a talk on it last year: https://news.ycombinator.com/item?id=35801844 https://news.ycombinator.com/item?id=35801844
      - tuukkah 3y ago
        GHC has a system for language extensions so they have developed a few proprietary ones for Mu. For the standard ones, see https://downloads.haskell.org/ghc/latest/docs/users_guide/exts.html https://downloads.haskell.org/ghc/latest/docs/users_guide/ex...
- mtl77 3y ago
  Gergo and I are working on a new implementation of the Mu compiler that uses GHC up to Core; from which we translate to the legacy Mu bytecode format. The original Mu compiler is not designed for separate compilation, our new version is, based on GHC's binary representation of compiled interfaces with unfoldings.
cosmic_quanta 3y ago
While Haskell jobs may appear scarce, Standard Chartered is often recruiting for Haskell devs: https://discourse.haskell.org/search?expanded=true&q=Standard%20Chartered https://discourse.haskell.org/search?expanded=true&q=Standar...
- dreixel 3y ago
  Thanks for finding these! More directly, here's my latest jobs post: https://discourse.haskell.org/t/haskell-jobs-at-standard-chartered-various-locations-and-seniority/6157 https://discourse.haskell.org/t/haskell-jobs-at-standard-cha...
lincpa 3y ago
[dead]
nemoniac 3y ago
So many questions left unasked! In particular they have "recursion disabled by default" but "we have all the usual recursive combinators". Does this mean that they can't explicitly write recursion buy may achieve recursive effects only through the combinators? I'd like to know more about this. Is it as expressive as recursion only through "properly optimised" combinators?
- Laaas 3y ago
  By combinators they mean things like folding/reducing over lists, i.e. combinators/higher order functions for operating on potentially infinite structures. It's not as expressive as raw recursion since many uses of recursion are hard to express through combinators, albeit a great deal of research has been done in this area (look up recursion schemes).
- ackfoobar 3y ago
  It has been argued that unrestricted recursion is like GOTO. I have some limited experience with "recursion schemes", they're quite expressive. But I feel like it's trying to give every possible pattern of recursion a name, to the point that it's almost becoming a joke - e.g. "zygohistomorphic prepromorphism".
- mtl77 3y ago
  The reason is that we don't have a general solution for tail-call optimisation for computations made of a mixture of Mu code and C++ primitives. We have higher order functions defined in C++ that can be called from Mu, and can call back Mu functions passed to them. This back and forth between languages has occurred naturally in recursive definitions, and we don't have a way of reusing stack frames between the Mu interpreter and the native C++ code for these higher-order functions. We decided a long time ago to disable recursion by default, encouraging programmers to use recursion operators. We do however have a Mu language extension that enables recursion, which is used when the recursion depth is known to be bounded.
boxed 3y ago
> Haskell in Production Does not use Haskell. Is it just me or is this... weird?
- gorjusborg 3y ago
  Should probably be 'Dialect of Haskell in Production'
- tuukkah 3y ago
  Sure, it's a bit weird but not that weird as they have a huge codebase in a specific domain. Also, they still end up using a lot of Haskell code (and semantics) although they write in a thin layer of "DSL" on top of Haskell and GHC (which itself is a lot of "Haskell in production"). > Mu has a strict runtime, which makes program performance easier to analyse and predict – but prevents us from just copy-pasting some Haskell libraries that rely crucially on lazy evaluation.
  - boxed 3y ago
    From what I understood they don't use Haskell underneath at all, but they copy paste code from Haskell and sometimes have to modify it.
    - tuukkah 3y ago
      They say their codebase includes 400k lines of Haskell code (plus 4.5M lines of Mu). Also, they use GHC which is a huge Haskell program, and they customize it with their own extensions etc. As they copy Haskell library code and only sometimes modify it, I would count it as Haskell code until they modify it.
- yakshaving_jgt 3y ago
  Mu is essentially Haskell. It's not clear to me where you got "does not use Haskell" from.
  - boxed 3y ago
mark_l_watson 3y ago
Interesting interview. I applied for a Haskell job at Standard Charter in Singapore about eight years ago. I didn’t really expect to get the job because the only Haskell experience I had was my own Haskell projects and I had written a short book on Haskell. The guy interviewing me and reviewing my take home programming test had done a PhD studying aspects of Haskell. It was an interesting experience for me.
- weavie 3y ago
  Did you get the job?
  - mark_l_watson 3y ago
    No. The interviewer suggested that I get one year experience elsewhere, then reapply for the job. I started to do that, but a month into a Haskell gig, I received an offer to manage a deep learning team at Capital One, and pivoted.
- grumpyprole 3y ago
  You won't need a Haskell PhD to work at a bank, even one that uses Haskell.
- eru 3y ago
  I worked in that team from 2012 to 2014. I don't have any degree and didn't publish any books. Perhaps they made the interviews more difficult since, or you got unlucky?
  - mark_l_watson 3y ago
    I just don’t think I had enough Haskell experience.
kerkeslager 3y ago
I've talked to a lot of people using Haskell in production over the years, and one of the things I've consistently noticed is a disconnect between the Haskell features that are often touted, and the Haskell features that people use in production. This isn't always a quiet omission, like "we didn't end up using this" either. The best example is that a lot of people end up doing some significant work to disable lazy evaluation. Simon Peyton Jones is reported to have said that lazy evaluation has been kept "to keep us honest about side effects", i.e., that size effects become immediately painful with pervasive lazy evaluation. But in a production system, functional purity (no side-effects) is a means to an end, and lazy evaluation comes at a pretty extreme performance cost. Not everyone disables lazy evaluation in production Haskell, because not everyone thinks they need to for performance, but I've definitely heard about it enough times that it seems like a relevant pattern. Production users of Haskell do seem to be happy about types, years into projects, so I'd take away that types are a benefit, but I do wonder if one might get those same benefits in another language without the problems which seem to crop up with Haskell, such as laziness causing performance issues, monadic complexity, and difficulty hiring.
- kqr 3y ago
  Small distinction that does not detract from your excellent point: the question is not about laziness vs. strictness since any reasonably large program uses both extensively. Rather, it is about the properties of laziness or strictness as a default evaluation strategy.
- yodsanklai 3y ago
  OCaml may be seen as more pragmatic solution.
  - thesz 3y ago
    It cannot. Arbitrary side effects allowed by strict-by-default language without types constraining these ruin easeness of software development provided by Haskell. When your Haskell program works and walks, add bangs and it will run. Lately I heard from my colleagues it will run even without bangs as ghc has improved.
tildeheimat 3y ago
Should probably look into Haskell. Have absolutely no clue what's it best suited for?
- Pirate-of-SV 3y ago
  Writing parsers and tokenizers probably.
  - ParetoOptimal 3y ago
    That's what it's best for, but personally I use it for everything. If I ever get into low-level code I'll probably use Rust though. You can confirm that parsers/tokenizers is ranked "best in class" here though: https://github.com/Gabriella439/post-rfc/blob/main/sotu.md https://github.com/Gabriella439/post-rfc/blob/main/sotu.md
- unhammer 3y ago
  Here are some stories of what it's actually used for: https://old.reddit.com/r/haskell/comments/12vhz1c/genuine_question_how_do_you_all_use_haskell_irl/ https://old.reddit.com/r/haskell/comments/12vhz1c/genuine_qu...
  - irmgardweber 3y ago
    Ha! I was also curious. Thank you for sharing.
- gilmi 3y ago
  Please allow me to share a blog post and a learning resource: - Consider Haskell - https://gilmi.me/blog/post/2020/04/28/consider-haskell https://gilmi.me/blog/post/2020/04/28/consider-haskell - Learn Haskell by building a blog generator - https://lhbg-book.link https://lhbg-book.link
- ParetoOptimal 3y ago
  You might find this survey of Haskell's maturity of different use cases very useful: https://github.com/Gabriella439/post-rfc/blob/main/sotu.md https://github.com/Gabriella439/post-rfc/blob/main/sotu.md
usgroup 3y ago
I'm struggling to understand how possibly inventing your own language -- one not even compatible with the Haskell eco-system -- was ever a good idea in production.
- tome 3y ago
  It was written to replace an existing functional DSL called "Lambda", and to target Lambda's runtime.
  - usgroup 3y ago
    Reminds of an epistemology joke. The reason the moon is made of cheese is because it is made of yellow cheese.
- grumpyprole 3y ago
  Why is it any different to "bank python" used by e.g. JPMorgan or BAML? Or Google writing Dart?
  - cuteboy19 3y ago
    I don't think they ever touched the interpreter. It's more like a huge framework on top of regular py.
    - grumpyprole 3y ago
      It's an entire fork of the Python ecosystem. So not really that much different to what we have here. Maintaining an ecosystem of libraries and tools is the hard part, not a compiler or interpreter.
- mtl77 3y ago
  Main reasons at the time: * strictness was preferred for consistency 1) with the C++ execution model (a lot of our code base is written in C++ by quants, and a typical Mu program will make extensive use of these external calls; they can even be higher-order, calling back into C++); and 2) because we have a lot of end-users who are not Haskell experts and are much more confortable with a strict execution model * code mobility is essential to what we do, and was not directly available in Haskell: we often capture computations on our users' desktops running Windows, and stream them to a Unix compute farm. * GHC was not openly available as an API years ago; this is different now which is why we have started a new implementation that is GHC-based.
liendolucas 3y ago
I'm shocked, in the first paragraph they assert that they handle 6 million lines of code. Is it common for the fintech industry to have codebases of that magnitude? I'm aware that for example OSes and browsers can achieve similar sizes (if I'm not wrong certainly more) but I'd have never thought that a fintech company could approach such size. It just seems way too much... Edit: Question out of ignorance for people that regularly program in Haskell in the industry: I want to believe that the language has nothing to do with this and is just their business logic, right?
- great_wubwub 3y ago
  I don't know about Standard Chartered specifically, but I remember one of the big US banks (I forget which one. JPMC or Goldman Sachs, probably - it was a while ago) telling us that they had more developers than Microsoft. Don't sleep on these finance companies, they're big development shops.
  - postexitus 3y ago
    Jamie Dimon said that they have enough engineers to build their own Windows, back in 2007/2008.
- mhh__ 3y ago
  6 million is easy, in fact its probably a small-ish fraction of their total code as a firm. Especially for a bank where their processes are a bit more defined than (say) a hedge fund. What do you think fintech companies do? (I'm not entirely sure either but I can potentially give a flavour of finance coding in general) If you want to come up with prices for things then you'll need a bunch of mathematical optimization code, interpolation code etc, then an enormous chunk of market conventions (Quick! How do you calculate the yield to maturity for a Brazilian treasury bill). Having established this you will then need to build a system for getting market data to the right places to be used - and probably snapping data for historical record too. This could be as simple as drinking from the Bloomberg firehose, or reading from an SFTP server in the morning (no really), or pulling in trading data from an exchange (https://www.eurex.com/ex-en/data/trading-files/efpi https://www.eurex.com/ex-en/data/trading-files/efpi for example). It all gets very messy. And now, if you are a bank rather than merely a player in a market you also probably have your fingers in many fiddly boring pies too (e.g. retail banking, mortgages stuff like that).
swyx 3y ago
I guess I have the relatively unique distinction of having worked at Standard Chartered and used Haskell/Mu in anger from 2012-14, as a currency/interest rates options trader using it for volatility surface mapping and making pricing/market data tools. Offering thoughts: > I’m the head of the Core Strats team at Standard Chartered Bank. Core Strats consists of over 40 developers using Haskell (well, actually Mu) as their main programming language. wow - this team used to be 8-10 people? amazing to see its grown so much. our old head used to be https://donsbot.com/about/ https://donsbot.com/about/ . i have a pet theory that Don and gang ensured Haskell's viability by building so much stuff at SCB that SCB was kind of pot-committed to Haskell, thereby ensuring that they would keep investing and hiring until the end of SCB. Similar move was played by Cheng Lou over at Facebook, for Reason/Rescript. if you want to grow a programming language at a bigcorp, embed it in a biz critical application and you have job security for life > Our core financial analytics library, Cortex, consists mostly of Mu/Haskell code (over 6.5 million lines) and forms the basis of our price and risk engine across all asset classes. It’s also pretty full-stack; we use it for everything, from server-side long-running batches handling millions of trades a day to front-end desktop graphical user interfaces. yep i can attest to this - i will say that Mu was the best developer experience i've had for a long time. It comes with its own storage/persistence layer (simple KV store iirc) but the joy of not having to choose or maintain or scale was great. also the entire internal package ecosystem was searchable via function signature (via Neil Mitchell's Hoogle) and versioned daily (so if any regressions happened you could just jump back day by day - we didnt use git in my day). i also missed if there was a testing suite. > We can seamlessly call C++ functions from Mu and vice-versa, and pure Mu functions can be used in Excel formulae via our Excel add-in yeah this was used a bunch. however the traders rarely touched this code so it probably required a lot of support from the devs and probably some misunderstandings where had. > What benefits do you see for using Haskell in banking and fintech areas? One great advantage is static typing.. no no no. correct me if i'm wrong but dont large fintechs like stanchart and jane st use functional languages because its so much better to parallelize for risk scenarios and complex option pricing? > Does Standard Chartered have an in-house training program for upskilling Haskell developers? For training absolute beginners in Haskell we typically engage an external training partner. ha. in my day i was just given Learn You A Haskell and told good luck! SCB people reading along, i hope some of my FXO apps are still around. Coriolis, Corona, and i'm blanking on the main pricing one. 10 years wipes away a lot.
koromak 3y ago
I'm just starting to learn Haskell and I honestly can't fathom how it gets written in production. Everything requires so much more consideration, which is fun and valuable in some situations, but not when you just need to get the damn thing done in time for a deadline. I'm not experienced enough to know, but it does feel like Haskell prioritizes "clever" code. Very beautiful, but hard to understand at a glance.
- jsbg 3y ago
  You can get features done very quickly in Haskell after the initial learning curve that most languages have. The type system lets you catch a lot of mistakes before runtime and in my experience things are more likely to work the first time in Haskell. Testability does require some forethought but if your codebase always takes that into account then it's as easy as other languages.
- justincredible 3y ago
  [dead]
- alex_lav 3y ago
  It's sad you're getting downvoted only because you've commented (ever so slightly) negatively about people's language de jour. I would argue that "Can people understand a unit of code without experience in that language/domain" is valuable, even if not the _most_ valuable aspect of a language. I also think dismissing this desire is usually an act of gatekeeping. "_I_ understand it so it's fine."
- grumpyprole 3y ago
  Sounds like you just need more practice. Haskell is more expressive than most other languages and offers much more consistent and powerful abstractions. This does result in more productivity once the learning curve is overcome.
- ParetoOptimal 3y ago
  > I'm just starting to learn Haskell and I honestly can't fathom how it gets written in production. Everything requires so much more consideration, which is fun and valuable in some situations, but not when you just need to get the damn thing done in time for a deadline. There are ways to write quick n dirty Haskell. Actually a super-power of Haskell is how easy it is to refactor from "quick n dirty" to "reasonable quality" without introducing defects.
mrose11 3y ago
> We can seamlessly call C++ functions from Mu and vice-versa, and pure Mu functions can be used in Excel formulae via our Excel add-in. wtf
- ykonstant 3y ago
  It makes perfect sense; non-technical users and employees with different specializations (like actuaries) use lots of Excel, especially in fintech; and if you can share function implementations verbatim between your codebase and Excel, you get higher confidence for coherence.
- WJW 3y ago
  What's weird about that? Excel is absolutely everywhere in finance, so if you have a special programming language that you consider to be part of your secret sauce it makes sense to integrate it into excel as well. A friend who works as a at risk manager at a major bank sometimes shares horror stories about the giant spreadsheet they have for some part of their holdings. They have disabled the "automatic recalculation" feature because it takes over 48 hours to recalculate everything. This is after running it through a special excel compiler to speed it up.
  - mrose11 3y ago
    Excel is everywhere used everywhere and for very important things. I just never... thought about writing a Haskell plugin for it. It makes sense! Better than using some other language. Functional for Excel makes sense. I love hearing horror stories about small (and large!) governments and companies abusing and misusing Excel. https://sheetcast.com/articles/ten-memorable-excel-disasters https://sheetcast.com/articles/ten-memorable-excel-disasters
- xboxnolifes 3y ago
  It's not very different from using python to hook C/Fortran libraries together.
- deleted 3y ago
  [deleted]

186 points by NaeosPsy 3y ago

> Mu can easily take several minutes to compile a single large application. We will circumvent those problems by switching our front-end to use GHC directly What does he mean by this?

Vosporos 3y ago
From what I understand: GHC takes care of the parsing of the source file, error reporting, etc. Then the AST (or an intermediate representation) is passed to the of the Mu compiler, which performs whole-program optimisations.
- nequo 3y ago
  Gergo Erdi gave a talk on the GHC efforts of Standard Chartered last year: https://www.youtube.com/watch?v=fZ66Pz7015Q https://www.youtube.com/watch?v=fZ66Pz7015Q
wathef 3y ago
I may be wrong, but if I remember correctly Standard Charter uses their own “dialect” called Mu[1]. 1 - https://icfp21.sigplan.org/details/hiw-2021-papers/14/Haskell-reinterpreted-large-scale-real-world-experience-with-the-Mu-compiler-in-Fin https://icfp21.sigplan.org/details/hiw-2021-papers/14/Haskel...
- deleted 3y ago
  [deleted]
grumpyprole 3y ago
He probably means that GHC will parse and type check faster. Switching the front-end doesn't sound like it will circumvent the whole program nature of their compilation. But I'm just guessing.
misja111 3y ago
I'm also interested. Just guessing, but I guess it means that while they use their own (slow compiling) 'Mu' dialect for most applications, they have switched to using 'standard' GHC for their front-end apps.
- tuukkah 3y ago
  Frontend here refers to the first stage of the three-stage structure of compilers, ie. the functionalities such as parsing and type checking that come before the optimization and target code generation phases: https://en.wikipedia.org/wiki/Compiler#Three-stage_compiler_structure https://en.wikipedia.org/wiki/Compiler#Three-stage_compiler_...
  - misja111 3y ago
    Didn't know that, thanks! But then I'm wondering how they can do parsing and type checking with GHC when they are using their own dialect, "Mu". How come GHC can parse that?
    - nequo 3y ago
      The differences between Mu and Haskell are not that large. They use GHC as a library in their front end. Gergo Erdi gave a talk on it last year: https://news.ycombinator.com/item?id=35801844 https://news.ycombinator.com/item?id=35801844
    - tuukkah 3y ago
      GHC has a system for language extensions so they have developed a few proprietary ones for Mu. For the standard ones, see https://downloads.haskell.org/ghc/latest/docs/users_guide/exts.html https://downloads.haskell.org/ghc/latest/docs/users_guide/ex...
mtl77 3y ago
Gergo and I are working on a new implementation of the Mu compiler that uses GHC up to Core; from which we translate to the legacy Mu bytecode format. The original Mu compiler is not designed for separate compilation, our new version is, based on GHC's binary representation of compiled interfaces with unfoldings.

cosmic_quanta 3y ago

While Haskell jobs may appear scarce, Standard Chartered is often recruiting for Haskell devs: https://discourse.haskell.org/search?expanded=true&q=Standard%20Chartered https://discourse.haskell.org/search?expanded=true&q=Standar...

dreixel 3y ago
Thanks for finding these! More directly, here's my latest jobs post: https://discourse.haskell.org/t/haskell-jobs-at-standard-chartered-various-locations-and-seniority/6157 https://discourse.haskell.org/t/haskell-jobs-at-standard-cha...

lincpa 3y ago

[dead]

nemoniac 3y ago

So many questions left unasked! In particular they have "recursion disabled by default" but "we have all the usual recursive combinators". Does this mean that they can't explicitly write recursion buy may achieve recursive effects only through the combinators? I'd like to know more about this. Is it as expressive as recursion only through "properly optimised" combinators?

Laaas 3y ago
By combinators they mean things like folding/reducing over lists, i.e. combinators/higher order functions for operating on potentially infinite structures. It's not as expressive as raw recursion since many uses of recursion are hard to express through combinators, albeit a great deal of research has been done in this area (look up recursion schemes).
ackfoobar 3y ago
It has been argued that unrestricted recursion is like GOTO. I have some limited experience with "recursion schemes", they're quite expressive. But I feel like it's trying to give every possible pattern of recursion a name, to the point that it's almost becoming a joke - e.g. "zygohistomorphic prepromorphism".
mtl77 3y ago
The reason is that we don't have a general solution for tail-call optimisation for computations made of a mixture of Mu code and C++ primitives. We have higher order functions defined in C++ that can be called from Mu, and can call back Mu functions passed to them. This back and forth between languages has occurred naturally in recursive definitions, and we don't have a way of reusing stack frames between the Mu interpreter and the native C++ code for these higher-order functions. We decided a long time ago to disable recursion by default, encouraging programmers to use recursion operators. We do however have a Mu language extension that enables recursion, which is used when the recursion depth is known to be bounded.

boxed 3y ago

> Haskell in Production Does not use Haskell. Is it just me or is this... weird?

gorjusborg 3y ago
Should probably be 'Dialect of Haskell in Production'
tuukkah 3y ago
Sure, it's a bit weird but not that weird as they have a huge codebase in a specific domain. Also, they still end up using a lot of Haskell code (and semantics) although they write in a thin layer of "DSL" on top of Haskell and GHC (which itself is a lot of "Haskell in production"). > Mu has a strict runtime, which makes program performance easier to analyse and predict – but prevents us from just copy-pasting some Haskell libraries that rely crucially on lazy evaluation.
- boxed 3y ago
  From what I understood they don't use Haskell underneath at all, but they copy paste code from Haskell and sometimes have to modify it.
  - tuukkah 3y ago
    They say their codebase includes 400k lines of Haskell code (plus 4.5M lines of Mu). Also, they use GHC which is a huge Haskell program, and they customize it with their own extensions etc. As they copy Haskell library code and only sometimes modify it, I would count it as Haskell code until they modify it.
yakshaving_jgt 3y ago
Mu is essentially Haskell. It's not clear to me where you got "does not use Haskell" from.
- boxed 3y ago

mark_l_watson 3y ago

Interesting interview. I applied for a Haskell job at Standard Charter in Singapore about eight years ago. I didn’t really expect to get the job because the only Haskell experience I had was my own Haskell projects and I had written a short book on Haskell. The guy interviewing me and reviewing my take home programming test had done a PhD studying aspects of Haskell. It was an interesting experience for me.

weavie 3y ago
Did you get the job?
- mark_l_watson 3y ago
  No. The interviewer suggested that I get one year experience elsewhere, then reapply for the job. I started to do that, but a month into a Haskell gig, I received an offer to manage a deep learning team at Capital One, and pivoted.
grumpyprole 3y ago
You won't need a Haskell PhD to work at a bank, even one that uses Haskell.
eru 3y ago
I worked in that team from 2012 to 2014. I don't have any degree and didn't publish any books. Perhaps they made the interviews more difficult since, or you got unlucky?
- mark_l_watson 3y ago
  I just don’t think I had enough Haskell experience.

kerkeslager 3y ago

I've talked to a lot of people using Haskell in production over the years, and one of the things I've consistently noticed is a disconnect between the Haskell features that are often touted, and the Haskell features that people use in production. This isn't always a quiet omission, like "we didn't end up using this" either. The best example is that a lot of people end up doing some significant work to disable lazy evaluation. Simon Peyton Jones is reported to have said that lazy evaluation has been kept "to keep us honest about side effects", i.e., that size effects become immediately painful with pervasive lazy evaluation. But in a production system, functional purity (no side-effects) is a means to an end, and lazy evaluation comes at a pretty extreme performance cost. Not everyone disables lazy evaluation in production Haskell, because not everyone thinks they need to for performance, but I've definitely heard about it enough times that it seems like a relevant pattern. Production users of Haskell do seem to be happy about types, years into projects, so I'd take away that types are a benefit, but I do wonder if one might get those same benefits in another language without the problems which seem to crop up with Haskell, such as laziness causing performance issues, monadic complexity, and difficulty hiring.

kqr 3y ago
Small distinction that does not detract from your excellent point: the question is not about laziness vs. strictness since any reasonably large program uses both extensively. Rather, it is about the properties of laziness or strictness as a default evaluation strategy.
yodsanklai 3y ago
OCaml may be seen as more pragmatic solution.
- thesz 3y ago
  It cannot. Arbitrary side effects allowed by strict-by-default language without types constraining these ruin easeness of software development provided by Haskell. When your Haskell program works and walks, add bangs and it will run. Lately I heard from my colleagues it will run even without bangs as ghc has improved.

tildeheimat 3y ago

Should probably look into Haskell. Have absolutely no clue what's it best suited for?

Pirate-of-SV 3y ago
Writing parsers and tokenizers probably.
- ParetoOptimal 3y ago
  That's what it's best for, but personally I use it for everything. If I ever get into low-level code I'll probably use Rust though. You can confirm that parsers/tokenizers is ranked "best in class" here though: https://github.com/Gabriella439/post-rfc/blob/main/sotu.md https://github.com/Gabriella439/post-rfc/blob/main/sotu.md
unhammer 3y ago
Here are some stories of what it's actually used for: https://old.reddit.com/r/haskell/comments/12vhz1c/genuine_question_how_do_you_all_use_haskell_irl/ https://old.reddit.com/r/haskell/comments/12vhz1c/genuine_qu...
- irmgardweber 3y ago
  Ha! I was also curious. Thank you for sharing.
gilmi 3y ago
Please allow me to share a blog post and a learning resource: - Consider Haskell - https://gilmi.me/blog/post/2020/04/28/consider-haskell https://gilmi.me/blog/post/2020/04/28/consider-haskell - Learn Haskell by building a blog generator - https://lhbg-book.link https://lhbg-book.link
ParetoOptimal 3y ago
You might find this survey of Haskell's maturity of different use cases very useful: https://github.com/Gabriella439/post-rfc/blob/main/sotu.md https://github.com/Gabriella439/post-rfc/blob/main/sotu.md

usgroup 3y ago

I'm struggling to understand how possibly inventing your own language -- one not even compatible with the Haskell eco-system -- was ever a good idea in production.

tome 3y ago
It was written to replace an existing functional DSL called "Lambda", and to target Lambda's runtime.
- usgroup 3y ago
  Reminds of an epistemology joke. The reason the moon is made of cheese is because it is made of yellow cheese.
grumpyprole 3y ago
Why is it any different to "bank python" used by e.g. JPMorgan or BAML? Or Google writing Dart?
- cuteboy19 3y ago
  I don't think they ever touched the interpreter. It's more like a huge framework on top of regular py.
  - grumpyprole 3y ago
    It's an entire fork of the Python ecosystem. So not really that much different to what we have here. Maintaining an ecosystem of libraries and tools is the hard part, not a compiler or interpreter.
mtl77 3y ago
Main reasons at the time: * strictness was preferred for consistency 1) with the C++ execution model (a lot of our code base is written in C++ by quants, and a typical Mu program will make extensive use of these external calls; they can even be higher-order, calling back into C++); and 2) because we have a lot of end-users who are not Haskell experts and are much more confortable with a strict execution model * code mobility is essential to what we do, and was not directly available in Haskell: we often capture computations on our users' desktops running Windows, and stream them to a Unix compute farm. * GHC was not openly available as an API years ago; this is different now which is why we have started a new implementation that is GHC-based.

liendolucas 3y ago

I'm shocked, in the first paragraph they assert that they handle 6 million lines of code. Is it common for the fintech industry to have codebases of that magnitude? I'm aware that for example OSes and browsers can achieve similar sizes (if I'm not wrong certainly more) but I'd have never thought that a fintech company could approach such size. It just seems way too much... Edit: Question out of ignorance for people that regularly program in Haskell in the industry: I want to believe that the language has nothing to do with this and is just their business logic, right?

great_wubwub 3y ago
I don't know about Standard Chartered specifically, but I remember one of the big US banks (I forget which one. JPMC or Goldman Sachs, probably - it was a while ago) telling us that they had more developers than Microsoft. Don't sleep on these finance companies, they're big development shops.
- postexitus 3y ago
  Jamie Dimon said that they have enough engineers to build their own Windows, back in 2007/2008.
mhh__ 3y ago
6 million is easy, in fact its probably a small-ish fraction of their total code as a firm. Especially for a bank where their processes are a bit more defined than (say) a hedge fund. What do you think fintech companies do? (I'm not entirely sure either but I can potentially give a flavour of finance coding in general) If you want to come up with prices for things then you'll need a bunch of mathematical optimization code, interpolation code etc, then an enormous chunk of market conventions (Quick! How do you calculate the yield to maturity for a Brazilian treasury bill). Having established this you will then need to build a system for getting market data to the right places to be used - and probably snapping data for historical record too. This could be as simple as drinking from the Bloomberg firehose, or reading from an SFTP server in the morning (no really), or pulling in trading data from an exchange (https://www.eurex.com/ex-en/data/trading-files/efpi https://www.eurex.com/ex-en/data/trading-files/efpi for example). It all gets very messy. And now, if you are a bank rather than merely a player in a market you also probably have your fingers in many fiddly boring pies too (e.g. retail banking, mortgages stuff like that).

swyx 3y ago

I guess I have the relatively unique distinction of having worked at Standard Chartered and used Haskell/Mu in anger from 2012-14, as a currency/interest rates options trader using it for volatility surface mapping and making pricing/market data tools. Offering thoughts: > I’m the head of the Core Strats team at Standard Chartered Bank. Core Strats consists of over 40 developers using Haskell (well, actually Mu) as their main programming language. wow - this team used to be 8-10 people? amazing to see its grown so much. our old head used to be https://donsbot.com/about/ https://donsbot.com/about/ . i have a pet theory that Don and gang ensured Haskell's viability by building so much stuff at SCB that SCB was kind of pot-committed to Haskell, thereby ensuring that they would keep investing and hiring until the end of SCB. Similar move was played by Cheng Lou over at Facebook, for Reason/Rescript. if you want to grow a programming language at a bigcorp, embed it in a biz critical application and you have job security for life > Our core financial analytics library, Cortex, consists mostly of Mu/Haskell code (over 6.5 million lines) and forms the basis of our price and risk engine across all asset classes. It’s also pretty full-stack; we use it for everything, from server-side long-running batches handling millions of trades a day to front-end desktop graphical user interfaces. yep i can attest to this - i will say that Mu was the best developer experience i've had for a long time. It comes with its own storage/persistence layer (simple KV store iirc) but the joy of not having to choose or maintain or scale was great. also the entire internal package ecosystem was searchable via function signature (via Neil Mitchell's Hoogle) and versioned daily (so if any regressions happened you could just jump back day by day - we didnt use git in my day). i also missed if there was a testing suite. > We can seamlessly call C++ functions from Mu and vice-versa, and pure Mu functions can be used in Excel formulae via our Excel add-in yeah this was used a bunch. however the traders rarely touched this code so it probably required a lot of support from the devs and probably some misunderstandings where had. > What benefits do you see for using Haskell in banking and fintech areas? One great advantage is static typing.. no no no. correct me if i'm wrong but dont large fintechs like stanchart and jane st use functional languages because its so much better to parallelize for risk scenarios and complex option pricing? > Does Standard Chartered have an in-house training program for upskilling Haskell developers? For training absolute beginners in Haskell we typically engage an external training partner. ha. in my day i was just given Learn You A Haskell and told good luck! SCB people reading along, i hope some of my FXO apps are still around. Coriolis, Corona, and i'm blanking on the main pricing one. 10 years wipes away a lot.

koromak 3y ago

I'm just starting to learn Haskell and I honestly can't fathom how it gets written in production. Everything requires so much more consideration, which is fun and valuable in some situations, but not when you just need to get the damn thing done in time for a deadline. I'm not experienced enough to know, but it does feel like Haskell prioritizes "clever" code. Very beautiful, but hard to understand at a glance.

jsbg 3y ago
You can get features done very quickly in Haskell after the initial learning curve that most languages have. The type system lets you catch a lot of mistakes before runtime and in my experience things are more likely to work the first time in Haskell. Testability does require some forethought but if your codebase always takes that into account then it's as easy as other languages.
justincredible 3y ago
[dead]
alex_lav 3y ago
It's sad you're getting downvoted only because you've commented (ever so slightly) negatively about people's language de jour. I would argue that "Can people understand a unit of code without experience in that language/domain" is valuable, even if not the _most_ valuable aspect of a language. I also think dismissing this desire is usually an act of gatekeeping. "_I_ understand it so it's fine."
grumpyprole 3y ago
Sounds like you just need more practice. Haskell is more expressive than most other languages and offers much more consistent and powerful abstractions. This does result in more productivity once the learning curve is overcome.
ParetoOptimal 3y ago
> I'm just starting to learn Haskell and I honestly can't fathom how it gets written in production. Everything requires so much more consideration, which is fun and valuable in some situations, but not when you just need to get the damn thing done in time for a deadline. There are ways to write quick n dirty Haskell. Actually a super-power of Haskell is how easy it is to refactor from "quick n dirty" to "reasonable quality" without introducing defects.

mrose11 3y ago

> We can seamlessly call C++ functions from Mu and vice-versa, and pure Mu functions can be used in Excel formulae via our Excel add-in. wtf

ykonstant 3y ago
It makes perfect sense; non-technical users and employees with different specializations (like actuaries) use lots of Excel, especially in fintech; and if you can share function implementations verbatim between your codebase and Excel, you get higher confidence for coherence.
WJW 3y ago
What's weird about that? Excel is absolutely everywhere in finance, so if you have a special programming language that you consider to be part of your secret sauce it makes sense to integrate it into excel as well. A friend who works as a at risk manager at a major bank sometimes shares horror stories about the giant spreadsheet they have for some part of their holdings. They have disabled the "automatic recalculation" feature because it takes over 48 hours to recalculate everything. This is after running it through a special excel compiler to speed it up.
- mrose11 3y ago
  Excel is everywhere used everywhere and for very important things. I just never... thought about writing a Haskell plugin for it. It makes sense! Better than using some other language. Functional for Excel makes sense. I love hearing horror stories about small (and large!) governments and companies abusing and misusing Excel. https://sheetcast.com/articles/ten-memorable-excel-disasters https://sheetcast.com/articles/ten-memorable-excel-disasters
xboxnolifes 3y ago
It's not very different from using python to hook C/Fortran libraries together.
deleted 3y ago
[deleted]