3 ms·

The difference here is scale: Jane Street is a 20 year old company with over 1000 employees, Channable has maybe a fifth of that. The proportion of engineering

by wespiser_2018 4y ago

The difference here is scale: Jane Street is a 20 year old company with over 1000 employees, Channable has maybe a fifth of that. The proportion of engineering effort that will have to go into language expertise will just be higher at a smaller shop pushing the bounds and this will simply take away from feature development. I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. This is a huge issue for adoption: that an engineering team using Haskell will have to devote a significant amount of time/effort to tooling/resuability/infra that would otherwise be available to directly deliver value to customers if they had gone with a different language. Stuff like pagination, custom json/typescript encodings and route generation, organizing the code base so complication time isn't a nightmare, these are all "inner source" projects I've worked on that needed to be done for us to use Haskell for a decent sized team. The fact that Channable had to go down the road of ghc-compact regions, to me, suggests that on a strictly technical basis they would be much more productive using a language like Rust of C++. Better for their end users, and better for their investors. Of course, large companies can make these deep investments into languages, tooling, and infrastructure (and by all accounts Channable is well on their way) but when they do it's only a small proportion of their entire dev team. For Haskell, I contend that its' lack of state of the art library support and tooling is a massive impediment to wide scale adoption: companies just don't want to put 1/3 of a 30 person dev group on a "platform" team in order to be productive.

ParetoOptimal 4y ago
> I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. Can you share some of the successes, failures, and composition of teams Haskell experience? Also any success/failure causes you believe they had in common.
agentultra 4y ago
Any team is eventually going to have to invest in tooling. If they had gone with C++ they would be fighting different fires: memory safety problems, build tool complexity, bloated compile times, memory leaks, etc. If they went with Javascript; run-time errors, async memory pressure, complex deployment artifacts, etc. Haskell has benefits for its trade-offs. Its type system is world class. I get a lot of leverage out of its type class constraint resolution. It has a sufficient compiler that generates good, fast code and a run-time system that makes dealing with concurrency much safer and easier than other systems I've worked with. You can choose any language we have out there right now and a decently sized engineering team, if they're smart, are going to have an internal team or some amount of their backlog of work dedicated to tooling. Even Twitter has a tooling team! And Java has an ecosystem where millions of dollars of full-time engineers are building IDEs and DUX tools constantly. It's the nature of the beast.
- wespiser_2018 4y ago
  yes, but I'd argue the "buy vs. build" balance is shifted too far to the build side right now when you build all your backend services (or most of them) with Haskell. In the Haskell ecosystem there's a good deal of solutionism, but this can be solved with diligent decision making and good technical leadership, the greater issue, IMO, is that Haskell lacks the breadth and depth of libraries that allow Haskell to be a low risk language choice that allows you to build a project, see how things go, then restart with a different team/assumptions if need be. I want Haskell to win and solve these problems: I've invested years of my life learning the ecosystem and actively maintain open sources libraries, but after years of using the language and running into problems, I've lived through the situations when Haskell is less than stellar as a software engineering tool and it's quite disappointing to see.
  - agentultra 4y ago
    I think the only solution here is to dump a war chest of cash into the ecosystem. One the reasons Java and C++ are incumbents is due to the network effects that made them what they are today. Oracle literally spent hundreds of millions of dollars convincing developers Java was the next big thing. C++ enjoyed platform support by being compatible with C and was embraced by big players with deep pockets. Haskell doesn’t have a shot at becoming “a safe choice” in that regard. It's not an exclusive language to a desire-able platform, it doesn't have an organization with deep pockets to fund its development, it doesn't have a killer application. But it is an oracle for the future of where programming is headed. Many languages are trying their best to steal ideas from Haskell/FP: immutable data structures, pattern matching, lambda functions, lack of null, constrained parametric polymorphism, etc; patterns from libraries like monads, streams, lenses, functors; all making their way into Java, C#, etc. That being said I haven’t felt that the ecosystem is lacking core libraries for almost anything I’ve been working on. The IDE support is still lacking and profiling tools are there but lacking shiny DUX. Otherwise I can’t really complain.
markeibes 4y ago
Calling compilation time "complication time" is such a good typo
tikhonj 4y ago
Jane Street's been doing this for most of their existence. I interned there a decade ago when they had 300 people and maybe 100 engineers and they had already built up an impressive internal infrastructure—including lots of in-house tools (eg a code review system) that were custom because they got value from building them in-house, not because of OCaml. At that point, they had been on this trajectory for a decade already, certainly well before they had a massive team of developers. I forget the exact details, but they started using OCaml in the early 2000s when the entire company had a single-digit or maybe low-double-digit employee count. Building internal tools, libraries and expertise was clearly a massive net gain in productivity for them—it did not "take away from feature development" in anything but the most superficial and short-sighted accounting. Most recently I worked at Target and the contrast was pretty extreme: on a lot of fronts, Jane Street had better in-house systems than Target had with thousands of tech employees pushed to use mainstream technologies and existing open source systems. It's obviously not a 1:1 comparison, but seeing how productive a relatively small organization like Jane Street could be despite (or, really, thanks to) building a ton of things in house really got me to question the traditional wisdom here. Of course, everything you've said is a real obstacle when that's how managers or other teams think, regardless of whether it's accurate! Dealing with perception and legibility is definitely one of the difficulties in trying to have a Haskell team in a big company. "Nobody got fired for IBM" isn't so much a cute aphorism as corporate gospel—even if they probably should have been.
danielscrubs 4y ago
Would you even know what Jane Street was if they didn’t stand out in this way? Seems like a good way to get smart people interested in working with them.

ParetoOptimal 4y ago

> I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. Can you share some of the successes, failures, and composition of teams Haskell experience? Also any success/failure causes you believe they had in common.

agentultra 4y ago

Any team is eventually going to have to invest in tooling. If they had gone with C++ they would be fighting different fires: memory safety problems, build tool complexity, bloated compile times, memory leaks, etc. If they went with Javascript; run-time errors, async memory pressure, complex deployment artifacts, etc. Haskell has benefits for its trade-offs. Its type system is world class. I get a lot of leverage out of its type class constraint resolution. It has a sufficient compiler that generates good, fast code and a run-time system that makes dealing with concurrency much safer and easier than other systems I've worked with. You can choose any language we have out there right now and a decently sized engineering team, if they're smart, are going to have an internal team or some amount of their backlog of work dedicated to tooling. Even Twitter has a tooling team! And Java has an ecosystem where millions of dollars of full-time engineers are building IDEs and DUX tools constantly. It's the nature of the beast.

wespiser_2018 4y ago
yes, but I'd argue the "buy vs. build" balance is shifted too far to the build side right now when you build all your backend services (or most of them) with Haskell. In the Haskell ecosystem there's a good deal of solutionism, but this can be solved with diligent decision making and good technical leadership, the greater issue, IMO, is that Haskell lacks the breadth and depth of libraries that allow Haskell to be a low risk language choice that allows you to build a project, see how things go, then restart with a different team/assumptions if need be. I want Haskell to win and solve these problems: I've invested years of my life learning the ecosystem and actively maintain open sources libraries, but after years of using the language and running into problems, I've lived through the situations when Haskell is less than stellar as a software engineering tool and it's quite disappointing to see.
- agentultra 4y ago
  I think the only solution here is to dump a war chest of cash into the ecosystem. One the reasons Java and C++ are incumbents is due to the network effects that made them what they are today. Oracle literally spent hundreds of millions of dollars convincing developers Java was the next big thing. C++ enjoyed platform support by being compatible with C and was embraced by big players with deep pockets. Haskell doesn’t have a shot at becoming “a safe choice” in that regard. It's not an exclusive language to a desire-able platform, it doesn't have an organization with deep pockets to fund its development, it doesn't have a killer application. But it is an oracle for the future of where programming is headed. Many languages are trying their best to steal ideas from Haskell/FP: immutable data structures, pattern matching, lambda functions, lack of null, constrained parametric polymorphism, etc; patterns from libraries like monads, streams, lenses, functors; all making their way into Java, C#, etc. That being said I haven’t felt that the ecosystem is lacking core libraries for almost anything I’ve been working on. The IDE support is still lacking and profiling tools are there but lacking shiny DUX. Otherwise I can’t really complain.

markeibes 4y ago

Calling compilation time "complication time" is such a good typo

tikhonj 4y ago

Jane Street's been doing this for most of their existence. I interned there a decade ago when they had 300 people and maybe 100 engineers and they had already built up an impressive internal infrastructure—including lots of in-house tools (eg a code review system) that were custom because they got value from building them in-house, not because of OCaml. At that point, they had been on this trajectory for a decade already, certainly well before they had a massive team of developers. I forget the exact details, but they started using OCaml in the early 2000s when the entire company had a single-digit or maybe low-double-digit employee count. Building internal tools, libraries and expertise was clearly a massive net gain in productivity for them—it did not "take away from feature development" in anything but the most superficial and short-sighted accounting. Most recently I worked at Target and the contrast was pretty extreme: on a lot of fronts, Jane Street had better in-house systems than Target had with thousands of tech employees pushed to use mainstream technologies and existing open source systems. It's obviously not a 1:1 comparison, but seeing how productive a relatively small organization like Jane Street could be despite (or, really, thanks to) building a ton of things in house really got me to question the traditional wisdom here. Of course, everything you've said is a real obstacle when that's how managers or other teams think, regardless of whether it's accurate! Dealing with perception and legibility is definitely one of the difficulties in trying to have a Haskell team in a big company. "Nobody got fired for IBM" isn't so much a cute aphorism as corporate gospel—even if they probably should have been.

danielscrubs 4y ago

Would you even know what Jane Street was if they didn’t stand out in this way? Seems like a good way to get smart people interested in working with them.