5 ms·

> we also encountered some quadratic runtime algorithms in the memory allocator of the RTS – and even compact regions didn’t help then. We documented that parti

> we also encountered some quadratic runtime algorithms in the memory allocator of the RTS – and even compact regions didn’t help then. We documented that particular bug and have since provided a fix for it as well. Just like you never want to be the smartest person in a room, you never want to be the one pushing a language to its limits. IMHO when you’re providing language patches you probably choose the wrong language.

WJW 4y ago
That is the most business-efficient course, but it is also an excellent way to ensure you have no true expertise in your company when a gnarly problem eventually strikes. Avoiding the hardest problems does not attract the kind of people who like solving hard problems, after all.
rkrzr 4y ago
> Just like you never want to be the smartest person in a room, you never want to be the one pushing a language to its limits. (Channable co-founder here) If everybody took this view, then a language could never improve. I have actually been very impressed with how far we have been able to take Haskell before having to push its limits. Anecdotally, we previously were using Scala (and Spark) and ran into issues much earlier. In those cases we were pushing the limits of Spark, but underlying those were the limits of the JVM. > IMHO when you’re providing language patches you probably choose the wrong language. This is a very myopic view. There is much more to consider when choosing a language (what kind of team do you have, what kind of problems are you working on, maintainability, performance, ecosystem, hiring, etc.).
- DonaldPShimoda 4y ago
  I really appreciate your view on all this, and I totally agree with you that the other commenter has a very nearsighted perception of things. I think the go-to example of a group taking this to the extreme is Jane Street with OCaml. For many intents and purposes, Jane Street are really the major shepherds of the language these days. They contribute tons of compiler fixes and improvements, and have written their own replacement of the standard library (which is used by many people). I think it'd be silly to say that they "chose the wrong language"; rather, they've invested in a language ecosystem that benefits them and reflects their needs, while at the same time growing and improving a whole language community. That's an awesome feat.
  - rkrzr 4y ago
    That's how I see it as well - Haskell has been very beneficial for us as a company, so it's nice if we can also contribute something back to the community.
  - wespiser_2018 4y ago
    The difference here is scale: Jane Street is a 20 year old company with over 1000 employees, Channable has maybe a fifth of that. The proportion of engineering effort that will have to go into language expertise will just be higher at a smaller shop pushing the bounds and this will simply take away from feature development. I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. This is a huge issue for adoption: that an engineering team using Haskell will have to devote a significant amount of time/effort to tooling/resuability/infra that would otherwise be available to directly deliver value to customers if they had gone with a different language. Stuff like pagination, custom json/typescript encodings and route generation, organizing the code base so complication time isn't a nightmare, these are all "inner source" projects I've worked on that needed to be done for us to use Haskell for a decent sized team. The fact that Channable had to go down the road of ghc-compact regions, to me, suggests that on a strictly technical basis they would be much more productive using a language like Rust of C++. Better for their end users, and better for their investors. Of course, large companies can make these deep investments into languages, tooling, and infrastructure (and by all accounts Channable is well on their way) but when they do it's only a small proportion of their entire dev team. For Haskell, I contend that its' lack of state of the art library support and tooling is a massive impediment to wide scale adoption: companies just don't want to put 1/3 of a 30 person dev group on a "platform" team in order to be productive.
    - ParetoOptimal 4y ago
      > I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. Can you share some of the successes, failures, and composition of teams Haskell experience? Also any success/failure causes you believe they had in common.
    - agentultra 4y ago
- lmm 4y ago
  Can you talk more about what kind of limits you hit with Scala and Spark? I definitely have my own biases, but my feeling is you get 95% of the good stuff of Haskell while avoiding most of the problems mentioned elsewhere in this thread (e.g. you get a proper IDE-integrated debugger with breakpoints/stepping/etc. that Just Works).
tinco 4y ago
You could be correct, or you might have an exaggerated preconceived notion on how hard it is to provide language level patches because of how the industry was in the 90's and early 2000's. Due to chance I've worked in a company where a colleague had patched the language we were using to save some memory. Back then I (and many people in the Ruby community) considered him a rockstar for doing so. That praise definitely was deserved, since he's done all sorts of cool stuff in his career, but in retrospect that while bold, this feat wasn't all that inconceivable. The Ruby interpreter isn't some magical blackbox, it's got all the features we've learned about in university neatly organized in C files. Somewhere in there is a garbage collector, it manages memory. You can spend a week or so and understand its basic workings. Making it behave slightly different for a specific use case isn't such a moon shot, I believe any smart and experienced engineer could do a proof of concept in a week or two. I'd like to flip it around. If you're coding on a platform that is so complicated that you couldn't patch it if you needed to, then you're on the wrong platform. That platform is a liability to your company, a liability that you might have to pay off by getting a support contract. That said, I'm pretty sure the Haskell runtime is very complex, so it might still be the wrong language. I'm just saying, I remember the early 2000's, and if you ran into a problem with your .Net or Java runtime, or god forbid Windows itself, you better be working at a big fat company with a nice support contract, or you'd basically be screwed.
- rkrzr 4y ago
  > That said, I'm pretty sure the Haskell runtime is very complex, so it might still be the wrong language. While the Haskell RTS as a whole is a big (and marvelous) piece of engineering, it is not actually necessary to understand all of it to contribute some smaller patches. The RTS is written in fairly readable and clean C code and it is possible to make local changes to e.g. the memory allocator without having to touch code in lots of different places.
tikhonj 4y ago
I don't think I've ever seen a large project that wasn't pushing the limits of its language in one way or another—just with popular languages, the language is treated as a given and people jump right to finding workarounds or tweaking their own designs rather than even considering contributing to GCC or Python or whatever they're using. (Not quite general-purpose languages, but don't get me started on the pain I've had with Hadoop and Hive... but trying to change those was, somehow, never on anybody's mind.)
chrisseaton 4y ago
> IMHO when you’re providing language patches you probably choose the wrong language. In your world, who does provide language patches then?
whateveracct 4y ago
It takes guts to be amazing It's fine not to have guts though But it is a lot more fun to have them :)
- whateveracct 4y ago
  On a more concrete note: Your conclusion that needing to hack on the compiler is a sign you made the wrong decision is a bit iffy I've had ghc cloned and building on my computers for years. It's easy. Seems like a cultural difference driving your opinion. Nothing more than that. Maybe that means you dislike the culture?

WJW 4y ago

That is the most business-efficient course, but it is also an excellent way to ensure you have no true expertise in your company when a gnarly problem eventually strikes. Avoiding the hardest problems does not attract the kind of people who like solving hard problems, after all.

rkrzr 4y ago

> Just like you never want to be the smartest person in a room, you never want to be the one pushing a language to its limits. (Channable co-founder here) If everybody took this view, then a language could never improve. I have actually been very impressed with how far we have been able to take Haskell before having to push its limits. Anecdotally, we previously were using Scala (and Spark) and ran into issues much earlier. In those cases we were pushing the limits of Spark, but underlying those were the limits of the JVM. > IMHO when you’re providing language patches you probably choose the wrong language. This is a very myopic view. There is much more to consider when choosing a language (what kind of team do you have, what kind of problems are you working on, maintainability, performance, ecosystem, hiring, etc.).

DonaldPShimoda 4y ago
I really appreciate your view on all this, and I totally agree with you that the other commenter has a very nearsighted perception of things. I think the go-to example of a group taking this to the extreme is Jane Street with OCaml. For many intents and purposes, Jane Street are really the major shepherds of the language these days. They contribute tons of compiler fixes and improvements, and have written their own replacement of the standard library (which is used by many people). I think it'd be silly to say that they "chose the wrong language"; rather, they've invested in a language ecosystem that benefits them and reflects their needs, while at the same time growing and improving a whole language community. That's an awesome feat.
- rkrzr 4y ago
  That's how I see it as well - Haskell has been very beneficial for us as a company, so it's nice if we can also contribute something back to the community.
- wespiser_2018 4y ago
  The difference here is scale: Jane Street is a 20 year old company with over 1000 employees, Channable has maybe a fifth of that. The proportion of engineering effort that will have to go into language expertise will just be higher at a smaller shop pushing the bounds and this will simply take away from feature development. I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. This is a huge issue for adoption: that an engineering team using Haskell will have to devote a significant amount of time/effort to tooling/resuability/infra that would otherwise be available to directly deliver value to customers if they had gone with a different language. Stuff like pagination, custom json/typescript encodings and route generation, organizing the code base so complication time isn't a nightmare, these are all "inner source" projects I've worked on that needed to be done for us to use Haskell for a decent sized team. The fact that Channable had to go down the road of ghc-compact regions, to me, suggests that on a strictly technical basis they would be much more productive using a language like Rust of C++. Better for their end users, and better for their investors. Of course, large companies can make these deep investments into languages, tooling, and infrastructure (and by all accounts Channable is well on their way) but when they do it's only a small proportion of their entire dev team. For Haskell, I contend that its' lack of state of the art library support and tooling is a massive impediment to wide scale adoption: companies just don't want to put 1/3 of a 30 person dev group on a "platform" team in order to be productive.
  - ParetoOptimal 4y ago
    > I've worked writing Haskell in multiple shops, seen it be successful, and also seen it fail to many times. Can you share some of the successes, failures, and composition of teams Haskell experience? Also any success/failure causes you believe they had in common.
  - agentultra 4y ago
lmm 4y ago
Can you talk more about what kind of limits you hit with Scala and Spark? I definitely have my own biases, but my feeling is you get 95% of the good stuff of Haskell while avoiding most of the problems mentioned elsewhere in this thread (e.g. you get a proper IDE-integrated debugger with breakpoints/stepping/etc. that Just Works).

tinco 4y ago

You could be correct, or you might have an exaggerated preconceived notion on how hard it is to provide language level patches because of how the industry was in the 90's and early 2000's. Due to chance I've worked in a company where a colleague had patched the language we were using to save some memory. Back then I (and many people in the Ruby community) considered him a rockstar for doing so. That praise definitely was deserved, since he's done all sorts of cool stuff in his career, but in retrospect that while bold, this feat wasn't all that inconceivable. The Ruby interpreter isn't some magical blackbox, it's got all the features we've learned about in university neatly organized in C files. Somewhere in there is a garbage collector, it manages memory. You can spend a week or so and understand its basic workings. Making it behave slightly different for a specific use case isn't such a moon shot, I believe any smart and experienced engineer could do a proof of concept in a week or two. I'd like to flip it around. If you're coding on a platform that is so complicated that you couldn't patch it if you needed to, then you're on the wrong platform. That platform is a liability to your company, a liability that you might have to pay off by getting a support contract. That said, I'm pretty sure the Haskell runtime is very complex, so it might still be the wrong language. I'm just saying, I remember the early 2000's, and if you ran into a problem with your .Net or Java runtime, or god forbid Windows itself, you better be working at a big fat company with a nice support contract, or you'd basically be screwed.

rkrzr 4y ago
> That said, I'm pretty sure the Haskell runtime is very complex, so it might still be the wrong language. While the Haskell RTS as a whole is a big (and marvelous) piece of engineering, it is not actually necessary to understand all of it to contribute some smaller patches. The RTS is written in fairly readable and clean C code and it is possible to make local changes to e.g. the memory allocator without having to touch code in lots of different places.

tikhonj 4y ago

I don't think I've ever seen a large project that wasn't pushing the limits of its language in one way or another—just with popular languages, the language is treated as a given and people jump right to finding workarounds or tweaking their own designs rather than even considering contributing to GCC or Python or whatever they're using. (Not quite general-purpose languages, but don't get me started on the pain I've had with Hadoop and Hive... but trying to change those was, somehow, never on anybody's mind.)

chrisseaton 4y ago

> IMHO when you’re providing language patches you probably choose the wrong language. In your world, who does provide language patches then?

whateveracct 4y ago

It takes guts to be amazing It's fine not to have guts though But it is a lot more fun to have them :)

whateveracct 4y ago
On a more concrete note: Your conclusion that needing to hack on the compiler is a sign you made the wrong decision is a bit iffy I've had ghc cloned and building on my computers for years. It's easy. Seems like a cultural difference driving your opinion. Nothing more than that. Maybe that means you dislike the culture?