6 ms·

I really wish that Erlang/Elixir were strongly typed. I've blown my foot off with the "dynamic language gets us going fast and now let's scale" footgun too many

I really wish that Erlang/Elixir were strongly typed. I've blown my foot off with the "dynamic language gets us going fast and now let's scale" footgun too many times to not be afraid. Obviously big important systems are built in Erlang, but I don't think I've got the guts to do it.

bfrydl 8y ago
For what it's worth, while I don't really disagree, Erlang's pattern matching is very powerful and covers a lot of the same ground as static typing. Erlang is not in my opinion a “dynamic language gets us going fast” language. It's a language specifically designed to handle large scale reliably for long periods of time.
- zaphar 8y ago
  I think Erlang and possibly Elixir get about as close as you can get to having a rock solid reliable service in a dynamic language. Largely due to the pattern matching and the emphasis on coding the happy path and letting the non-happy path crash with a supervisor restarting it and logging the crashes for later analysis. You will get a lot of milage out the Erlang ecosystem.
dmitriid 8y ago
Phillip Wadler tried to add static typing to Erlang in the 90s. He could only make it work for a subset of the language. IIRC message passing greatly complicates things.
- dev_dull 8y ago
  Static types with message passing works great and is used in production at any company which uses Akka typed.
  - barrkel 8y ago
    Static types in a distributed system with hot swapping, live upgrades of parts of a running system, isn't trivial. Types are great at stating and preserving global invariants. Invariants sometimes do vary as systems evolve, though. You may need to handle polymorphism in your data flow mid-transition. And runtime polymorphism is just another phrase for dynamic typing.
  - rad_gruchalski 8y ago
    Yes and no. It works great in Scala thanks to extractors but Java is a bit of a mess, example: https://github.com/patriknw/akka-typed-blog/blob/master/src/main/java/blog/typed/javadsl/Greeter1.java https://github.com/patriknw/akka-typed-blog/blob/master/src/...
- YorkshireSeason 8y ago
  The problem all extant typing systems for message passing (in particular session types) have is that they cannot really deal well with data dependent patterns of message passing. Erlang's mailbox-based message passing is especially prone to data dependent patterns of message passing. Yes, you can add enough expressive power to the typing system to make this work (e.g. dependent types), but then it stops being pragmatically viable, as far as we know in 2019.
  - AnaniasAnanas 8y ago
    Claiming that dependent types are not pragmatically viable is quite a bold claim, would you mind backing it up? My experience with them so far has been more than great.
    - YorkshireSeason 8y ago
      Nothing bold about it, here is some evidence. - No type inference, since that's not decidable for dependent types. - In most industrial software engineering, getting the specifications is a big problem, whence agile methods. Without detailed specifications early on in a software project's lifecycle, what's the point of paying the price for type dependency when you don't even use it. - Relatedly, dependent types don't solve the oracle problem in software engineering: i.e. where do you get the specs from and how do you know that your specs (in this case types) are correct rather than false? - Rewriting & refactoring code becomes much more expensive, because you now also have to rewrite / refactor the specifications. (This is already a reason why post-Java, exception specifications have been abandoned) - No widely used programming language offers full dependent types (yes I'm aware of Scala's path dependent types and Haskell's forays into dependency), and that despite dependent types are at least half a century old [1]. So it's not like programming language designers don't know about them. - Dependent types have really only been worked out beyond research prototypes for functional languages, not e.g. for message passing concurrency. My experience with them so far has been more than great. I'd be interested to learn what non-trivial code you've written in dependently typed programming languages. By non-trivial, I mean: industrial code, say > 20k LoCs, at least 3 programmers involved, specification changed over time. Specification was not fully available at the start of the project. [1] N. de Bruijn, Automath, a language for mathematics.
sacado2 8y ago
Erlang's niche is distributed systems anyway, an area where static typing is at odds more often than not. I mean, you can guarantee the executable you are working on is exempt of typing errors, but it's just a small component of the overall architecture. What about data you are receiving? What if some remote client is using an older version of your software? What if the web service you are calling changed its API recently? I use Elm, which is as statically-typed as possible for a client webapp, but I know I can never guarantee that JSON data I receive from remote servers is of the expected type.
- mcintyre1994 8y ago
  I find static typing is extremely useful in the case where you're receiving JSON though - I write the server side and we use Scala. For us we write the type we expect, validate the input as soon as it hits the server (in the controller), fail immediately if it doesn't validate and if it does validate then we're dealing with that type in all the rest of the code. Obviously that's not going to handle receiving a shape of data that you don't expect, but you'll error immediately and I find it really robust in practice.
  - fulafel 8y ago
    Dynamic languages do this with schema validation. It can be more expressive, checking value ranges or string formats, multi field conditional constraints, and so on. And allows better& customizable diagnostics. Edit: also, versioning, cross language sharing, use in generative testing, api client code generation and other metaprogramming uses. And documentation generation, eg swagger
    - guitarbill 8y ago
      See e.g. JSON schema, which has a wide language support and tooling (even some IDEs support it for validation while you're writing a compliant document)
    - roca 8y ago
      and once you've validated a value your code immediately forgets what type it is.
lugg 8y ago
The guts or the discipline? Dynamic doesn't really cause problems in its own. Only abuse of it which can be said about most things. I guess there is an argument that f forces you to have good habits but that comes at a pretty big cost if your team is half decent.
- Someone1234 8y ago
  "Just write good code" or "Just hire perfect programmers" the go to retort for any language weakness. I've been in this industry long enough to just roll my eyes every time I hear that now, it simply doesn't scale beyond one individual.
  - lugg 8y ago
    Good thing I didn't say or imply either of those then. I said don't abuse language features. That is a far cry from writing perfect code. Non perfect code is expected early in a startup, I'm just saying don't confuse your non perfect code with a language failure. I'd also like to say if you've never seen "hire perfect developers" scale passed one you've been optimising for the wrong thing. I've seen a team go from technical debt hell and dread to a powerhouse team where everyone commits daily. It doesn't take much, and it doesn't take perfect developers, and it certainly doesn't have anything to do with the language. It had everything to do with honest and healthy code review. It had everything to do with people feeling comfortable enough to say, your solution is a hack, why are you taking this shortcut. That isn't write perfect code, or hire the right people. That is hire pretty much whoever will take the job. That is prioritising and optimising for quality. We still had a legacy code base, it still made us money, and it gave us the time opportunity to do things right, but you still need to take the time.
- biesnecker 8y ago
  My team might be half decent. So might the dozens of other teams working on the code base over years and years. Everyone will certainly have the best of intentions. But mistakes will certainly be made. Misunderstandings will occur. Note I said "now let's scale" -- it's pretty easy to rely on good practices and a steady hand when it's a dozen engineers in a room. When it's a thousand engineers in three timezones it's a fair bit harder. Edit: typo
rdtsc 8y ago
It had type specification for years: https://learnyousomeerlang.com/dialyzer https://learnyousomeerlang.com/dialyzer They are opt in of course, but have the property that the more specifications you add and the more precise they are, the more type errors they'd discover. After it run it tells you if there is a type discrepancy. If it can't decide it doesn't say anything. The technical term for that is "success typing" http://www.it.uu.se/research/group/hipe/papers/succ_types.pdf http://www.it.uu.se/research/group/hipe/papers/succ_types.pd...
- convolvatron 8y ago
  is this exactly the same as 'soft types' (http://wiki.c2.com/?SoftTyping http://wiki.c2.com/?SoftTyping) or subtly different?
  - ianleeclark 8y ago
    I write elixir professionally and that's how I'd describe the type specification system.
tniemi 8y ago
Isn't Erlang the language that you can edit/patch while it is executing? I think hard typing would be extremely difficult in that kind of environment.
- abstractbeliefs 8y ago
  It is, but you can still running static analysis tools on written code before executing a code switch. The key thing is that you track the changes applied so you understand what's happening. The nice thing about erlang though is that the nature of the message passing interface and pattern matching means that you can get relatively understandable interfaces.
- koala_man 8y ago
  Someone should correct me if I'm wrong, but I don't think that this is a special feature of Erlang. It's done by writing your handlers to accept a closure to use for future requests. It's not inherently different from doing this in Java: if (req.isUpdate()) { handler = Class.forName(req.getUpdateName()).newInstance(); } else { handler.handle(req); } It's way easier when you have message passing and every level is set up to restart on crashes, but I don't think anything necessarily stops you from doing the same in any other language.
  - bitwalker 8y ago
    A bigger problem is this: - Given a distributed Erlang system with two nodes, where both nodes are running the same release; - Given the same process (Erlang thread) on both nodes, running the same code (i.e. a GenServer backed by the same module) - Given a new version of the release is being applied You can't assume that the two processes will be upgraded at the same time. This means that messages sent between those processes may violate the types expected by one version or the other. Furthermore, in Erlang, any process can send a message to another process, so there is no way to enforce that the data a particular `receive` expression gets will even be a type defined in the code it is running. Furthermore, even on the same node, during an upgrade, some parts of the system are still running old code, while some parts are running new code, so it isn't even specifically about distribution. There is a whole area of type system research around session types, which are designed for more or less this use case, but from what I've seen, none of them handle the case of arbitrary messages, or the case where system upgrades are being rolled out and old code may receive messages from new code. I still think there is a place for a type system in Erlang/Elixir, but it would have to deliberately ignore process messaging at the very least, at least until a type theoretic solution is available.
mercer 8y ago
I suppose the 'Let it crash' characteristic of the BEAM kind of compensates for a lack of a strongly typed approach, if the concern is stability and being able to stay up despite triggering a bug.

bfrydl 8y ago

For what it's worth, while I don't really disagree, Erlang's pattern matching is very powerful and covers a lot of the same ground as static typing. Erlang is not in my opinion a “dynamic language gets us going fast” language. It's a language specifically designed to handle large scale reliably for long periods of time.

zaphar 8y ago
I think Erlang and possibly Elixir get about as close as you can get to having a rock solid reliable service in a dynamic language. Largely due to the pattern matching and the emphasis on coding the happy path and letting the non-happy path crash with a supervisor restarting it and logging the crashes for later analysis. You will get a lot of milage out the Erlang ecosystem.

dmitriid 8y ago

Phillip Wadler tried to add static typing to Erlang in the 90s. He could only make it work for a subset of the language. IIRC message passing greatly complicates things.

dev_dull 8y ago
Static types with message passing works great and is used in production at any company which uses Akka typed.
- barrkel 8y ago
  Static types in a distributed system with hot swapping, live upgrades of parts of a running system, isn't trivial. Types are great at stating and preserving global invariants. Invariants sometimes do vary as systems evolve, though. You may need to handle polymorphism in your data flow mid-transition. And runtime polymorphism is just another phrase for dynamic typing.
- rad_gruchalski 8y ago
  Yes and no. It works great in Scala thanks to extractors but Java is a bit of a mess, example: https://github.com/patriknw/akka-typed-blog/blob/master/src/main/java/blog/typed/javadsl/Greeter1.java https://github.com/patriknw/akka-typed-blog/blob/master/src/...
YorkshireSeason 8y ago
The problem all extant typing systems for message passing (in particular session types) have is that they cannot really deal well with data dependent patterns of message passing. Erlang's mailbox-based message passing is especially prone to data dependent patterns of message passing. Yes, you can add enough expressive power to the typing system to make this work (e.g. dependent types), but then it stops being pragmatically viable, as far as we know in 2019.
- AnaniasAnanas 8y ago
  Claiming that dependent types are not pragmatically viable is quite a bold claim, would you mind backing it up? My experience with them so far has been more than great.
  - YorkshireSeason 8y ago
    Nothing bold about it, here is some evidence. - No type inference, since that's not decidable for dependent types. - In most industrial software engineering, getting the specifications is a big problem, whence agile methods. Without detailed specifications early on in a software project's lifecycle, what's the point of paying the price for type dependency when you don't even use it. - Relatedly, dependent types don't solve the oracle problem in software engineering: i.e. where do you get the specs from and how do you know that your specs (in this case types) are correct rather than false? - Rewriting & refactoring code becomes much more expensive, because you now also have to rewrite / refactor the specifications. (This is already a reason why post-Java, exception specifications have been abandoned) - No widely used programming language offers full dependent types (yes I'm aware of Scala's path dependent types and Haskell's forays into dependency), and that despite dependent types are at least half a century old [1]. So it's not like programming language designers don't know about them. - Dependent types have really only been worked out beyond research prototypes for functional languages, not e.g. for message passing concurrency. My experience with them so far has been more than great. I'd be interested to learn what non-trivial code you've written in dependently typed programming languages. By non-trivial, I mean: industrial code, say > 20k LoCs, at least 3 programmers involved, specification changed over time. Specification was not fully available at the start of the project. [1] N. de Bruijn, Automath, a language for mathematics.

sacado2 8y ago

Erlang's niche is distributed systems anyway, an area where static typing is at odds more often than not. I mean, you can guarantee the executable you are working on is exempt of typing errors, but it's just a small component of the overall architecture. What about data you are receiving? What if some remote client is using an older version of your software? What if the web service you are calling changed its API recently? I use Elm, which is as statically-typed as possible for a client webapp, but I know I can never guarantee that JSON data I receive from remote servers is of the expected type.

mcintyre1994 8y ago
I find static typing is extremely useful in the case where you're receiving JSON though - I write the server side and we use Scala. For us we write the type we expect, validate the input as soon as it hits the server (in the controller), fail immediately if it doesn't validate and if it does validate then we're dealing with that type in all the rest of the code. Obviously that's not going to handle receiving a shape of data that you don't expect, but you'll error immediately and I find it really robust in practice.
- fulafel 8y ago
  Dynamic languages do this with schema validation. It can be more expressive, checking value ranges or string formats, multi field conditional constraints, and so on. And allows better& customizable diagnostics. Edit: also, versioning, cross language sharing, use in generative testing, api client code generation and other metaprogramming uses. And documentation generation, eg swagger
  - guitarbill 8y ago
    See e.g. JSON schema, which has a wide language support and tooling (even some IDEs support it for validation while you're writing a compliant document)
  - roca 8y ago
    and once you've validated a value your code immediately forgets what type it is.

lugg 8y ago

The guts or the discipline? Dynamic doesn't really cause problems in its own. Only abuse of it which can be said about most things. I guess there is an argument that f forces you to have good habits but that comes at a pretty big cost if your team is half decent.

Someone1234 8y ago
"Just write good code" or "Just hire perfect programmers" the go to retort for any language weakness. I've been in this industry long enough to just roll my eyes every time I hear that now, it simply doesn't scale beyond one individual.
- lugg 8y ago
  Good thing I didn't say or imply either of those then. I said don't abuse language features. That is a far cry from writing perfect code. Non perfect code is expected early in a startup, I'm just saying don't confuse your non perfect code with a language failure. I'd also like to say if you've never seen "hire perfect developers" scale passed one you've been optimising for the wrong thing. I've seen a team go from technical debt hell and dread to a powerhouse team where everyone commits daily. It doesn't take much, and it doesn't take perfect developers, and it certainly doesn't have anything to do with the language. It had everything to do with honest and healthy code review. It had everything to do with people feeling comfortable enough to say, your solution is a hack, why are you taking this shortcut. That isn't write perfect code, or hire the right people. That is hire pretty much whoever will take the job. That is prioritising and optimising for quality. We still had a legacy code base, it still made us money, and it gave us the time opportunity to do things right, but you still need to take the time.
biesnecker 8y ago
My team might be half decent. So might the dozens of other teams working on the code base over years and years. Everyone will certainly have the best of intentions. But mistakes will certainly be made. Misunderstandings will occur. Note I said "now let's scale" -- it's pretty easy to rely on good practices and a steady hand when it's a dozen engineers in a room. When it's a thousand engineers in three timezones it's a fair bit harder. Edit: typo

rdtsc 8y ago

It had type specification for years: https://learnyousomeerlang.com/dialyzer https://learnyousomeerlang.com/dialyzer They are opt in of course, but have the property that the more specifications you add and the more precise they are, the more type errors they'd discover. After it run it tells you if there is a type discrepancy. If it can't decide it doesn't say anything. The technical term for that is "success typing" http://www.it.uu.se/research/group/hipe/papers/succ_types.pdf http://www.it.uu.se/research/group/hipe/papers/succ_types.pd...

convolvatron 8y ago
is this exactly the same as 'soft types' (http://wiki.c2.com/?SoftTyping http://wiki.c2.com/?SoftTyping) or subtly different?
- ianleeclark 8y ago
  I write elixir professionally and that's how I'd describe the type specification system.

tniemi 8y ago

Isn't Erlang the language that you can edit/patch while it is executing? I think hard typing would be extremely difficult in that kind of environment.

abstractbeliefs 8y ago
It is, but you can still running static analysis tools on written code before executing a code switch. The key thing is that you track the changes applied so you understand what's happening. The nice thing about erlang though is that the nature of the message passing interface and pattern matching means that you can get relatively understandable interfaces.
koala_man 8y ago
Someone should correct me if I'm wrong, but I don't think that this is a special feature of Erlang. It's done by writing your handlers to accept a closure to use for future requests. It's not inherently different from doing this in Java: if (req.isUpdate()) { handler = Class.forName(req.getUpdateName()).newInstance(); } else { handler.handle(req); } It's way easier when you have message passing and every level is set up to restart on crashes, but I don't think anything necessarily stops you from doing the same in any other language.
- bitwalker 8y ago
  A bigger problem is this: - Given a distributed Erlang system with two nodes, where both nodes are running the same release; - Given the same process (Erlang thread) on both nodes, running the same code (i.e. a GenServer backed by the same module) - Given a new version of the release is being applied You can't assume that the two processes will be upgraded at the same time. This means that messages sent between those processes may violate the types expected by one version or the other. Furthermore, in Erlang, any process can send a message to another process, so there is no way to enforce that the data a particular `receive` expression gets will even be a type defined in the code it is running. Furthermore, even on the same node, during an upgrade, some parts of the system are still running old code, while some parts are running new code, so it isn't even specifically about distribution. There is a whole area of type system research around session types, which are designed for more or less this use case, but from what I've seen, none of them handle the case of arbitrary messages, or the case where system upgrades are being rolled out and old code may receive messages from new code. I still think there is a place for a type system in Erlang/Elixir, but it would have to deliberately ignore process messaging at the very least, at least until a type theoretic solution is available.

mercer 8y ago

I suppose the 'Let it crash' characteristic of the BEAM kind of compensates for a lack of a strongly typed approach, if the concern is stability and being able to stay up despite triggering a bug.