4 ms·

It's a lot of fun but as projects got larger and larger for me (thousands of lines, or even tens of thousands), I found the dynamic typing taking up more and mo

by jb1991 7y ago

It's a lot of fun but as projects got larger and larger for me (thousands of lines, or even tens of thousands), I found the dynamic typing taking up more and more of my time. I've since moved on to statically typed systems where the compiler takes a big load off the cognitive requirements of maintaining and debugging software.

kickopotomus 7y ago
This has been my experience as well. I have a couple of Clojure(Script) applications that are approximately 3-5K lines each and those have been a pleasure too work on. However, my latest project is now pushing past 20K lines and the mental load has jumped exponentially. There is a much greater need for spec, asserts, type hints, and comments just to keep everything straight.
- slifin 7y ago
  That's interesting do you have any idea why the mental load jumped? would a static analysis tool working with your type hints help? Or is there many things to consider at once in the system instead of many individual things?
  - kickopotomus 7y ago
    The biggest issue I keep running into is just the concept of data "shape". I love that clojure gives you so much freedom but it can be quite the footgun in a large system because you see that a function expects a map with keys :foo, :bar, and :baz but what are the values for those keys? Spec helps a little bit here for primitive values but for complex nested structures (e.g. {:a [{:b [1 2]} {:c "bar"}]}), it doesn't do much. So, as data moves through the system, and as the system grows it has become increasingly difficult to track the mutations to the underlying structures. I do think that a static analysis tool would be of some help. Some sort of tooling to better handle tree structures would be very handy. I often find myself being off-by-one level with get-in calls on tree data (E.g. (get-in m [:a :b :d]) where m is {:a {:b {:c {:d 1}}}}), which is annoying because the NPE gets thrown 3 function calls up the stack.
    - sooheon 7y ago
      Totally not saying "you're holding it wrong", but maybe once you're more than a few couple levels deep into a nested map it's time to look at an in memory db like [Datascript](https://github.com/tonsky/datascript https://github.com/tonsky/datascript)? Actual Datalog queries can replace get-in vectors growing out of hand, and you also get better mutations with transactions.
    - escherize 7y ago
      I'm late to the thread, but I've felt the same pain and wrote a library to help deal with the cognitive load of working on "shapes" of data [1]. [1] https://github.com/escherize/tracks https://github.com/escherize/tracks
didibus 7y ago
How long have you been programming professionally? Did you have prior experience with dynamic languages? If so, how much? Did you have prior experience with statically typed languages? If so, how much? I'm trying to see if Clojure requires more fundamental programming intuition and experience to sustain in large projects. I work with tens of thousands of Clojure LOC, I don't feel these issues and can't relate, and love Clojure, so I'm curious to understand what context it best applies too. I wouldn't want to force it on a team that wouldn't benefit from it, so I'm interested about learning these aspects, so I'm able to recognize in what context it would make sense for me to influence a team to adopt it or not. Thanks.
- dkersten 7y ago
  12ish years (more as student & hobbyist), lots (mainly python, javascript and clojure), lots (mainly Java and C++, sadly inly got to tinker with better type systems). I consider myself a clojure developer and have been using it on and off since summer 09. Im using it for a large project now. And yet... I agree with GP. I love clojure, but over time I’m becoming less and less sold as dynamic typing. Spec helps, a little. Property-based generative tests (especially when used with spec) helps a little too. Neither are a replacement for proper static types, though, especially an ML-esque type system with type inference. Bonus points if spec validation could pass type data to the compiler/type inference (eg the code path after validation can assume that the data is of the the types described in the spec). I dream of a statically typed Clojure with type inference, spec-inferred types and optional dynamic typing for REPL experimentation and glue code.
  - didibus 7y ago
    Very interesting, and thanks for answering. Could I ask what kind of project it is? I wonder if I'm just lucky that Clojure fits perfectly my use case, which is mostly a set of distributed systems of all kinds. So while as a whole there's tens of thousands of LOC. The components have very strong boundaries being as it's a set of services assembled together through RPC, PubSubs and DBs. Maybe that alleviate the lack of a static type checker. We've adopted Clojure about 3 years ago, team of 10. We've had a few people leave and join throughout. Only one person knew Clojure beforehand. Our stack is about 50% Clojure, 40% Java and 10% Scala. Of all three, Clojure has given us the least issues, has been pretty easy to maintain and generally has fewer defects. Java tend to have the most bugs, almost always related to some shared state. Scala I find the hardest to extend and maintain, but the code base we have for it I think does Scala wrong, it's like the worst mix of OOP and FP. > I dream of a statically typed Clojure with type inference, spec-inferred types and optional dynamic typing for REPL experimentation and glue code That's pretty much exactly core.typed: https://blog.ambrosebs.com/2018/09/20/towards-typed-clj.html https://blog.ambrosebs.com/2018/09/20/towards-typed-clj.html That said, the project never managed to get more contributors. If you're looking for a typed Lisp, I've been keeping my eyes out on Carp: https://github.com/carp-lang/Carp https://github.com/carp-lang/Carp

by jb1991 7y ago

kickopotomus 7y ago

This has been my experience as well. I have a couple of Clojure(Script) applications that are approximately 3-5K lines each and those have been a pleasure too work on. However, my latest project is now pushing past 20K lines and the mental load has jumped exponentially. There is a much greater need for spec, asserts, type hints, and comments just to keep everything straight.

slifin 7y ago
That's interesting do you have any idea why the mental load jumped? would a static analysis tool working with your type hints help? Or is there many things to consider at once in the system instead of many individual things?
- kickopotomus 7y ago
  The biggest issue I keep running into is just the concept of data "shape". I love that clojure gives you so much freedom but it can be quite the footgun in a large system because you see that a function expects a map with keys :foo, :bar, and :baz but what are the values for those keys? Spec helps a little bit here for primitive values but for complex nested structures (e.g. {:a [{:b [1 2]} {:c "bar"}]}), it doesn't do much. So, as data moves through the system, and as the system grows it has become increasingly difficult to track the mutations to the underlying structures. I do think that a static analysis tool would be of some help. Some sort of tooling to better handle tree structures would be very handy. I often find myself being off-by-one level with get-in calls on tree data (E.g. (get-in m [:a :b :d]) where m is {:a {:b {:c {:d 1}}}}), which is annoying because the NPE gets thrown 3 function calls up the stack.
  - sooheon 7y ago
    Totally not saying "you're holding it wrong", but maybe once you're more than a few couple levels deep into a nested map it's time to look at an in memory db like [Datascript](https://github.com/tonsky/datascript https://github.com/tonsky/datascript)? Actual Datalog queries can replace get-in vectors growing out of hand, and you also get better mutations with transactions.
  - escherize 7y ago
    I'm late to the thread, but I've felt the same pain and wrote a library to help deal with the cognitive load of working on "shapes" of data [1]. [1] https://github.com/escherize/tracks https://github.com/escherize/tracks

didibus 7y ago

How long have you been programming professionally? Did you have prior experience with dynamic languages? If so, how much? Did you have prior experience with statically typed languages? If so, how much? I'm trying to see if Clojure requires more fundamental programming intuition and experience to sustain in large projects. I work with tens of thousands of Clojure LOC, I don't feel these issues and can't relate, and love Clojure, so I'm curious to understand what context it best applies too. I wouldn't want to force it on a team that wouldn't benefit from it, so I'm interested about learning these aspects, so I'm able to recognize in what context it would make sense for me to influence a team to adopt it or not. Thanks.

dkersten 7y ago
12ish years (more as student & hobbyist), lots (mainly python, javascript and clojure), lots (mainly Java and C++, sadly inly got to tinker with better type systems). I consider myself a clojure developer and have been using it on and off since summer 09. Im using it for a large project now. And yet... I agree with GP. I love clojure, but over time I’m becoming less and less sold as dynamic typing. Spec helps, a little. Property-based generative tests (especially when used with spec) helps a little too. Neither are a replacement for proper static types, though, especially an ML-esque type system with type inference. Bonus points if spec validation could pass type data to the compiler/type inference (eg the code path after validation can assume that the data is of the the types described in the spec). I dream of a statically typed Clojure with type inference, spec-inferred types and optional dynamic typing for REPL experimentation and glue code.
- didibus 7y ago
  Very interesting, and thanks for answering. Could I ask what kind of project it is? I wonder if I'm just lucky that Clojure fits perfectly my use case, which is mostly a set of distributed systems of all kinds. So while as a whole there's tens of thousands of LOC. The components have very strong boundaries being as it's a set of services assembled together through RPC, PubSubs and DBs. Maybe that alleviate the lack of a static type checker. We've adopted Clojure about 3 years ago, team of 10. We've had a few people leave and join throughout. Only one person knew Clojure beforehand. Our stack is about 50% Clojure, 40% Java and 10% Scala. Of all three, Clojure has given us the least issues, has been pretty easy to maintain and generally has fewer defects. Java tend to have the most bugs, almost always related to some shared state. Scala I find the hardest to extend and maintain, but the code base we have for it I think does Scala wrong, it's like the worst mix of OOP and FP. > I dream of a statically typed Clojure with type inference, spec-inferred types and optional dynamic typing for REPL experimentation and glue code That's pretty much exactly core.typed: https://blog.ambrosebs.com/2018/09/20/towards-typed-clj.html https://blog.ambrosebs.com/2018/09/20/towards-typed-clj.html That said, the project never managed to get more contributors. If you're looking for a typed Lisp, I've been keeping my eyes out on Carp: https://github.com/carp-lang/Carp https://github.com/carp-lang/Carp