5 ms·

Clojure is dynamically typed, right? Just curious: how do you refactor large codebases without a compiler telling you if the pieces still fit together? (I'm com

by manx 4y ago

Clojure is dynamically typed, right? Just curious: how do you refactor large codebases without a compiler telling you if the pieces still fit together? (I'm coming from Scala)

kaba0 4y ago
While there are some static type libs for clojure if I’m not mistaken, the standard solution is to use spec, which employs contracts. These can be used to check arbitrary things at runtime, with quite readable error messages. Also, due to the extensive use of FP and not much side effects, you can catch most of these mistakes at test time - not the same as a good static type system, but it is an interesting tradeoff, better expressivity, but not happening at compile time.
- KingMob 4y ago
  The standard advice with spec is to use it at the edges of the system, and not for every function and input. Probably because it incurs runtime overhead to use, and design overhead to apply. On top of which, it's still listed as "alpha" after 6+ years. As a runtime contract, spec can check things type systems can't, but IMO it's not pervasive enough in most code bases to really satisfy.
  - kaba0 4y ago
    Yeah, I absolutely agree. (Though perhaps using them internally as well may be okay, just disable it at prod)
lukashrb 4y ago
Could you specify, what you mean by "refactor large codebases"? > compiler telling you if the pieces still fit together? You still write tests in scala do you?
- manx 4y ago
  Since types give me guarantees that pieces still fit together, I usually only need a handful of unit tests for my non-trivial algorithms, plus some integration tests. Never for functions which hand things around. After fixing all compile errors caused by a massive code change it almost always still just works.
casion 4y ago
This is often a question asked by non-lisp devs. It's not Clojure specific - you just use the code itself. Open a REPL, and start working. You can just execute the code in realtime as you're refactoring. The size of the codebase is largely irrelevant. Ironically, in Clojure the compiler _does_ tell you what is working and what isn't when you work this way :) And anyway, you write tests don't you?
TacticalCoder 4y ago
Well ten years ago there was nothing. Nowadays there's a LSP server for Clojure and it'll tell you where, say, a function is used. It's not IntelliJ for refactoring a Java codebase but it's not either "nothing telling you if the pieces still fit together". As it's LSP, it works in any text editor supporting the LSP protocol, so it's here to stay. Besides that as others have mentioned: Clojure spec, various kind of automated testing, including, if that's your thing, generative-testing/property-based testing generated based on your specs (it's not always trivial but can really help at times).
throwamon 4y ago
This is the one question for which I still haven't understood the answers. But not just because Clojure is dynamically typed, and more because of its concept of "it's just maps". More specifically, how do you know for sure that a map that holds "everything" and has been passed around hundreds of functions will have the field you want with the shape you want at a certain point? Even with specs, how do you even reason about that?
mjul 4y ago
This is its key weakness. You have to write many more tests and add a lot of predicates as pre/post conditions (invariants) to functions to check at runtime what a more modern compiler could have told you at compile time. It adds a lot of friction and slows down the development process. Combined with Clojure having lazy evaluation and nil punning it becomes frustrating in larger, domain heavy code bases. Simple common “CRUD” line of business applications with shallow code paths and a database schema to serve as a secondary type system would probably suffer less from this, explaining why some devs say they don’t experience this. For comparison, Typescript became so popular precisely because it mitigates similar problems for JavaScript once code based grew to non-trivial sizes.
coutego 4y ago
I made some toy projects and a mid sized one in Clojure, so I'm far from an expert, but tried Clojure seriously. What you point out is exactly what I think is the main drawback as Clojure (any dynamic language, really). It gets very tedious to waste so much time refactoring code and in the increased number of unit tests you need to write in order to avoid this from happening. Personally, I think the lack of types in Clojure (and other dynamic languages) has another major drawback in that it doesn't allow you to easily express (and take advantage of) useful abstractions. In typed languages I got used to first write a "skeleton" of the application / library / module as a set of interfaces (traits, protocols, whatever expresses contracts in your language) and then develop and tests the implementations mainly independently. I find this extremely productive, in actual terms of amount of functionality per unit of time. In Clojure I find myself doing diagrams or writing down data structures, etc. because there is no practical way of doing that in the language itself. Yes, I know protocols, types, spec, malli, etc. but they are not nearly as clean and useful (IDE help, etc) as let's say Typescript types. I find this lack of expressiveness very frustrating, because I love Clojure otherwise. Persistent data structures are a game changer and the power you can get from a very small, generic API is mind blowing. And the data oriented paradigm, where everything tends to be expressed as data structures which are trivial to operate on, is simple conceptually and very powerful. I even got to like the syntax a lot. Once you get used to it, it's extremely clean, elegant and even clear (!). Being as it is, though, I'm not sure that I would chose Clojure for a large project, even if I was the only one working on it. My brain needs typed languages as soon as I have more than 200 likes of code, it seems.
Barrin92 4y ago
>telling you if the pieces still fit together? you're not supposed to do that in a dynamic language. Clojure's mentality towards data is "take what you need and ignore the rest", it's always the receiver of some data who is responsible to handle what they receive. A lot of problems people have with dynamic languages would go away if they embraced the dynamism. If you're constantly refactoring because you make assumptions about the state of the program globally you're not actually utilizing the dynamic aspect of the language. In practical terms, treat data in clojure the way you treat data over the web. When you get a some json data over a network you don't necessarily make absolute assumptions over its structure, you take what you need and deal with it when you receive it.
kamma4434 4y ago
You don’t. Not in the Java sense, where you can change a method’s name and it will just work, or a method signature and you will immediately spot the 14 places you need to update.
yogthos 4y ago
Any large codebase can be broken up into small isolated components that can be reasoned about independently. This is how you structure Clojure projects if you want them to be maintainable. Clojure inherently encourages doing this by defaulting to immutability. The contract between components is the data being passed to the component and returned by it. Using Malli schemas at the edges of the components is a typical approach to documenting their APIs https://github.com/metosin/malli https://github.com/metosin/malli I see the fact that people often end up creating large and tightly coupled monolithic codebases in static languages as a negative aspect of static typing. Such codebases are difficult to reason about even if you have guarantees that the types align. Ultimately, you need to understand the relationships in code, and how they relate to business logic. The more coupling an application has the harder it becomes to reason about it as a whole. Ideally, I think applications should be structured as a bunch of Lego blocks that can be composed together. Each component should encapsulate some functionality, and then the flow of the business logic should bubble up to the top and expressed in how these components are chained together.

by manx 4y ago

Clojure is dynamically typed, right? Just curious: how do you refactor large codebases without a compiler telling you if the pieces still fit together? (I'm coming from Scala)

kaba0 4y ago

While there are some static type libs for clojure if I’m not mistaken, the standard solution is to use spec, which employs contracts. These can be used to check arbitrary things at runtime, with quite readable error messages. Also, due to the extensive use of FP and not much side effects, you can catch most of these mistakes at test time - not the same as a good static type system, but it is an interesting tradeoff, better expressivity, but not happening at compile time.

KingMob 4y ago
The standard advice with spec is to use it at the edges of the system, and not for every function and input. Probably because it incurs runtime overhead to use, and design overhead to apply. On top of which, it's still listed as "alpha" after 6+ years. As a runtime contract, spec can check things type systems can't, but IMO it's not pervasive enough in most code bases to really satisfy.
- kaba0 4y ago
  Yeah, I absolutely agree. (Though perhaps using them internally as well may be okay, just disable it at prod)

lukashrb 4y ago

Could you specify, what you mean by "refactor large codebases"? > compiler telling you if the pieces still fit together? You still write tests in scala do you?

manx 4y ago
Since types give me guarantees that pieces still fit together, I usually only need a handful of unit tests for my non-trivial algorithms, plus some integration tests. Never for functions which hand things around. After fixing all compile errors caused by a massive code change it almost always still just works.

casion 4y ago

This is often a question asked by non-lisp devs. It's not Clojure specific - you just use the code itself. Open a REPL, and start working. You can just execute the code in realtime as you're refactoring. The size of the codebase is largely irrelevant. Ironically, in Clojure the compiler _does_ tell you what is working and what isn't when you work this way :) And anyway, you write tests don't you?

TacticalCoder 4y ago

Well ten years ago there was nothing. Nowadays there's a LSP server for Clojure and it'll tell you where, say, a function is used. It's not IntelliJ for refactoring a Java codebase but it's not either "nothing telling you if the pieces still fit together". As it's LSP, it works in any text editor supporting the LSP protocol, so it's here to stay. Besides that as others have mentioned: Clojure spec, various kind of automated testing, including, if that's your thing, generative-testing/property-based testing generated based on your specs (it's not always trivial but can really help at times).

throwamon 4y ago

This is the one question for which I still haven't understood the answers. But not just because Clojure is dynamically typed, and more because of its concept of "it's just maps". More specifically, how do you know for sure that a map that holds "everything" and has been passed around hundreds of functions will have the field you want with the shape you want at a certain point? Even with specs, how do you even reason about that?

mjul 4y ago

This is its key weakness. You have to write many more tests and add a lot of predicates as pre/post conditions (invariants) to functions to check at runtime what a more modern compiler could have told you at compile time. It adds a lot of friction and slows down the development process. Combined with Clojure having lazy evaluation and nil punning it becomes frustrating in larger, domain heavy code bases. Simple common “CRUD” line of business applications with shallow code paths and a database schema to serve as a secondary type system would probably suffer less from this, explaining why some devs say they don’t experience this. For comparison, Typescript became so popular precisely because it mitigates similar problems for JavaScript once code based grew to non-trivial sizes.

coutego 4y ago

I made some toy projects and a mid sized one in Clojure, so I'm far from an expert, but tried Clojure seriously. What you point out is exactly what I think is the main drawback as Clojure (any dynamic language, really). It gets very tedious to waste so much time refactoring code and in the increased number of unit tests you need to write in order to avoid this from happening. Personally, I think the lack of types in Clojure (and other dynamic languages) has another major drawback in that it doesn't allow you to easily express (and take advantage of) useful abstractions. In typed languages I got used to first write a "skeleton" of the application / library / module as a set of interfaces (traits, protocols, whatever expresses contracts in your language) and then develop and tests the implementations mainly independently. I find this extremely productive, in actual terms of amount of functionality per unit of time. In Clojure I find myself doing diagrams or writing down data structures, etc. because there is no practical way of doing that in the language itself. Yes, I know protocols, types, spec, malli, etc. but they are not nearly as clean and useful (IDE help, etc) as let's say Typescript types. I find this lack of expressiveness very frustrating, because I love Clojure otherwise. Persistent data structures are a game changer and the power you can get from a very small, generic API is mind blowing. And the data oriented paradigm, where everything tends to be expressed as data structures which are trivial to operate on, is simple conceptually and very powerful. I even got to like the syntax a lot. Once you get used to it, it's extremely clean, elegant and even clear (!). Being as it is, though, I'm not sure that I would chose Clojure for a large project, even if I was the only one working on it. My brain needs typed languages as soon as I have more than 200 likes of code, it seems.

Barrin92 4y ago

>telling you if the pieces still fit together? you're not supposed to do that in a dynamic language. Clojure's mentality towards data is "take what you need and ignore the rest", it's always the receiver of some data who is responsible to handle what they receive. A lot of problems people have with dynamic languages would go away if they embraced the dynamism. If you're constantly refactoring because you make assumptions about the state of the program globally you're not actually utilizing the dynamic aspect of the language. In practical terms, treat data in clojure the way you treat data over the web. When you get a some json data over a network you don't necessarily make absolute assumptions over its structure, you take what you need and deal with it when you receive it.

kamma4434 4y ago

You don’t. Not in the Java sense, where you can change a method’s name and it will just work, or a method signature and you will immediately spot the 14 places you need to update.

yogthos 4y ago

Any large codebase can be broken up into small isolated components that can be reasoned about independently. This is how you structure Clojure projects if you want them to be maintainable. Clojure inherently encourages doing this by defaulting to immutability. The contract between components is the data being passed to the component and returned by it. Using Malli schemas at the edges of the components is a typical approach to documenting their APIs https://github.com/metosin/malli https://github.com/metosin/malli I see the fact that people often end up creating large and tightly coupled monolithic codebases in static languages as a negative aspect of static typing. Such codebases are difficult to reason about even if you have guarantees that the types align. Ultimately, you need to understand the relationships in code, and how they relate to business logic. The more coupling an application has the harder it becomes to reason about it as a whole. Ideally, I think applications should be structured as a bunch of Lego blocks that can be composed together. Each component should encapsulate some functionality, and then the flow of the business logic should bubble up to the top and expressed in how these components are chained together.