4 ms·

Are there any technical posts from Nubank on how they manage such a large Clojure codebase (I guess it must be Clojure's largest user in the industry). I'm loo

Are there any technical posts from Nubank on how they manage such a large Clojure codebase (I guess it must be Clojure's largest user in the industry). I'm looking for things like how to refactor safely in the absence of static typing, for example. Do they make use of spec? I'm a big fan of static typing, but when using a dynamic language with immutable values and pattern matching, like Erlang, I don't miss it that much. Maybe with Clojure is the same?

hcarvalhoalves 6y ago
Not sure we have any talks about that specifically, but what I can say is that on the Clojure codebases: - We rely heavily on testing (unit tests, property-based/generative and integration) - We adopt a micro-services architecture, so most codebases are small enough that refactoring is easy, or sometimes we just replace the service altogether - We leverage Schema (https://github.com/plumatic/schema https://github.com/plumatic/schema) and/or clojure.spec (https://clojure.org/guides/spec https://clojure.org/guides/spec) to annotate functions and data schemas, but it's opt-in - We have static checking of data schemas across boundaries (e.g. checking data producers did not break consumers over HTTP or Kafka), which is where we found the most value (replaced complicated end-to-end tests)
- andrenth 6y ago
  Thanks! Can you talk a bit more about the static checking of cross-boundaries schemas? Any relation to tools that allow a schema to evolve without breaking compatibility, like protobuf?
  - hcarvalhoalves 6y ago
    It's not based on protobuf. There's some in-house tooling that checks the schemas of producers against consumers against known versions (aka the versions in production), and we run it as part of the CI. Since services implement standard components/protocols there are standardised hooks to extract these schemas definitions from each HTTP endpoint or Kafka consumer.
    - andrenth 6y ago
      Does this rely on code generators or maybe a schema generator from code? I’m curious about how schema and code are kept in sync.
      - hcarvalhoalves 6y ago
        Yes. The schema definition is part of the code/data itself, so you can reference a particular schema by its git rev hash.
- stopachka 6y ago
  Curious, did you really find that microservices helped? The downsides can be so tremendous -- I wonder if nubank would have done better to services in the low digit range. (context: my thoughts on it -> https://stopa.io/post/236 https://stopa.io/post/236)
  - hcarvalhoalves 6y ago
    I believe micro-services change the trade-offs, and it's a matter of what you value more. We certainly have to invest more on infra-structure, monitoring, standardisation, and we have been doing since day one - by now it's pretty robust, to the point our internal tooling allowed us to migrate from raw EC2 to kubernetes (https://www.cncf.io/case-study/nubank/ https://www.cncf.io/case-study/nubank/) and it was a transparent change to all services and teams. On the other hand, failures are more localised and we gain robustness and speed to do certain kind of changes. Your point about getting domain boundaries wrong is true, but IMO it's a pain orthogonal to micro-services. The solution is the same as if you defined a sub-optimal class interface on a big mono-repo: relentless refactoring. We are often strangling (https://martinfowler.com/bliki/StranglerFigApplication.html https://martinfowler.com/bliki/StranglerFigApplication.html) and replacing legacy services.
- hestefisk 6y ago
  Questions: transactions are really important in banking systems. If you have 100s of Microservices how do you manage transactions across them? Or maybe all of your microservices run on top of a monolith database with transaction safety?
  - hcarvalhoalves 6y ago
    It's probably a misconception that database-level transactions are the primary way to implement business transactions - many business processes w/ customers or 3rd parties rely on ACKs and/or compensating transactions, escaping the scope of a single database-level transaction. In practice, if the services are loosely-coupled enough, transactional boundaries will be local in scope and transactions spanning multiple services are unnecessary. Downstream services can be eventually consistent if idempotency is guaranteed and retries/replays are free. Inside the scope of a single service, when needed, we use Datomic, which provides ACID guarantees for that (https://docs.datomic.com/on-prem/acid.html https://docs.datomic.com/on-prem/acid.html) and also helps w/ idempotency guarantees by allowing time-travel (https://docs.datomic.com/on-prem/getting-started/see-historic-data.html https://docs.datomic.com/on-prem/getting-started/see-histori...).
ballpark 6y ago
I've watched a number of Clojure centric talks from Nubank employees. Actually, I think it was one particular employee, but I don't remember his name. One was on how they do testing. The others were about Fulcro, which is a very interesting web frontend library/framework.
timgilbert 6y ago
Nubank did a presentation on their approach to testing at the Clojure/NYC meetup: https://www.youtube.com/watch?v=1dW4V3stFX0 https://www.youtube.com/watch?v=1dW4V3stFX0

hcarvalhoalves 6y ago

Not sure we have any talks about that specifically, but what I can say is that on the Clojure codebases: - We rely heavily on testing (unit tests, property-based/generative and integration) - We adopt a micro-services architecture, so most codebases are small enough that refactoring is easy, or sometimes we just replace the service altogether - We leverage Schema (https://github.com/plumatic/schema https://github.com/plumatic/schema) and/or clojure.spec (https://clojure.org/guides/spec https://clojure.org/guides/spec) to annotate functions and data schemas, but it's opt-in - We have static checking of data schemas across boundaries (e.g. checking data producers did not break consumers over HTTP or Kafka), which is where we found the most value (replaced complicated end-to-end tests)

andrenth 6y ago
Thanks! Can you talk a bit more about the static checking of cross-boundaries schemas? Any relation to tools that allow a schema to evolve without breaking compatibility, like protobuf?
- hcarvalhoalves 6y ago
  It's not based on protobuf. There's some in-house tooling that checks the schemas of producers against consumers against known versions (aka the versions in production), and we run it as part of the CI. Since services implement standard components/protocols there are standardised hooks to extract these schemas definitions from each HTTP endpoint or Kafka consumer.
  - andrenth 6y ago
    Does this rely on code generators or maybe a schema generator from code? I’m curious about how schema and code are kept in sync.
    - hcarvalhoalves 6y ago
      Yes. The schema definition is part of the code/data itself, so you can reference a particular schema by its git rev hash.
stopachka 6y ago
Curious, did you really find that microservices helped? The downsides can be so tremendous -- I wonder if nubank would have done better to services in the low digit range. (context: my thoughts on it -> https://stopa.io/post/236 https://stopa.io/post/236)
- hcarvalhoalves 6y ago
  I believe micro-services change the trade-offs, and it's a matter of what you value more. We certainly have to invest more on infra-structure, monitoring, standardisation, and we have been doing since day one - by now it's pretty robust, to the point our internal tooling allowed us to migrate from raw EC2 to kubernetes (https://www.cncf.io/case-study/nubank/ https://www.cncf.io/case-study/nubank/) and it was a transparent change to all services and teams. On the other hand, failures are more localised and we gain robustness and speed to do certain kind of changes. Your point about getting domain boundaries wrong is true, but IMO it's a pain orthogonal to micro-services. The solution is the same as if you defined a sub-optimal class interface on a big mono-repo: relentless refactoring. We are often strangling (https://martinfowler.com/bliki/StranglerFigApplication.html https://martinfowler.com/bliki/StranglerFigApplication.html) and replacing legacy services.
hestefisk 6y ago
Questions: transactions are really important in banking systems. If you have 100s of Microservices how do you manage transactions across them? Or maybe all of your microservices run on top of a monolith database with transaction safety?
- hcarvalhoalves 6y ago
  It's probably a misconception that database-level transactions are the primary way to implement business transactions - many business processes w/ customers or 3rd parties rely on ACKs and/or compensating transactions, escaping the scope of a single database-level transaction. In practice, if the services are loosely-coupled enough, transactional boundaries will be local in scope and transactions spanning multiple services are unnecessary. Downstream services can be eventually consistent if idempotency is guaranteed and retries/replays are free. Inside the scope of a single service, when needed, we use Datomic, which provides ACID guarantees for that (https://docs.datomic.com/on-prem/acid.html https://docs.datomic.com/on-prem/acid.html) and also helps w/ idempotency guarantees by allowing time-travel (https://docs.datomic.com/on-prem/getting-started/see-historic-data.html https://docs.datomic.com/on-prem/getting-started/see-histori...).

ballpark 6y ago

I've watched a number of Clojure centric talks from Nubank employees. Actually, I think it was one particular employee, but I don't remember his name. One was on how they do testing. The others were about Fulcro, which is a very interesting web frontend library/framework.

timgilbert 6y ago

Nubank did a presentation on their approach to testing at the Clojure/NYC meetup: https://www.youtube.com/watch?v=1dW4V3stFX0 https://www.youtube.com/watch?v=1dW4V3stFX0