4 ms·

Looks like picking Clojure and Datomic has created a great deal of technical debt for them. They started adding Spec everywhere to specify their data because th

Looks like picking Clojure and Datomic has created a great deal of technical debt for them. They started adding Spec everywhere to specify their data because they were having big problems scaling their wild west code base. But now Spec is dead and there's a new Spec2.alpha version. I don't even know how they have manage to scale datomic to that level, the support contract we had for datomic was only really used to report bugs[0] but they have more than 2000 datomic transactors? ouch. [0] Yes, too much bugs and slow, but databases are hard so I guess this was expected for a closed-source niche DB with little users.

rapfaria 7y ago
Not only that, but they had intentions of using `Clojure for everything`. From configuration management (currently using ruby if I recall correctly) to deployment, they want to use Clojure on all fronts.
dustingetz 7y ago
Isn't nubank adding 50k users per day? Founded in 2013, with seven years later a $10B valuation? They don't seem to have a technical debt problem I was at the talk, they don't use spec they use prismatic schema iirc while waiting for spec2 to stabilize.
- fnordsensei 7y ago
  They officially launched in 2014, I believe. They have some 20 million customers, last I checked, which gives ~10k customers per day on average from 2014. It makes sense that it would have been fewer per day in the beginning, and then accelerated.
jonahbenton 7y ago
I was at the talk. From having engaged with a lot of organizations working at scale, it was pretty clear they were near the top in terms of not being hampered by technical debt. They have the capability to move quickly and evolve the system in pragmatic ways to dial the knobs towards correctness or speed or functionality or whatever. A tremendous achievement and speaks volumes for the tooling. There is no "Datomic scale" problem. Datomic transactors are just another singleton service to deploy with your microservice pod. The underlying storage can obviously be consolidated and scaled independently. I don't recall what they're using, would guess it is pg.
- Scarbutt 7y ago
  Multiple datomic transactors can't talk to the same database, even if they share the same storage.
  - jonahbenton 7y ago
    Sure they can. Just give them different database names.
    - Scarbutt 7y ago
      A different name is literally a different database.
      - jonahbenton 7y ago
        A single postgres daemon- a unit of scaling- can support many datomic transactors, each talking to a named "database" hosted by that one daemon. You don't have to scale your postgres daemons with your microservices- each of which have their own transactor- which would be painful and out of the ordinary for an ops team. Scaling datomic on top of postgres is no different from scaling any other microservice. That all said, the architectural point that did sound painful in the presentation- and which is a common pain point in microservice architectures, not unique to what NuBank is doing with Datomic- was having to maintain an ETL for analytics purposes, to pull together all of the distinct microservice-specific Datomic-hosted data sets into a single uniformly SQL-queryable data set. The details of that implementation, and whether it made use of the new SQL interface supported by Datomic, were not discussed. But it smelled brittle and fragile.
        Scarbutt 7y ago
        A single postgres daemon- a unit of scaling- can support many datomic transactors yes ,each talking to a named "database" hosted by that one daemon. No. That would imply distributed writes and break the single write serializability property of datomic. Think about it, transactors don't sync with each other(only one for HA but that's orthogonal). To put it simply, multiple transactors can share the same storage but only one can write to a single database at a time.
slifin 7y ago
Datomic scales horizontally for reads but doesn't for writes, but then natively mysql doesn't scale for reads or writes, most people looking to scale mysql up use a clustered approach with many readers and one writer using a third party or enterprise solution I don't know of a reliable multi write system for mysql that doesn't make significant trade offs At work we use multiple mysql servers to handle "scaling" so it's not a surprise to me that Nubank are using multiple Datomic servers Unless you start getting into eventual consistency territory I don't think distributed writes are a trivial problem and targeting Datomic for this is a bit odd
- slifin 7y ago
  Also the spec stuff I don't think it's a surprise to Nubank that spec is being upgraded it literally says alpha in the namespace, they have some great Clojure engineers at Nubank particularly the developer of Pathom they're really pushing the envelope in terms of distributed graphs and front end

rapfaria 7y ago

Not only that, but they had intentions of using `Clojure for everything`. From configuration management (currently using ruby if I recall correctly) to deployment, they want to use Clojure on all fronts.

dustingetz 7y ago

Isn't nubank adding 50k users per day? Founded in 2013, with seven years later a $10B valuation? They don't seem to have a technical debt problem I was at the talk, they don't use spec they use prismatic schema iirc while waiting for spec2 to stabilize.

fnordsensei 7y ago
They officially launched in 2014, I believe. They have some 20 million customers, last I checked, which gives ~10k customers per day on average from 2014. It makes sense that it would have been fewer per day in the beginning, and then accelerated.

jonahbenton 7y ago

I was at the talk. From having engaged with a lot of organizations working at scale, it was pretty clear they were near the top in terms of not being hampered by technical debt. They have the capability to move quickly and evolve the system in pragmatic ways to dial the knobs towards correctness or speed or functionality or whatever. A tremendous achievement and speaks volumes for the tooling. There is no "Datomic scale" problem. Datomic transactors are just another singleton service to deploy with your microservice pod. The underlying storage can obviously be consolidated and scaled independently. I don't recall what they're using, would guess it is pg.

Scarbutt 7y ago
Multiple datomic transactors can't talk to the same database, even if they share the same storage.
- jonahbenton 7y ago
  Sure they can. Just give them different database names.
  - Scarbutt 7y ago
    A different name is literally a different database.
    - jonahbenton 7y ago
      A single postgres daemon- a unit of scaling- can support many datomic transactors, each talking to a named "database" hosted by that one daemon. You don't have to scale your postgres daemons with your microservices- each of which have their own transactor- which would be painful and out of the ordinary for an ops team. Scaling datomic on top of postgres is no different from scaling any other microservice. That all said, the architectural point that did sound painful in the presentation- and which is a common pain point in microservice architectures, not unique to what NuBank is doing with Datomic- was having to maintain an ETL for analytics purposes, to pull together all of the distinct microservice-specific Datomic-hosted data sets into a single uniformly SQL-queryable data set. The details of that implementation, and whether it made use of the new SQL interface supported by Datomic, were not discussed. But it smelled brittle and fragile.
      - Scarbutt 7y ago
        A single postgres daemon- a unit of scaling- can support many datomic transactors yes ,each talking to a named "database" hosted by that one daemon. No. That would imply distributed writes and break the single write serializability property of datomic. Think about it, transactors don't sync with each other(only one for HA but that's orthogonal). To put it simply, multiple transactors can share the same storage but only one can write to a single database at a time.

slifin 7y ago

Datomic scales horizontally for reads but doesn't for writes, but then natively mysql doesn't scale for reads or writes, most people looking to scale mysql up use a clustered approach with many readers and one writer using a third party or enterprise solution I don't know of a reliable multi write system for mysql that doesn't make significant trade offs At work we use multiple mysql servers to handle "scaling" so it's not a surprise to me that Nubank are using multiple Datomic servers Unless you start getting into eventual consistency territory I don't think distributed writes are a trivial problem and targeting Datomic for this is a bit odd

slifin 7y ago
Also the spec stuff I don't think it's a surprise to Nubank that spec is being upgraded it literally says alpha in the namespace, they have some great Clojure engineers at Nubank particularly the developer of Pathom they're really pushing the envelope in terms of distributed graphs and front end