4 ms·

Why use nosql for a ledger?

weitzj 6y ago
I was wondering as well. How does that work with eventual consistency? Shouldn’t there be double bookkeeping so you don’t generate money out of the air?
- weitzj 6y ago
  Also it seems like the amount is of type number. Ideally you would calculate in Dollarcents or Eurocents (I.e. solely use integers) like 100 Eurocents equal 1€ So you do not have any rounding involved Also if a request comes in twice there are no requestids involved. Therefore it might be possible to generate another transaction it seems. There are great blog posts from Stripe engineers about these request ids to make requests idempotent with Postgres databases. I highly recommend reading them.
  - kondro 6y ago
    Not sure about how Clojure treats numbers, but in DynamoDB these are treated as Decimals and don’t have floating-point rounding errors.
    - tappio 6y ago
      Does it allow to save values with floating point imprecision such as 1.000000001? I don't think the issue is never in the database but the service that uses the API to send values. If such value is allowed, it will cause headache and errors when the consumers of the API make mistakes. I think the general practice is to store them in integers, just for the sake of being foolproof.
      - andriosr 6y ago
        Yep. Integers they are: https://news.ycombinator.com/item?id=24435148 https://news.ycombinator.com/item?id=24435148
      - kondro 6y ago
        You pass “numbers” to DynamoDB as strings defined with the type N. As long as you aren’t using a floating point initially, DDB isn’t going to add any floating point imprecision itself.
  - andriosr 6y ago
    Two great points. The amounts are actually integer, the same pattern used by Stripe and others. See it here: https://github.com/decimals/sequence/blob/master/src/decimals/specs.clj#L10 https://github.com/decimals/sequence/blob/master/src/decimal... I'm working on enabling idempotent requests, should come out next week. I'm using the same pattern used by Stripe as well (big fan). An optional `x-request-id` header that will ensure only once processing if present. This is very important to enable consistency over the network with other services using the ledger.
- andriosr 6y ago
  It maintains double-entry consistency of balances with DynamoDB. I uses DynamoDB (or any No-SQL) conditional puts. You can think of it as a update + where of SQL databases. But infinitely more scalable and available. Here is a piece I wrote about how it's achieved: https://decimals.substack.com/p/things-i-wish-i-knew-before-building https://decimals.substack.com/p/things-i-wish-i-knew-before-....
  - zrail 6y ago
    Is DynamoDB baked into the product or could new adapters be written to use different key/value stores? Things like leveldb for local-only ledgers or even an adapter on top of PostgreSQL's jsonb field might be useful.
    - andriosr 6y ago
      It was designed to support multiple DBs. Adding a new adapter should be pretty easy. I will add the first for Datomic in the coming weeks. This should pave the way for the next ones (I will add some docs on this). I really believe on it supporting many DBs.
      - andriosr 6y ago
        just created an issue on Github to keep track of this :)
hn_throwaway_99 6y ago
Was my top question as well, but note the underlying DB is Dynamo, which supports transactions, and a quick peek at the code looks like it's using them as intended (I don't know closure and it's been ages since I've used Dynamo so take with a grain of salt). https://github.com/decimals/sequence/blob/master/src/decimals/transactions.clj#L47 https://github.com/decimals/sequence/blob/master/src/decimal...
kondro 6y ago
It uses DynamoDB and transactions. Each double-entry appears to be 4 items, 1 for each balance update and 1 for each transaction. Each “journal” will use 2x4 WCU for the write to the table and 4 WCU for the global index updates (a total of 12 WCU for each journal). I do have a few issues with the structure though. They seem to use LSIs to add a timestamp (instead of just using KSUID or similar for the ID), which ends up limiting each partition to a maximum of 10GB (a limit that doesn’t exist if you don’t use LSIs). 10GB might seem like a lot per account (which seems like the PK the partition is based off), but even at a conservative 200 bytes per transaction, that limits a single account to 50 million transactions forever. Coupled with the fact this only seems to support literal double-entry accounting (not 2+ balanced transactions per journal item), you could quickly hit that limit if you are processing multi-item invoices for instance where items end up being allocated to different sales accounts. Although if you were processing multi-item invoices you’d also have to work out how to make this transactionally-safe yourself, as this only guarantees ACID-compliance of a single DR/CR transaction. Unfortunately, this wouldn’t necessarily be as easy as just allowing extra entries to a single DynamoDB transaction as those are limited to a max of 25 (and as covered, each entry requires 4 item updates). DynamoDB isn’t actually a bad idea if you need an ACID-compliant database with read-after-write consistency. But I think the model needs to be rethought a bit to deal with some of the other design considerations DynamoDB forces on you.
- andriosr 6y ago
  Wow, that was a great review, thanks! I agree with most points. There are definitely improvements to be made. This is the first version, I wanted to get it out and start getting feedback sooner. This is my take on each point: For some reason Dynamodb was not sorting the partition with KSUIDs string, so I added the LSI. This is probably something I did wrong and is just matter of fixing the query or the insert and move back to using the PK. Adding more than one balanced transaction per request is on the roadmap, but this will be limited to the DynamoDB transactions API limit of 25 items as you mentioned (25 / 2*2 WCU per tx = 6 transactions per request). Each entry only uses 4 items in the worst case (seeding a new currency in an account). In most cases (existing balances/currencies), there will be only 2 new items. I think even more improvements can be made. And the goal of open-sourcing is to get more heads thinking about how we can do it :)
andriosr 6y ago
I wrote a piece that touches a bit on the why: https://decimals.substack.com/p/things-i-wish-i-knew-before-building https://decimals.substack.com/p/things-i-wish-i-knew-before-... But in a nutshell: availability and scalability. Yes you can achieve this with relational DBs, but it will be quite complex, while No-SQL gives it by design. All while ensuring all the goodness of relational data modeling, consistency and queries. The only requirement is a good table design: https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1s https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1s

I was wondering as well. How does that work with eventual consistency? Shouldn’t there be double bookkeeping so you don’t generate money out of the air?

weitzj 6y ago
Also it seems like the amount is of type number. Ideally you would calculate in Dollarcents or Eurocents (I.e. solely use integers) like 100 Eurocents equal 1€ So you do not have any rounding involved Also if a request comes in twice there are no requestids involved. Therefore it might be possible to generate another transaction it seems. There are great blog posts from Stripe engineers about these request ids to make requests idempotent with Postgres databases. I highly recommend reading them.
- kondro 6y ago
  Not sure about how Clojure treats numbers, but in DynamoDB these are treated as Decimals and don’t have floating-point rounding errors.
  - tappio 6y ago
    Does it allow to save values with floating point imprecision such as 1.000000001? I don't think the issue is never in the database but the service that uses the API to send values. If such value is allowed, it will cause headache and errors when the consumers of the API make mistakes. I think the general practice is to store them in integers, just for the sake of being foolproof.
    - andriosr 6y ago
      Yep. Integers they are: https://news.ycombinator.com/item?id=24435148 https://news.ycombinator.com/item?id=24435148
    - kondro 6y ago
      You pass “numbers” to DynamoDB as strings defined with the type N. As long as you aren’t using a floating point initially, DDB isn’t going to add any floating point imprecision itself.
- andriosr 6y ago
  Two great points. The amounts are actually integer, the same pattern used by Stripe and others. See it here: https://github.com/decimals/sequence/blob/master/src/decimals/specs.clj#L10 https://github.com/decimals/sequence/blob/master/src/decimal... I'm working on enabling idempotent requests, should come out next week. I'm using the same pattern used by Stripe as well (big fan). An optional `x-request-id` header that will ensure only once processing if present. This is very important to enable consistency over the network with other services using the ledger.
andriosr 6y ago
It maintains double-entry consistency of balances with DynamoDB. I uses DynamoDB (or any No-SQL) conditional puts. You can think of it as a update + where of SQL databases. But infinitely more scalable and available. Here is a piece I wrote about how it's achieved: https://decimals.substack.com/p/things-i-wish-i-knew-before-building https://decimals.substack.com/p/things-i-wish-i-knew-before-....
- zrail 6y ago
  Is DynamoDB baked into the product or could new adapters be written to use different key/value stores? Things like leveldb for local-only ledgers or even an adapter on top of PostgreSQL's jsonb field might be useful.
  - andriosr 6y ago
    It was designed to support multiple DBs. Adding a new adapter should be pretty easy. I will add the first for Datomic in the coming weeks. This should pave the way for the next ones (I will add some docs on this). I really believe on it supporting many DBs.
    - andriosr 6y ago
      just created an issue on Github to keep track of this :)

hn_throwaway_99 6y ago

Was my top question as well, but note the underlying DB is Dynamo, which supports transactions, and a quick peek at the code looks like it's using them as intended (I don't know closure and it's been ages since I've used Dynamo so take with a grain of salt). https://github.com/decimals/sequence/blob/master/src/decimals/transactions.clj#L47 https://github.com/decimals/sequence/blob/master/src/decimal...

kondro 6y ago

It uses DynamoDB and transactions. Each double-entry appears to be 4 items, 1 for each balance update and 1 for each transaction. Each “journal” will use 2x4 WCU for the write to the table and 4 WCU for the global index updates (a total of 12 WCU for each journal). I do have a few issues with the structure though. They seem to use LSIs to add a timestamp (instead of just using KSUID or similar for the ID), which ends up limiting each partition to a maximum of 10GB (a limit that doesn’t exist if you don’t use LSIs). 10GB might seem like a lot per account (which seems like the PK the partition is based off), but even at a conservative 200 bytes per transaction, that limits a single account to 50 million transactions forever. Coupled with the fact this only seems to support literal double-entry accounting (not 2+ balanced transactions per journal item), you could quickly hit that limit if you are processing multi-item invoices for instance where items end up being allocated to different sales accounts. Although if you were processing multi-item invoices you’d also have to work out how to make this transactionally-safe yourself, as this only guarantees ACID-compliance of a single DR/CR transaction. Unfortunately, this wouldn’t necessarily be as easy as just allowing extra entries to a single DynamoDB transaction as those are limited to a max of 25 (and as covered, each entry requires 4 item updates). DynamoDB isn’t actually a bad idea if you need an ACID-compliant database with read-after-write consistency. But I think the model needs to be rethought a bit to deal with some of the other design considerations DynamoDB forces on you.

andriosr 6y ago
Wow, that was a great review, thanks! I agree with most points. There are definitely improvements to be made. This is the first version, I wanted to get it out and start getting feedback sooner. This is my take on each point: For some reason Dynamodb was not sorting the partition with KSUIDs string, so I added the LSI. This is probably something I did wrong and is just matter of fixing the query or the insert and move back to using the PK. Adding more than one balanced transaction per request is on the roadmap, but this will be limited to the DynamoDB transactions API limit of 25 items as you mentioned (25 / 2*2 WCU per tx = 6 transactions per request). Each entry only uses 4 items in the worst case (seeding a new currency in an account). In most cases (existing balances/currencies), there will be only 2 new items. I think even more improvements can be made. And the goal of open-sourcing is to get more heads thinking about how we can do it :)

andriosr 6y ago

I wrote a piece that touches a bit on the why: https://decimals.substack.com/p/things-i-wish-i-knew-before-building https://decimals.substack.com/p/things-i-wish-i-knew-before-... But in a nutshell: availability and scalability. Yes you can achieve this with relational DBs, but it will be quite complex, while No-SQL gives it by design. All while ensuring all the goodness of relational data modeling, consistency and queries. The only requirement is a good table design: https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1s https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1s