7 ms·

At a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air. Prior to one of these hiccups, I hy

At a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air. Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly. This led to an argument with my boss, who assumed things magically worked. Days later, we received an email announcing an audit one one of these accounting discrepancies. JPMC proposed using crypto, internally, to consistently manage cash flow. Not sure if it went anywhere.

hooverd 2y ago
It's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.
- trog 2y ago
  Is a Merkle tree needed or is good old basic double ledger accounting in a central database sufficient? If a key requirement is not a distributed ledger then it seems like a waste of time.
  - Onavo 2y ago
    Merkle tree is to prevent tampering, not bad accounting practices
    - nly 2y ago
      It only prevents tampering if the cost of generating hashes is extremely high. Internally in your company you're not going to spend millions of $'s a year in GPU compute just to replace a database.
      - xorcist 2y ago
        "Prevents tampering" lacks specificity. git is a blockchain that prevents tampering in some aspects, but you can still force push if you have that privilege. What is important is understand what the guarantees are.
      - limit499karma 2y ago
        ? If I use something like Blake3 (which is super fast and emits gobs of good bits) and encode a node with say 512 bits of the hash, you are claiming that somehow I am vulnerable to tampering because the hash function is fast? What is the probable number of attempts to forge a document D' that hashes to the very same hash? And if the document in structured per a standard format, you have even less degrees of freedom in forging a fake. So yes, a Merkel tree definitely can provide very strong guarantees against tampering.
        oconnor663 2y ago
        Fwiw, increasing the BLAKE3 output size beyond 256 bits doesn't add security, because the internal "chaining values" are still 256 bits regardless of the final output length. But 256 bits of security should be enough for any practical purpose.
        limit499karma 2y ago
        Good to know. But does that also mean that e.g. splitting the full output to n 256 chunks would mean there is correlation between the chunks? (I always assumed one could grab any number of bits (from anywhere) in a cryptographic hash.)
        oconnor663 2y ago
        You can take as many bytes from the output stream as you want, and they should all be indistinguishable from random to someone who can't guess the input. (Similar to how each of the bytes of a SHA-256 hash should appear independently random. I don't think that's a formal design goal in the SHA-2 spec, but in practice we'd be very surprised and worried if that property didn't hold.) But for example in the catastrophic case where someone found a collision in the default 256-bit BLAKE3 output, they would probably be able to construct colliding outputs of unlimited length with little additional effort.
      - benlivengood 2y ago
        Certificate transparency logs achieve tamper-resistance without expensive hashes.
    - agentultra 2y ago
      Write-Once, Read Many drives also prevent tampering. Not everything needs crypto.
      - chaboud 2y ago
        In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority.
        trog 2y ago
        > In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority. Yeh, but that's kinda my point: if your primary use case is not "needs to be distributed" then there's almost never a benefit, because there is always a trusted authority and the benefits of centralisation outweigh (massively, IMO) any benefit you get from a blockchain approach.
        chaboud 2y ago
        100% agreed there. A central authority can just sign stuff. Merkle trees can still be very valuable for integrity and synchronization management, but burning a bunch of energy to bogo-search nonces is silly if the writer (or federated writers) can be cryptographic authorities.
  - jchanimal 2y ago
    We launched Fireproof earlier this month on HN. It’s a tamperproof Merkle CRDT in TypeScript, with an object storage backend for portability. See our Show HN: https://news.ycombinator.com/item?id=42184362 https://news.ycombinator.com/item?id=42184362 We’ve seen interest from trading groups for edge collaboration, so multi-user apps can run on-site without cloud latency.
    - nearting 2y ago
      This doesn't address the question in any way except to note that you also use Merkle Trees. Do you reply to any comment mentioning TypeScript with a link to your Show HN post as well?
    - hluska 2y ago
      What disrespectful marketing. We don’t care that you use Merkle trees because that’s irrelevant. I guess I can add Fireproof to my big list of sketchy products to avoid. It’s embarrassing.
      - jchanimal 2y ago
        I figured the responses would be more interesting. Questions about CRDT guarantees etc. Perhaps worth seeding the convo with a remark about finality.
        hluska 2y ago
        While your intentions may have been around discussion, I don’t want to be marketed to when I’m trying to understand something unrelated. I have a business degree so I intimately understand that HN is technically free and it’s nice to get free eyeballs, but we are people too. I’m so much more than a credit card number, yet you’ve reduced me to a user acquisition in the most insulting way possible. Perhaps instead of your ideas, it’s worth seeding your own personal make up with a firm statement of ethics?? Are you the kind of person who will hijack conversations to promote your product? Or do you have integrity? Just purely out of concern for your business, do you have a cofounder who could handle marketing for you? If so, consider letting her have complete control over that function. It’s genuinely sad to see a founder squander goodwill on shitty marketing.
- csomar 2y ago
  Crypto/Blockchain makes it harder to have an incorrect state. If you fk up, you need to take down the whole operation and reverse everything back to the block in question. This ensures that everything was accounted for. On the other hand, if you fk in a traditional ledger system you might be tempted to keep things running and resolve "only" the affected accounts.
  - delfinom 2y ago
    It's a question of business case. While ensuring you are always accounted correctly seems like a plus, if errors happen too often potentially due to volume, it makes more business sense sometimes to handle it while running rather than costing the business millions per minute having a pause.
  - necovek 2y ago
    It's mostly a different approach to "editing" a transaction. With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest. The difference is that you've got a ledger that's clearly a fork because of cryptographic signing. With a traditional ledger, you fix the wrong transaction in place. You could also cryptographically sign them, and you could make those signatures depend on previous state, where you basically get two "blockchains". Distributed trust mechanisms, usually used with crypto and blockchain, only matter when you want to keep the entire ledger public and decentralized (as in, allow untrusted parties to modify it).
    - koolba 2y ago
      > With a traditional ledger, you fix the wrong transaction in place. No you don’t. You reverse out the old transaction by posting journal lines for the negation. And in the same transactions you include the proper booking of the balance movements. You never edit old transactions. It’s always the addition of new transactions so you can go back and see what was corrected.
      - necovek 2y ago
        Right, thanks for the correction: I wanted to highlight the need for "replaying" all the other transactions with a blockchain.
im3w1l 2y ago
If its for internal why not just use a normal append only log. x amount transferred from account y to account z. A three column csv oughta do it.
- sneak 2y ago
  Any time your proposal entails a “why not just”, it is almost certainly underestimating the mental abilities of the people and teams who implemented it. A good option is “what would happen if we” instead of anything involving the word “just”.
  - foobarbecue 2y ago
    Lots of threads on this here, most recently https://news.ycombinator.com/item?id=42038139#42038572 https://news.ycombinator.com/item?id=42038139#42038572 . I think this example is perfect, with the "oughta do it"
  - PittleyDunkin 2y ago
    Counterfactuals strike me as even less useful than underestimating competency would be. Surely basic double-entry accounting (necessarily implying the use of ledgers) should be considered table stakes for fintech competency.
  - qazxcvbnmlp 2y ago
    “Just” usually implies a lack of understanding of the problem space in question. If someone says “solution X” was considered because of these factors which lead to these tradeoffs however since then fundamental assumption Y has changed which allows this new solution then it’s very interesting.
    - jknoepfler 2y ago
      Sure. When I ask "why don't we just" I'm suggesting that the engineering solutions on the table sound over-engineered to the task, and I'm asking why we aren't opting for a straightforward, obvious, simple solution. Sometimes the answer is legitimate complexity. Equally as often, especially with less experienced engineers, the answer is that they started running with a shiny and didn't back up and say "why don't we just..." themselves.
HolyLampshade 2y ago
At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there. If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.
- ajb 2y ago
  Wait, pcap as in wireshark packet capture?
  - Loic 2y ago
    I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.
  - w23j 2y ago
    I would also really like to know that! It generally seems to be a thing in trading: https://databento.com/pcaps https://databento.com/pcaps There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history/tick-history-pcap https://www.lseg.com/en/data-analytics/market-data/data-feed...
    - hnbear 2y ago
      Look up Corvil devices by Pico. Commonly used in finance. https://www.pico.net/corvil-analytics/ https://www.pico.net/corvil-analytics/
  - tnlnbn 2y ago
    I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!
DanielHB 2y ago
> I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly. That is like half of the plot of Office Space
naasking 2y ago
> JPMC proposed using crypto, internally, to consistently manage cash flow. Yikes, how hard is it to just capture an immutable event log. Way cheaper than running crypto, even if only internally.
- limit499karma 2y ago
  Theoretically they even have a better security environment (since it is internal and they control users, code base and network) so the consensus mechanism may not even require BFT.
- imglorp 2y ago
  Harder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense. Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
  - baq 2y ago
    I'll just leave that here for no particular reason at all: https://www.sec.gov/enforcement-litigation/whistleblower-program https://www.sec.gov/enforcement-litigation/whistleblower-pro...
    - fragmede 2y ago
      Importantly, the SEC is empowered to give 10-30% of the money siezed via whistleblowing too the whistle blower.
    - voidfunc 2y ago
      Better hurry, Elon is gonna dismantle the SEC in about 45 days
phire 2y ago
Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow. As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem). The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.
- oblio 2y ago
  > Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow. I know you're not arguing in their favor, just describing a reality, but the irony of that phrase is through the roof :-))) Especially the "centralized crypto".
  - phire 2y ago
    Yeah, that fact alone goes a long way to proving there is no technical merit to cryptocurrencies. The reason they are now called "centralised crypto exchanges" is that "decentralised crypto exchanges" now exist, where trades do actually happen on a public blockchain. Though, a large chunk of those are "fake", where they look like a decentralised exchange, but there is a central entity holding all the coins in central wallets and can misplace them, or even reverse trades. You kind of get the worst of both worlds, as you are now venerable to front-running, they are slow, and the exchange can still rug pull you. The legit decentralised exchanges are limited to only trading tokens on a given blockchain (usually ethereum), are even slower, are still vulnerable to front-running. Plus, they spam those blockchains with loads of transactions, driving up transaction fees.
mlloyd 2y ago
This sounds like a situation that I know about at the placed identified by name in your comment. It took months to track down the issue.

hooverd 2y ago

It's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.

trog 2y ago
Is a Merkle tree needed or is good old basic double ledger accounting in a central database sufficient? If a key requirement is not a distributed ledger then it seems like a waste of time.
- Onavo 2y ago
  Merkle tree is to prevent tampering, not bad accounting practices
  - nly 2y ago
    It only prevents tampering if the cost of generating hashes is extremely high. Internally in your company you're not going to spend millions of $'s a year in GPU compute just to replace a database.
    - xorcist 2y ago
      "Prevents tampering" lacks specificity. git is a blockchain that prevents tampering in some aspects, but you can still force push if you have that privilege. What is important is understand what the guarantees are.
    - limit499karma 2y ago
      ? If I use something like Blake3 (which is super fast and emits gobs of good bits) and encode a node with say 512 bits of the hash, you are claiming that somehow I am vulnerable to tampering because the hash function is fast? What is the probable number of attempts to forge a document D' that hashes to the very same hash? And if the document in structured per a standard format, you have even less degrees of freedom in forging a fake. So yes, a Merkel tree definitely can provide very strong guarantees against tampering.
      - oconnor663 2y ago
        Fwiw, increasing the BLAKE3 output size beyond 256 bits doesn't add security, because the internal "chaining values" are still 256 bits regardless of the final output length. But 256 bits of security should be enough for any practical purpose.
        limit499karma 2y ago
        Good to know. But does that also mean that e.g. splitting the full output to n 256 chunks would mean there is correlation between the chunks? (I always assumed one could grab any number of bits (from anywhere) in a cryptographic hash.)
        oconnor663 2y ago
        You can take as many bytes from the output stream as you want, and they should all be indistinguishable from random to someone who can't guess the input. (Similar to how each of the bytes of a SHA-256 hash should appear independently random. I don't think that's a formal design goal in the SHA-2 spec, but in practice we'd be very surprised and worried if that property didn't hold.) But for example in the catastrophic case where someone found a collision in the default 256-bit BLAKE3 output, they would probably be able to construct colliding outputs of unlimited length with little additional effort.
    - benlivengood 2y ago
      Certificate transparency logs achieve tamper-resistance without expensive hashes.
  - agentultra 2y ago
    Write-Once, Read Many drives also prevent tampering. Not everything needs crypto.
    - chaboud 2y ago
      In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority.
      - trog 2y ago
        > In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority. Yeh, but that's kinda my point: if your primary use case is not "needs to be distributed" then there's almost never a benefit, because there is always a trusted authority and the benefits of centralisation outweigh (massively, IMO) any benefit you get from a blockchain approach.
        chaboud 2y ago
        100% agreed there. A central authority can just sign stuff. Merkle trees can still be very valuable for integrity and synchronization management, but burning a bunch of energy to bogo-search nonces is silly if the writer (or federated writers) can be cryptographic authorities.
- jchanimal 2y ago
  We launched Fireproof earlier this month on HN. It’s a tamperproof Merkle CRDT in TypeScript, with an object storage backend for portability. See our Show HN: https://news.ycombinator.com/item?id=42184362 https://news.ycombinator.com/item?id=42184362 We’ve seen interest from trading groups for edge collaboration, so multi-user apps can run on-site without cloud latency.
  - nearting 2y ago
    This doesn't address the question in any way except to note that you also use Merkle Trees. Do you reply to any comment mentioning TypeScript with a link to your Show HN post as well?
  - hluska 2y ago
    What disrespectful marketing. We don’t care that you use Merkle trees because that’s irrelevant. I guess I can add Fireproof to my big list of sketchy products to avoid. It’s embarrassing.
    - jchanimal 2y ago
      I figured the responses would be more interesting. Questions about CRDT guarantees etc. Perhaps worth seeding the convo with a remark about finality.
      - hluska 2y ago
        While your intentions may have been around discussion, I don’t want to be marketed to when I’m trying to understand something unrelated. I have a business degree so I intimately understand that HN is technically free and it’s nice to get free eyeballs, but we are people too. I’m so much more than a credit card number, yet you’ve reduced me to a user acquisition in the most insulting way possible. Perhaps instead of your ideas, it’s worth seeding your own personal make up with a firm statement of ethics?? Are you the kind of person who will hijack conversations to promote your product? Or do you have integrity? Just purely out of concern for your business, do you have a cofounder who could handle marketing for you? If so, consider letting her have complete control over that function. It’s genuinely sad to see a founder squander goodwill on shitty marketing.
csomar 2y ago
Crypto/Blockchain makes it harder to have an incorrect state. If you fk up, you need to take down the whole operation and reverse everything back to the block in question. This ensures that everything was accounted for. On the other hand, if you fk in a traditional ledger system you might be tempted to keep things running and resolve "only" the affected accounts.
- delfinom 2y ago
  It's a question of business case. While ensuring you are always accounted correctly seems like a plus, if errors happen too often potentially due to volume, it makes more business sense sometimes to handle it while running rather than costing the business millions per minute having a pause.
- necovek 2y ago
  It's mostly a different approach to "editing" a transaction. With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest. The difference is that you've got a ledger that's clearly a fork because of cryptographic signing. With a traditional ledger, you fix the wrong transaction in place. You could also cryptographically sign them, and you could make those signatures depend on previous state, where you basically get two "blockchains". Distributed trust mechanisms, usually used with crypto and blockchain, only matter when you want to keep the entire ledger public and decentralized (as in, allow untrusted parties to modify it).
  - koolba 2y ago
    > With a traditional ledger, you fix the wrong transaction in place. No you don’t. You reverse out the old transaction by posting journal lines for the negation. And in the same transactions you include the proper booking of the balance movements. You never edit old transactions. It’s always the addition of new transactions so you can go back and see what was corrected.
    - necovek 2y ago
      Right, thanks for the correction: I wanted to highlight the need for "replaying" all the other transactions with a blockchain.

im3w1l 2y ago

If its for internal why not just use a normal append only log. x amount transferred from account y to account z. A three column csv oughta do it.

sneak 2y ago
Any time your proposal entails a “why not just”, it is almost certainly underestimating the mental abilities of the people and teams who implemented it. A good option is “what would happen if we” instead of anything involving the word “just”.
- foobarbecue 2y ago
  Lots of threads on this here, most recently https://news.ycombinator.com/item?id=42038139#42038572 https://news.ycombinator.com/item?id=42038139#42038572 . I think this example is perfect, with the "oughta do it"
- PittleyDunkin 2y ago
  Counterfactuals strike me as even less useful than underestimating competency would be. Surely basic double-entry accounting (necessarily implying the use of ledgers) should be considered table stakes for fintech competency.
- qazxcvbnmlp 2y ago
  “Just” usually implies a lack of understanding of the problem space in question. If someone says “solution X” was considered because of these factors which lead to these tradeoffs however since then fundamental assumption Y has changed which allows this new solution then it’s very interesting.
  - jknoepfler 2y ago
    Sure. When I ask "why don't we just" I'm suggesting that the engineering solutions on the table sound over-engineered to the task, and I'm asking why we aren't opting for a straightforward, obvious, simple solution. Sometimes the answer is legitimate complexity. Equally as often, especially with less experienced engineers, the answer is that they started running with a shiny and didn't back up and say "why don't we just..." themselves.

HolyLampshade 2y ago

At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there. If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.

ajb 2y ago
Wait, pcap as in wireshark packet capture?
- Loic 2y ago
  I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.
- w23j 2y ago
  I would also really like to know that! It generally seems to be a thing in trading: https://databento.com/pcaps https://databento.com/pcaps There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history/tick-history-pcap https://www.lseg.com/en/data-analytics/market-data/data-feed...
  - hnbear 2y ago
    Look up Corvil devices by Pico. Commonly used in finance. https://www.pico.net/corvil-analytics/ https://www.pico.net/corvil-analytics/
- tnlnbn 2y ago
  I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!

DanielHB 2y ago

> I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly. That is like half of the plot of Office Space

naasking 2y ago

> JPMC proposed using crypto, internally, to consistently manage cash flow. Yikes, how hard is it to just capture an immutable event log. Way cheaper than running crypto, even if only internally.

limit499karma 2y ago
Theoretically they even have a better security environment (since it is internal and they control users, code base and network) so the consensus mechanism may not even require BFT.
imglorp 2y ago
Harder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense. Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
- baq 2y ago
  I'll just leave that here for no particular reason at all: https://www.sec.gov/enforcement-litigation/whistleblower-program https://www.sec.gov/enforcement-litigation/whistleblower-pro...
  - fragmede 2y ago
    Importantly, the SEC is empowered to give 10-30% of the money siezed via whistleblowing too the whistle blower.
  - voidfunc 2y ago
    Better hurry, Elon is gonna dismantle the SEC in about 45 days

phire 2y ago

Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow. As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem). The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.

oblio 2y ago
> Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow. I know you're not arguing in their favor, just describing a reality, but the irony of that phrase is through the roof :-))) Especially the "centralized crypto".
- phire 2y ago
  Yeah, that fact alone goes a long way to proving there is no technical merit to cryptocurrencies. The reason they are now called "centralised crypto exchanges" is that "decentralised crypto exchanges" now exist, where trades do actually happen on a public blockchain. Though, a large chunk of those are "fake", where they look like a decentralised exchange, but there is a central entity holding all the coins in central wallets and can misplace them, or even reverse trades. You kind of get the worst of both worlds, as you are now venerable to front-running, they are slow, and the exchange can still rug pull you. The legit decentralised exchanges are limited to only trading tokens on a given blockchain (usually ethereum), are even slower, are still vulnerable to front-running. Plus, they spam those blockchains with loads of transactions, driving up transaction fees.

mlloyd 2y ago

This sounds like a situation that I know about at the placed identified by name in your comment. It took months to track down the issue.