11 ms·
Tell that to Synapse customers. Many millions of dollars are missing. Banks have to follow strict rules to account for where all the money goes. But the way fi
by hn_throwaway_99 2y ago
Tell that to Synapse customers. Many millions of dollars are missing.
Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.
FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.
See https://www.forbes.com/sites/zennonkapron/2024/11/08/what-the-yotta-saga-reveals-about-fintech-shortcomings/ https://www.forbes.com/sites/zennonkapron/2024/11/08/what-th...
- mrfox321 2y agoAt a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air. Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly. This led to an argument with my boss, who assumed things magically worked. Days later, we received an email announcing an audit one one of these accounting discrepancies. JPMC proposed using crypto, internally, to consistently manage cash flow. Not sure if it went anywhere.
- hooverd 2y agoIt's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.
- trog 2y agoIs a Merkle tree needed or is good old basic double ledger accounting in a central database sufficient? If a key requirement is not a distributed ledger then it seems like a waste of time.
- Onavo 2y agoMerkle tree is to prevent tampering, not bad accounting practices
- nly 2y agoIt only prevents tampering if the cost of generating hashes is extremely high. Internally in your company you're not going to spend millions of $'s a year in GPU compute just to replace a database.
- xorcist 2y ago"Prevents tampering" lacks specificity. git is a blockchain that prevents tampering in some aspects, but you can still force push if you have that privilege. What is important is understand what the guarantees are.
- limit499karma 2y ago? If I use something like Blake3 (which is super fast and emits gobs of good bits) and encode a node with say 512 bits of the hash, you are claiming that somehow I am vulnerable to tampering because the hash function is fast? What is the probable number of attempts to forge a document D' that hashes to the very same hash? And if the document in structured per a standard format, you have even less degrees of freedom in forging a fake. So yes, a Merkel tree definitely can provide very strong guarantees against tampering.
- oconnor663 2y agoFwiw, increasing the BLAKE3 output size beyond 256 bits doesn't add security, because the internal "chaining values" are still 256 bits regardless of the final output length. But 256 bits of security should be enough for any practical purpose.
- im3w1l 2y agoIf its for internal why not just use a normal append only log. x amount transferred from account y to account z. A three column csv oughta do it.
- sneak 2y agoAny time your proposal entails a “why not just”, it is almost certainly underestimating the mental abilities of the people and teams who implemented it. A good option is “what would happen if we” instead of anything involving the word “just”.
- foobarbecue 2y agoLots of threads on this here, most recently https://news.ycombinator.com/item?id=42038139#42038572 https://news.ycombinator.com/item?id=42038139#42038572 . I think this example is perfect, with the "oughta do it"
- PittleyDunkin 2y agoCounterfactuals strike me as even less useful than underestimating competency would be. Surely basic double-entry accounting (necessarily implying the use of ledgers) should be considered table stakes for fintech competency.
- qazxcvbnmlp 2y ago“Just” usually implies a lack of understanding of the problem space in question. If someone says “solution X” was considered because of these factors which lead to these tradeoffs however since then fundamental assumption Y has changed which allows this new solution then it’s very interesting.
- jknoepfler 2y agoSure. When I ask "why don't we just" I'm suggesting that the engineering solutions on the table sound over-engineered to the task, and I'm asking why we aren't opting for a straightforward, obvious, simple solution. Sometimes the answer is legitimate complexity. Equally as often, especially with less experienced engineers, the answer is that they started running with a shiny and didn't back up and say "why don't we just..." themselves.
- HolyLampshade 2y agoAt all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there. If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.
- ajb 2y agoWait, pcap as in wireshark packet capture?
- Loic 2y agoI suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.
- w23j 2y agoI would also really like to know that! It generally seems to be a thing in trading: https://databento.com/pcaps https://databento.com/pcaps There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history/tick-history-pcap https://www.lseg.com/en/data-analytics/market-data/data-feed...
- hnbear 2y agoLook up Corvil devices by Pico. Commonly used in finance. https://www.pico.net/corvil-analytics/ https://www.pico.net/corvil-analytics/
- tnlnbn 2y agoI'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!
- DanielHB 2y ago> I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly. That is like half of the plot of Office Space
- naasking 2y ago> JPMC proposed using crypto, internally, to consistently manage cash flow. Yikes, how hard is it to just capture an immutable event log. Way cheaper than running crypto, even if only internally.
- limit499karma 2y agoTheoretically they even have a better security environment (since it is internal and they control users, code base and network) so the consensus mechanism may not even require BFT.
- imglorp 2y agoHarder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense. Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
- baq 2y agoI'll just leave that here for no particular reason at all: https://www.sec.gov/enforcement-litigation/whistleblower-program https://www.sec.gov/enforcement-litigation/whistleblower-pro...
- phire 2y agoFun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow. As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem). The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.
- oblio 2y ago> Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow. I know you're not arguing in their favor, just describing a reality, but the irony of that phrase is through the roof :-))) Especially the "centralized crypto".
- phire 2y agoYeah, that fact alone goes a long way to proving there is no technical merit to cryptocurrencies. The reason they are now called "centralised crypto exchanges" is that "decentralised crypto exchanges" now exist, where trades do actually happen on a public blockchain. Though, a large chunk of those are "fake", where they look like a decentralised exchange, but there is a central entity holding all the coins in central wallets and can misplace them, or even reverse trades. You kind of get the worst of both worlds, as you are now venerable to front-running, they are slow, and the exchange can still rug pull you. The legit decentralised exchanges are limited to only trading tokens on a given blockchain (usually ethereum), are even slower, are still vulnerable to front-running. Plus, they spam those blockchains with loads of transactions, driving up transaction fees.
- mlloyd 2y agoThis sounds like a situation that I know about at the placed identified by name in your comment. It took months to track down the issue.
- serbuvlad 2y ago> In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. When the banks do this it's called "fractional reserve banking", and they sell it as a good thing. :)
- thaumasiotes 2y agoThere's more to it than that; balances are exceeded by the sum of "assets held by the bank" and "assets owed to the bank".
- deleted 2y ago[deleted]
- Aurornis 2y agoI’m constantly amazed by how much the crypto community thinks they understand fractional reserve banking while getting it so completely wrong. In fractional reserve banking, money that is loaned out is accounted for as liabilities. These liabilities subtract from the overall balance stored (reserved) at the bank. The bank is not printing money new money, no matter how many times this idea gets repeated by people who are, ironically, pumping crypto coins that were printed out of thin air. I think it’s incredible that cryptocurrencies were literally manifested out of bits, but the same people try to criticize banks for doing this same thing (which they don’t).
- zmgsabst 2y agoClarifying question: So for every $1 deposited, I can lend $0.90 but must hold $0.10 as my reserve?
- ianburrell 2y agoThat is exactly what happens. Reserve ratio used to be 10%, same as your example. The reserve ratio is currently zero, lowered in 2020 during pandemics. But banks still can't lend out more than deposits.
- phonon 2y agoSynapse says that it was actually the Bank (Evolve) that made the accounting mistakes, including missing transactions, debits that weren't reported, sending in flight transaction to Mercury while debiting Synapse incorrectly etc. https://lex.substack.com/p/podcast-what-really-happened-at-synapse https://lex.substack.com/p/podcast-what-really-happened-at-s...
- abenga 2y agoFrom a cursory look at how it describes itself (BaaS, etc), Evolve is hardly a "bank" in the traditional sense of the word.
- hn_throwaway_99 2y agoThanks for posting this, I will definitely listen to it. While I haven't listened yet, one thing I don't really buy when it comes to blaming Evolve is that it should fundamentally be Synapse's responsibility to do reconciliation. This is what completely baffled me when I first worked with another BaaS company - they weren't doing any reconciliation of their ledgered accounts with the underlying FBO balances at the partner bank! This was insane to me, and it sound likes Synapse didn't do it either. So even if Evolve did make accounting mistakes and have missing transactions, Synapse should have caught this much earlier by having regular reconciliations and audits.
- phonon 2y agoThey claim they did, Evolve kept putting them off, until they ran out of money. There's a full transcript (with some images) below the player btw.
- rawgabbit 2y agoRambling interview. As best as I can tell Synapse said there were technical issues with Evolve the bank. Meanwhile this article said Synapse withdrew from Evolve the end user funds. Mr. Holmes of Evolve said the bank “transferred all end user funds” to other banks at the request of Synapse, but declined to identify them https://www.nytimes.com/2024/07/09/business/synapse-bankruptcy-fintech-fdic-insurance.html https://www.nytimes.com/2024/07/09/business/synapse-bankrupt...
- hgomersall 2y agoI had money disappear from my HSBC account. As in, the double entries didn't match by a small amount (it was a payment between two of my accounts that didn't match up, which I couldn't trivially reconcile in the books). I pursued this for a while but they never even properly acknowledged it let alone fix it. I had my unfounded suspicion it was some internal subtle theft going on, but incompetence is probably a better explanation.
- deleted 2y ago[deleted]
- tossandthrow 2y agoIf you live in a developed country it should be sufficient to ask them to account for it with a note that a formal complaint will be sent to relevant authorities if nor dealt with in timely manner. That stuff like this is in order is the foundation of kapital societies and is taken quite seriously.
- hgomersall 2y agoYou'd think so wouldn't you? But alas, the effort required to solve this was at least more than I was willing to make.
- sailfast 2y agoIf in the US, the CFPB would handle this for you.
- hombre_fatal 2y agoFunnily enough, that's an agency Musk wants to gut: https://www.politico.com/live-updates/2024/11/27/congress/delete-cfpb-musk-calls-for-elimination-of-consumer-bureau-00191994 https://www.politico.com/live-updates/2024/11/27/congress/de...
- lores 2y agoYou're likely correct it was theft. I was told by a CTO there that topping up accounts with bank money where it has been hacked away was daily routine and cheaper than fixing the systems. Incompetence would not manifest on routine operations like this.
- throwaway2037 2y ago> Fintechs also often put out the fake promise that deposits are FDIC insured Does this still happen?
- koblas 2y agoMany fintechs are not licensed to hold funds and work with bank partners who hold your actual funds. That allows them to say they're insured because they're not co-mingled with the corporate funds in the event of insolvency. This doesn't stop them from making accounting errors.
- fragmede 2y agothe problem is the discrepancy between what the Fintech means when they say fdic insured, and what the customer hears when they're told fdic insured. the customer (erroneously) assumes it means that if the Fintech or anyone else has problems, the customer is covered up to the 250k fdic limit. what the Fintech means, is that there's someone they're partnered with that is a bank and is fdic covered. How the money is deposited into the bank is up for interperation. if there Fintech is being dishonest, they have one bank account at a bank, and all of the customers money goes into that one shared account, they're not technically lying - the money is fdic insured. unfortunately for the customers, that's not the same as each of them being fdic insured is the Fintech goes under. fdic doesn't seem to want to clarify this issue either, which is a problem.
- zie 2y agoThe FDIC said you can't do this anymore starting Jan 1, 2025. So I expect it to stop in about 30 days. The FDIC will probably find a few laggards and throw some fines at them, and the process will then probably completely stop.
- superzamp 2y agoCoincidentally written something about this yesterday [1], but the gist of my take summed up is that the nature of accounting oriented data models doesn’t help when dealing with multiple FBO accounts. The main problem is that accounting defaults to full fungibility of monetary amounts represented in a ledger, which has the effect of losing track of the precise mapping between assets and liabilities, so you end up in a position where you simply cannot tell precisely to a bank _who_ are the actual customers they owe money to. [1] https://www.formance.com/blog/engineering/warehousing-promises https://www.formance.com/blog/engineering/warehousing-promis...
- moffkalast 2y agoThe fact that practically all funding most of the world runs on these days is just a bunch of variables in some shitty program never stops being weird to think about. All it takes to create or destroy trillions is one (or maybe a few) CPU ops. It really stretches the belief into fiat money to the absolute limit.
- bitwize 2y agoDade: This is every financial transaction Ellingson conducts, yeah? From million dollar deals to the ten bucks some guy pays for gas. Kate: The worm eats a few cents from each transaction. Dade: And no one's caught it because the money isn't really gone. It's just data being shifted around.
- PittleyDunkin 2y ago> Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger. I'm not sure there's much difference. Intent only matters so much.
- taeric 2y agoI mean... Fraud is defined by intent. You can argue negligence over mistake. But fraud definitely requires intent.
- deleted 2y ago[deleted]
- PittleyDunkin 2y agoI guess my point is it's as harmful as fraud regardless if we can throw someone in prison.
- taeric 2y agoOh, for sure. But treatment/remediation will heavily change between the cases. Right?
- TexanFeller 2y agoI’m skittish about real banking institutions as well. Vanguard for example outsourced a bunch of their dev work to India a few years ago. Had a friend that worked as a sysadmin for BoA. They were required to keep certain logging for 7 years but he would just delete it anyway when disks were starting to get full.
- deleted 2y ago[deleted]
- hn_throwaway_99 2y agoBut the fundamental difference is that the regulatory structures are in place to recover your money if a bank loses it. That's not the case with fintech middlemen. Take the Synapse case: * End customers are really a customer of Yotta, a (silly IMO) fintech where interest was essentially pooled into a sweepstakes prize. * Yotta was a customer of Synapse - they used Synapse BaaS APIs to open customer accounts (again, these accounts were really just entries in Synapse's ledger, and they underlying funds were supposed to be stored in an FBO account on Evolve). * Synapse partnered with Evolve, who is the FDIC insured bank. Synapse went bankrupt, and Yotta customers are finding out they can't access their money. But the bankruptcy court is at a loss as to really what to do. FDIC isn't getting involved, because as far as they can tell, Evolve hasn't failed. Synapse is basically out of the picture at this point as they are bankrupt and there isn't even enough money left to do a real audit, and Yotta is suing Evolve alleging they lost customer funds. But, in the meantime, Yotta customers are SOL. If you had a direct relationship with an FDIC-insured bank, and they lost your money, there would be a much clearer path for the FDIC to get involved and make you whole (up to $250k).
- Thorrez 2y agoHow about Wealthfront Cash accounts? Wealthfront provides me a statement that shows how my deposited money is distributed among its FDIC insured partner banks, and each transfer they do to and from one of those partner banks. Wealthfront does use a middleman, somewhat similar to how Yotta used Synology as a middleman. But Wealthfront's middleman is FDIC insured: Green Dot Bank.
- e40 2y agoI had been debating the merits of using Flourish, but I'm sticking with SNOXX on Schwab. Same rate and I think SNOXX has to be safer, right? Even with the Flourish FDIC guarantee, as others have pointed out, it's only for the underlying back not Flourish itself.
- ajuc 2y ago> In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger. Bugs are as likely to show more and less money than there really are. But bugs in production will almost always show more :)
- bradleyjg 2y agoIn Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger. If there was no malfeasance then no money would be gone. The totals would add up, they just wouldn’t no know who was owed what. Since the totals don’t add up, someone got the extra.
- chairmansteve 2y ago"Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money". Would you count Wealthfront as a fintech? I was finding their marketing compelling, but this thread makes me think twice.
- dmoy 2y agoYes, it's the same basic principle going on at Wealthfront/etc. It's possible (probable?) that they have better accounting controls. But I personally wouldn't keep anything above SIPC limits at Wealthfront (or any near competitor like Robinhood, M1, etc). And I'd be keeping records on my own. And I'd make peace with the fact that SIPC resolution is a completely different ballgame from FDIC turnaround for assets held directly at an insured bank (which is like single business day don't-even-notice fast). I.e. not use it as the sole source of emergency funds, have months of expenses at a different institution, etc.
- dmoy 2y ago> same basic principle Well yes and no - synapses pass-through-banking wasn't covered by SIPC, and neither would wealthfronts comparable product. But keeping it just in a standard Wealthfront (or synapse even) sweep account with no underlying banking shenanigans happening, is different from SIPC's perspective. Just keeping stocks (up to $500k) or sweep (up to $250k) at a SIPC broker is probably okay, even if it's a new fintech. Fooling around with their weird passthrough stuff, less so.
- hnthrowaway7394 2y agoThere is a pretty fundamental difference, and it’s that Wealthfront (and M1, Robinhood, Fidelity, etc) are registered broker-dealers. Broker-dealers are regulated just as stringently as banks, but by the SEC and FINRA as opposed to the Fed. Broker-dealers have been running passthrough FDIC programs for decades, and in a lot of ways have more stringent regulations than banks. The most notable is that they are forced to segregate assets (can’t put client assets on the balance sheet, they have to custody them separately), which is the ultimate way banks fail and need the FDIC to bail them out. Source: used to work in broker-dealer auditing
- chairmansteve 2y ago90 million dollars missing and 250 million dollars frozen. That 250m probably needed by some people to pay rent. Backed by Andreesen Horowitz who are conducting a scorched earth jihad against all government regulation. https://finance.yahoo.com/personal-finance/synapse-bankruptcy-fintech-safety-183845965.html https://finance.yahoo.com/personal-finance/synapse-bankruptc...
- abirch 2y agoThe sad thing is that most people don't learn lessons from history. It took me far too long to start learning lessons from history. After an asset bubble and collapse people will understand why we have a lot of the regulations from the 1930s.
- manquer 2y agoSadly I don’t think it will happen. In previous crises people could depend being educated by mostly responsible media . Today both mainstream and social are entertainment first don’t care for truth or their role in educating the society . It is more likely they will be taught to blame some boogeymen who have nothing to do with the problem rather address the real one .
- dartos 2y agoMedia was never all that responsible.
- manquer 2y agoMedia biases and ethics in journalism are masters degrees sized topics of their own. Very briefly you are not wrong, yellow journalism has a long history is not new. In America from the times of federalist papers through slavery and jim crow and antisemitism of the 30s, to civil rights and into modern times it has been a powerful tool to shape public opinion. There is nuance to this however, the era of professional journalism has been brief, only 100 or so years. In that time, media had the most impact in shinning the light on the truth, notably with reporting on Watergate, Pentagon papers or Hershey on Hiroshima and so on, that era is coming to an end. As the cost of publishing drops orders of magnitude in every generation of technology as it happened with cheap and fast printing press, radio, broadcast and then cable TV and finally the internet and mobile, the problem becomes more massive and much harder to regulate, and also drops in the quality of public discourse and nature of truth. Basically it boils down to we had a good(relatively) 100 year run with media and corresponding improvement in civil liberties and good governance, we can no longer depend on educated public taking decisions sooner or later in the right direction like we have last century or so.
- ThinkingGuy 2y agoI like Cory Doctorow's saying: "When you hear the term 'fintech,' think 'unlicensed bank.'
- eweise 2y agoIn addition to a ledger, fintechs need a reconciliation system to ensure the ledger is correct. Does the card processor audit files match your ledger? Does your ACH and check processing systems match the ledger? What about external money movements at the sponsor bank. Are they recording in the ledger?