4 ms·

How? If Binance had a trillion transactions, that’s 100KB per transaction. What all are they logging?

tommek4077 2y ago
High frequency traders are making hundreds of billions of orders per day. And there are many bigger and smaller players.
francoismassot 2y ago
They have 181 trillion logs
- kstrauser 2y ago
  But of what? What has Binance done 181 trillion times? Obviously they have. I don’t think they’re throwing away money for logs they don’t generate or need. I just can’t imagine the scope of it. That is, I know this is a failing of my imagination, not their engineering decisions. I’d love to fill in my knowledge gaps.
  - xboxnolifes 2y ago
    They are application logs, so probably nearly every click on their website.
  - robxorb 2y ago
    If it's 181 trillion each year, it's only 6 million per second. There's a thousand milliseconds in each second so Binance would need only several thousand high frequency traders creating, and adjusting orders, through their API, to end up with those logs. Binance has hundreds of trading pairs available so a handful on each pair average would add up.
askl 2y ago
Don't forget the logs produced by the logging infrastructure.
- kstrauser 2y ago
  And what if that infra goes down? Who's watching that?
TacticalCoder 2y ago
Yeah I don't get it either, something seems deeply wrong here. These are impressive numbers but I wonder about something... Binance is a centralized cryptocurrencies exchange. But AIUI "defi" is a thing: instead of using a CEX (Centralized EXchange), people can use a DEX (Decentralized EXchange). And apparently there's a gigantic number of trades happening in the defi world. And it's all happening on public ledgers right? (I'm asking, I don't know: are the "level 2" chains public?). And the sum of all the public ledgers / blockchains transactions do not represent anywhere near 1.6 PB a day. And yet DEXes do work (and often now, seen that volume picked up, have liquidity and fees cheaper than centralized exchanges). Someone who know this stuff better than I do could comment but from a quick googling here's what I found: - a full Ethereum node today is 1.2 TB (not PB, but TB) - a full Bitcoin node today is 585 GB There are other blockchains but these two are the two most successful ones? So let's take Ethereum... For 1.2 TB you have the history of all the transactions that ever happened on Ethereum, since 2015 or something. And that's not just Ethereum but also all the "tokens" and "NFTs" on ethereum, including stablecoins like Circle/Coinbase's USDC. How do we go from a decentralized "1.2 TB for the entire Ethereum history" to "1.6 PB per day for Binance transactions"? That's 1500x the size of the freaking entire Ethereum blockchain generated by logs, in a day. And Ethereum is, basically, a public ledger. So it is... Logs? Or let's compared Binance's numbers to, say, the US equities options market. That feed is a bit less than 40 Gb/s I think, so 140 TB for a day of actual equities options trading datafeed. I understand these aren't "logs" but, same thing... How do we go from a daily 140 TB datafeed from the CBOE (where the big guys are and the real stuff if happening) to 10x that amount in daily Binance logs. Something doesn't sound right. You can say it's apples vs oranges but I don't it's that much of apples vs oranges. I mean: if these 1.6 PB of Binance logs per day are justified, it makes me think it's time to sell everything and go all-in in cryptocurrencies, because there may be way more interest and activity than people think. (yeah, I'm kidding) EDIT: 1.2 TB for Ethereum seems to be a full but not an "archive" node. An "archive" node is 6 TB. Still a far cry from 1.6 PB a day.
- arandomusername 2y ago
  Archive node can go as low as ~2TB depending on the client, but regardless you do want to compare it to a full node, as the only difference is an "archive" node keeps the state at any block (e.g you want to replay a transaction that occured in the past, or see a state at a certain block). The full nodes still contain all events/txs (DEXes will emit events, akin to logs, for transfers) - so it's fair to compare it to a full node. On CEXes you see a lot more trades, market makers/high frequency trades just doing a ton of volume/creating orders and so on, since it's a lot cheaper. CEXes without a doubt have more trades than DEX by orders of magnitude. Additionally, they probably log multiple stuff for every request, including API request, which they most definitely get a ton of.

High frequency traders are making hundreds of billions of orders per day. And there are many bigger and smaller players.

francoismassot 2y ago

They have 181 trillion logs

kstrauser 2y ago
But of what? What has Binance done 181 trillion times? Obviously they have. I don’t think they’re throwing away money for logs they don’t generate or need. I just can’t imagine the scope of it. That is, I know this is a failing of my imagination, not their engineering decisions. I’d love to fill in my knowledge gaps.
- xboxnolifes 2y ago
  They are application logs, so probably nearly every click on their website.
- robxorb 2y ago
  If it's 181 trillion each year, it's only 6 million per second. There's a thousand milliseconds in each second so Binance would need only several thousand high frequency traders creating, and adjusting orders, through their API, to end up with those logs. Binance has hundreds of trading pairs available so a handful on each pair average would add up.

askl 2y ago

Don't forget the logs produced by the logging infrastructure.

kstrauser 2y ago
And what if that infra goes down? Who's watching that?

TacticalCoder 2y ago

Yeah I don't get it either, something seems deeply wrong here. These are impressive numbers but I wonder about something... Binance is a centralized cryptocurrencies exchange. But AIUI "defi" is a thing: instead of using a CEX (Centralized EXchange), people can use a DEX (Decentralized EXchange). And apparently there's a gigantic number of trades happening in the defi world. And it's all happening on public ledgers right? (I'm asking, I don't know: are the "level 2" chains public?). And the sum of all the public ledgers / blockchains transactions do not represent anywhere near 1.6 PB a day. And yet DEXes do work (and often now, seen that volume picked up, have liquidity and fees cheaper than centralized exchanges). Someone who know this stuff better than I do could comment but from a quick googling here's what I found: - a full Ethereum node today is 1.2 TB (not PB, but TB) - a full Bitcoin node today is 585 GB There are other blockchains but these two are the two most successful ones? So let's take Ethereum... For 1.2 TB you have the history of all the transactions that ever happened on Ethereum, since 2015 or something. And that's not just Ethereum but also all the "tokens" and "NFTs" on ethereum, including stablecoins like Circle/Coinbase's USDC. How do we go from a decentralized "1.2 TB for the entire Ethereum history" to "1.6 PB per day for Binance transactions"? That's 1500x the size of the freaking entire Ethereum blockchain generated by logs, in a day. And Ethereum is, basically, a public ledger. So it is... Logs? Or let's compared Binance's numbers to, say, the US equities options market. That feed is a bit less than 40 Gb/s I think, so 140 TB for a day of actual equities options trading datafeed. I understand these aren't "logs" but, same thing... How do we go from a daily 140 TB datafeed from the CBOE (where the big guys are and the real stuff if happening) to 10x that amount in daily Binance logs. Something doesn't sound right. You can say it's apples vs oranges but I don't it's that much of apples vs oranges. I mean: if these 1.6 PB of Binance logs per day are justified, it makes me think it's time to sell everything and go all-in in cryptocurrencies, because there may be way more interest and activity than people think. (yeah, I'm kidding) EDIT: 1.2 TB for Ethereum seems to be a full but not an "archive" node. An "archive" node is 6 TB. Still a far cry from 1.6 PB a day.

arandomusername 2y ago
Archive node can go as low as ~2TB depending on the client, but regardless you do want to compare it to a full node, as the only difference is an "archive" node keeps the state at any block (e.g you want to replay a transaction that occured in the past, or see a state at a certain block). The full nodes still contain all events/txs (DEXes will emit events, akin to logs, for transfers) - so it's fair to compare it to a full node. On CEXes you see a lot more trades, market makers/high frequency trades just doing a ton of volume/creating orders and so on, since it's a lot cheaper. CEXes without a doubt have more trades than DEX by orders of magnitude. Additionally, they probably log multiple stuff for every request, including API request, which they most definitely get a ton of.