4 ms·

> The only thing relevant here is that blockchain data is almost exclusively append-only So, Apache Kafka. Edit: as anyone who actually does this for a living

> The only thing relevant here is that blockchain data is almost exclusively append-only So, Apache Kafka. Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it > so you can build something neater because you've got guarantees that many other problems don't have. So, first you have to find problems for your solution

jacobr1 4y ago
I think you identified the use case: For services that are using the "append-only log as your source of data" then you need a tool that manages the "expensive to reconstruct it" side of things. I think that is what this does: it materializes and abstracts interacting with the reconstructed view of the data.
IanCal 4y ago
> So, Apache Kafka. Does that give access to the full history quickly? > Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it Yeah I don't want to do that, but I do need to in order to process the data. For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
- dmitriid 4y ago
  > Does that give access to the full history quickly? It doesn't because no append-only log does. Apache Kafka is that "unique web3-only thing never seen before in traditional systems" that no only has been seen in traditional systems, but has been successfully used in those systems at scale dwarfing all of crypto, and has well-understood behaviours and patterns. Edit: In 2019 Linkedin (who developed Kafka) were processing 7 trillion messages a day: https://engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages https://engineering.linkedin.com/blog/2019/apache-kafka-tril... Edit: In 2018 Pinterest was handling 15 million messages a day with 1.2 petabytes of data per day https://medium.com/pinterest-engineering/how-pinterest-runs-kafka-at-scale-ff9c6f735be https://medium.com/pinterest-engineering/how-pinterest-runs-... A view of the final state of data is literally one of the things they require, and everyone knows how to make one. > For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer. So this service just creates a "traditional" database with a "traditional" access method (actually an ancient access method, RPC) with a "traditional" ElasticSearch on top and pretends it's something new, never made at scale, and something no "traditional system" was ever prepared for. Edit: see LinkedIn and Pinterest above. Or Facebook. Or Amazon. Or AliExpress. Or... "Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie that I can't imagine people who can look at it and say "true, true". The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
  - IanCal 4y ago
    > It doesn't because no append-only log does. Then it doesn't solve the problem people in the space have. I understand the visceral hate for web3, I do, but it really gets in the way of talking about a very bland technical issue. I think you're misunderstanding what this service is. It's about access to Blockchain data not about running a chain. > A view of the final state of data is literally one of the things they require, and everyone knows how to make one. That's one thing it needs, but only one thing. Keeping up to date with this is a problem of monitoring mostly to keep on the right side of any forks. It's not a data problem, but that's not what the API is providing. > Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie It totally would be, but it's not what the site said. It's literally just saying that they're doing better than off the shelf tools because of guarantees most other problems don't have. And I'm not surprised, the data isn't of a crazy scale though is a bit annoying, access patterns are extremely predictable and depending on which APIs you're supporting building a specific thing for it is fairly straightforward. > The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads. Nobody is talking about ingest speed really. Unless you're also applying all the state transitions because that's quite annoying.
    - dmitriid 4y ago
      This is what the website literally claims: === start quote === Web2 Databases were meant to deal with datasets that change frequently. They have to provide ACID guarantees to ensure correctness. The solution to this problem becomes more complex with scale. Distributed transactions, sharding, replication, and fault tolerance are notoriously slow. Web2 databases are just NOT designed for web3 A system designed to take full advantage of the fact that blockchain data is immutable, and correctness has been dealt with already. We built a cheaper and simpler storage and indexing system from scratch that can scale to the heaviest blockchain workloads out there. === end quote === Then here's what you claimed === start quote === Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees. The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. === end quote === So, to give you a doze of reality and dispel the lies on the web site I showed that your view of the world does not reflect what the world is: - Blockchain isn't the only thing to have an immutable log. Apache Kafka is one, and is being used at scale that dwarfs anything in crypto space - "The heaviest blockchain workloads out there" as claimed by the web site have literally nothing on "traditional/classical" whatever because traditional/classical handle workloads that exceed blockchains by multiple orders of magnitude - There's nothing revolutionary in the service despite its bullshit claims: they dump blockchain data into a traditional database, slap a traditional (I'd say conservative) JSON-RPC API on top, and probably add a traditional Elastic Search on the side. Everything that they accuse "traditional systems" incapable of doing and incapable of handling. Even though, again, traditional systems can probably handle all this on a free tier of any cloud hosting provider We've known how to handle append-only mostly-immutable data at scale using "traditional" methods for ages. Literally every banking system is all but required to have an append-only log of transactions for audit purposes. And yet here we are with clueless gullible fools pretending this is something new and something that can't be handled by "outdated slow technologies of yore".

jacobr1 4y ago

I think you identified the use case: For services that are using the "append-only log as your source of data" then you need a tool that manages the "expensive to reconstruct it" side of things. I think that is what this does: it materializes and abstracts interacting with the reconstructed view of the data.

IanCal 4y ago

> So, Apache Kafka. Does that give access to the full history quickly? > Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it Yeah I don't want to do that, but I do need to in order to process the data. For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.

dmitriid 4y ago
> Does that give access to the full history quickly? It doesn't because no append-only log does. Apache Kafka is that "unique web3-only thing never seen before in traditional systems" that no only has been seen in traditional systems, but has been successfully used in those systems at scale dwarfing all of crypto, and has well-understood behaviours and patterns. Edit: In 2019 Linkedin (who developed Kafka) were processing 7 trillion messages a day: https://engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages https://engineering.linkedin.com/blog/2019/apache-kafka-tril... Edit: In 2018 Pinterest was handling 15 million messages a day with 1.2 petabytes of data per day https://medium.com/pinterest-engineering/how-pinterest-runs-kafka-at-scale-ff9c6f735be https://medium.com/pinterest-engineering/how-pinterest-runs-... A view of the final state of data is literally one of the things they require, and everyone knows how to make one. > For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer. So this service just creates a "traditional" database with a "traditional" access method (actually an ancient access method, RPC) with a "traditional" ElasticSearch on top and pretends it's something new, never made at scale, and something no "traditional system" was ever prepared for. Edit: see LinkedIn and Pinterest above. Or Facebook. Or Amazon. Or AliExpress. Or... "Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie that I can't imagine people who can look at it and say "true, true". The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
- IanCal 4y ago
  > It doesn't because no append-only log does. Then it doesn't solve the problem people in the space have. I understand the visceral hate for web3, I do, but it really gets in the way of talking about a very bland technical issue. I think you're misunderstanding what this service is. It's about access to Blockchain data not about running a chain. > A view of the final state of data is literally one of the things they require, and everyone knows how to make one. That's one thing it needs, but only one thing. Keeping up to date with this is a problem of monitoring mostly to keep on the right side of any forks. It's not a data problem, but that's not what the API is providing. > Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie It totally would be, but it's not what the site said. It's literally just saying that they're doing better than off the shelf tools because of guarantees most other problems don't have. And I'm not surprised, the data isn't of a crazy scale though is a bit annoying, access patterns are extremely predictable and depending on which APIs you're supporting building a specific thing for it is fairly straightforward. > The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads. Nobody is talking about ingest speed really. Unless you're also applying all the state transitions because that's quite annoying.
  - dmitriid 4y ago
    This is what the website literally claims: === start quote === Web2 Databases were meant to deal with datasets that change frequently. They have to provide ACID guarantees to ensure correctness. The solution to this problem becomes more complex with scale. Distributed transactions, sharding, replication, and fault tolerance are notoriously slow. Web2 databases are just NOT designed for web3 A system designed to take full advantage of the fact that blockchain data is immutable, and correctness has been dealt with already. We built a cheaper and simpler storage and indexing system from scratch that can scale to the heaviest blockchain workloads out there. === end quote === Then here's what you claimed === start quote === Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees. The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. === end quote === So, to give you a doze of reality and dispel the lies on the web site I showed that your view of the world does not reflect what the world is: - Blockchain isn't the only thing to have an immutable log. Apache Kafka is one, and is being used at scale that dwarfs anything in crypto space - "The heaviest blockchain workloads out there" as claimed by the web site have literally nothing on "traditional/classical" whatever because traditional/classical handle workloads that exceed blockchains by multiple orders of magnitude - There's nothing revolutionary in the service despite its bullshit claims: they dump blockchain data into a traditional database, slap a traditional (I'd say conservative) JSON-RPC API on top, and probably add a traditional Elastic Search on the side. Everything that they accuse "traditional systems" incapable of doing and incapable of handling. Even though, again, traditional systems can probably handle all this on a free tier of any cloud hosting provider We've known how to handle append-only mostly-immutable data at scale using "traditional" methods for ages. Literally every banking system is all but required to have an append-only log of transactions for audit purposes. And yet here we are with clueless gullible fools pretending this is something new and something that can't be handled by "outdated slow technologies of yore".