5 ms·

It's not arguing about the difference between web2 and web3. It's arguing two main things 1. Web3 data is different from what most classic databases are optimi

by IanCal 4y ago

It's not arguing about the difference between web2 and web3. It's arguing two main things 1. Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees. 2. This service is better than other web3 api providers. FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me. If it helps, you can remove web3 entirely from the problem to understand it. The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. The other main thing is that there is a standard API that people provide (read: sell) access to, which can be slow and/or expensive if you've got to pull a lot of data from it.

dmitriid 4y ago
> The only thing relevant here is that blockchain data is almost exclusively append-only So, Apache Kafka. Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it > so you can build something neater because you've got guarantees that many other problems don't have. So, first you have to find problems for your solution
- jacobr1 4y ago
  I think you identified the use case: For services that are using the "append-only log as your source of data" then you need a tool that manages the "expensive to reconstruct it" side of things. I think that is what this does: it materializes and abstracts interacting with the reconstructed view of the data.
- IanCal 4y ago
  > So, Apache Kafka. Does that give access to the full history quickly? > Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it Yeah I don't want to do that, but I do need to in order to process the data. For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
  - dmitriid 4y ago
    > Does that give access to the full history quickly? It doesn't because no append-only log does. Apache Kafka is that "unique web3-only thing never seen before in traditional systems" that no only has been seen in traditional systems, but has been successfully used in those systems at scale dwarfing all of crypto, and has well-understood behaviours and patterns. Edit: In 2019 Linkedin (who developed Kafka) were processing 7 trillion messages a day: https://engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages https://engineering.linkedin.com/blog/2019/apache-kafka-tril... Edit: In 2018 Pinterest was handling 15 million messages a day with 1.2 petabytes of data per day https://medium.com/pinterest-engineering/how-pinterest-runs-kafka-at-scale-ff9c6f735be https://medium.com/pinterest-engineering/how-pinterest-runs-... A view of the final state of data is literally one of the things they require, and everyone knows how to make one. > For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer. So this service just creates a "traditional" database with a "traditional" access method (actually an ancient access method, RPC) with a "traditional" ElasticSearch on top and pretends it's something new, never made at scale, and something no "traditional system" was ever prepared for. Edit: see LinkedIn and Pinterest above. Or Facebook. Or Amazon. Or AliExpress. Or... "Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie that I can't imagine people who can look at it and say "true, true". The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
    - IanCal 4y ago
      > It doesn't because no append-only log does. Then it doesn't solve the problem people in the space have. I understand the visceral hate for web3, I do, but it really gets in the way of talking about a very bland technical issue. I think you're misunderstanding what this service is. It's about access to Blockchain data not about running a chain. > A view of the final state of data is literally one of the things they require, and everyone knows how to make one. That's one thing it needs, but only one thing. Keeping up to date with this is a problem of monitoring mostly to keep on the right side of any forks. It's not a data problem, but that's not what the API is providing. > Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie It totally would be, but it's not what the site said. It's literally just saying that they're doing better than off the shelf tools because of guarantees most other problems don't have. And I'm not surprised, the data isn't of a crazy scale though is a bit annoying, access patterns are extremely predictable and depending on which APIs you're supporting building a specific thing for it is fairly straightforward. > The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads. Nobody is talking about ingest speed really. Unless you're also applying all the state transitions because that's quite annoying.
acaloiar 4y ago
"Web3 data" means nothing. If you're referring to blockchain data, say blockchain data. "Web3" doesn't lend anything to technical conversations and is best used on board rooms full of coked up investors interested in dropping money on another dumb crypto bro's vision to change the world.
- IanCal 4y ago
  The descriptions made absolute sense to me, the target audience - someone developing with these systems. You can complain about it if you want, but it's like complaining that serverless things need servers underneath - maybe technically accurate but not a helpful discussion to have.
  - acaloiar 4y ago
    "Serverless" is a terrible adjective, and yet it's still considerably more precise than "web3". Even so, in a technical setting I often hear engineers refers to "hosted function" or "function service" when referring to "serverless" infrastructure.
    - IanCal 4y ago
      The description is precise enough for anyone who would be a potential customer to know exactly what was meant. As marketing copy for a product that seems like all that really matters here. I won't carry on as that's the only point I'm making and I don't think it's productive to try and convince anyone further.
CharlesW 4y ago
> Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees. Nah. Blockchains are (of course?) not the first database use case where immutability is necessary. Blockchains aren't unique in terms of access patterns, either.
- IanCal 4y ago
  I didn't say they were unique, so we agree I guess.
com2kid 4y ago
> The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. Append only databases have been around for decades. Heck Kafka is an obscenely popular example today. I've talked to people working in fields with legally mandated immutable databases, again decades old technology. In the most extreme cases the immutability is enforced using technology, such as a storage medium that only supports writing and not deleting or changing data once it has been written. Distributed append only databases based on not trusting the people you are doing business with, and where writes cost small fortune, sure, that is new. But the # times I want to make large transactions where: 1. I don't trust the other party 2. I can't use one of the insanely well trusted escrow agencies, some of whom have been around for literally hundreds of years and have built up trust accordingly 3. I am OK with the transaction not being reversible through the typical legal system remedies is miniscule. The idea of a distributed ownership ledger is more interesting, but again, if you have any level of trust with your partners, it is cheaper to throw away the entire proof of X system and replace it with a regular append only database. That said, the provided service may very well be a better way to access existing web3 services. > FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me. Have you found any uses for blockchain that are not better served, or cannot be served, by traditional solutions?
- IanCal 4y ago
  I think you're arguing around something very different to what I'm saying. None of this is to do with whether web3 stuff is a good idea, or whether the underlying Blockchain setup/concept is useful or whether distributed whatevers are good. This is a product launch giving access to that data under a common API.
  - com2kid 4y ago
    I agree that the product itself seems like a nicer way to access web3 stuff! And I'd actually be interested in some blog posts about how the authors got such dramatic speedups.

by IanCal 4y ago

dmitriid 4y ago

> The only thing relevant here is that blockchain data is almost exclusively append-only So, Apache Kafka. Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it > so you can build something neater because you've got guarantees that many other problems don't have. So, first you have to find problems for your solution

jacobr1 4y ago
I think you identified the use case: For services that are using the "append-only log as your source of data" then you need a tool that manages the "expensive to reconstruct it" side of things. I think that is what this does: it materializes and abstracts interacting with the reconstructed view of the data.
IanCal 4y ago
> So, Apache Kafka. Does that give access to the full history quickly? > Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it Yeah I don't want to do that, but I do need to in order to process the data. For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
- dmitriid 4y ago
  > Does that give access to the full history quickly? It doesn't because no append-only log does. Apache Kafka is that "unique web3-only thing never seen before in traditional systems" that no only has been seen in traditional systems, but has been successfully used in those systems at scale dwarfing all of crypto, and has well-understood behaviours and patterns. Edit: In 2019 Linkedin (who developed Kafka) were processing 7 trillion messages a day: https://engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages https://engineering.linkedin.com/blog/2019/apache-kafka-tril... Edit: In 2018 Pinterest was handling 15 million messages a day with 1.2 petabytes of data per day https://medium.com/pinterest-engineering/how-pinterest-runs-kafka-at-scale-ff9c6f735be https://medium.com/pinterest-engineering/how-pinterest-runs-... A view of the final state of data is literally one of the things they require, and everyone knows how to make one. > For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer. So this service just creates a "traditional" database with a "traditional" access method (actually an ancient access method, RPC) with a "traditional" ElasticSearch on top and pretends it's something new, never made at scale, and something no "traditional system" was ever prepared for. Edit: see LinkedIn and Pinterest above. Or Facebook. Or Amazon. Or AliExpress. Or... "Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie that I can't imagine people who can look at it and say "true, true". The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
  - IanCal 4y ago
    > It doesn't because no append-only log does. Then it doesn't solve the problem people in the space have. I understand the visceral hate for web3, I do, but it really gets in the way of talking about a very bland technical issue. I think you're misunderstanding what this service is. It's about access to Blockchain data not about running a chain. > A view of the final state of data is literally one of the things they require, and everyone knows how to make one. That's one thing it needs, but only one thing. Keeping up to date with this is a problem of monitoring mostly to keep on the right side of any forks. It's not a data problem, but that's not what the API is providing. > Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie It totally would be, but it's not what the site said. It's literally just saying that they're doing better than off the shelf tools because of guarantees most other problems don't have. And I'm not surprised, the data isn't of a crazy scale though is a bit annoying, access patterns are extremely predictable and depending on which APIs you're supporting building a specific thing for it is fairly straightforward. > The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads. Nobody is talking about ingest speed really. Unless you're also applying all the state transitions because that's quite annoying.

acaloiar 4y ago

"Web3 data" means nothing. If you're referring to blockchain data, say blockchain data. "Web3" doesn't lend anything to technical conversations and is best used on board rooms full of coked up investors interested in dropping money on another dumb crypto bro's vision to change the world.

IanCal 4y ago
The descriptions made absolute sense to me, the target audience - someone developing with these systems. You can complain about it if you want, but it's like complaining that serverless things need servers underneath - maybe technically accurate but not a helpful discussion to have.
- acaloiar 4y ago
  "Serverless" is a terrible adjective, and yet it's still considerably more precise than "web3". Even so, in a technical setting I often hear engineers refers to "hosted function" or "function service" when referring to "serverless" infrastructure.
  - IanCal 4y ago
    The description is precise enough for anyone who would be a potential customer to know exactly what was meant. As marketing copy for a product that seems like all that really matters here. I won't carry on as that's the only point I'm making and I don't think it's productive to try and convince anyone further.

CharlesW 4y ago

> Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees. Nah. Blockchains are (of course?) not the first database use case where immutability is necessary. Blockchains aren't unique in terms of access patterns, either.

IanCal 4y ago
I didn't say they were unique, so we agree I guess.

com2kid 4y ago

> The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. Append only databases have been around for decades. Heck Kafka is an obscenely popular example today. I've talked to people working in fields with legally mandated immutable databases, again decades old technology. In the most extreme cases the immutability is enforced using technology, such as a storage medium that only supports writing and not deleting or changing data once it has been written. Distributed append only databases based on not trusting the people you are doing business with, and where writes cost small fortune, sure, that is new. But the # times I want to make large transactions where: 1. I don't trust the other party 2. I can't use one of the insanely well trusted escrow agencies, some of whom have been around for literally hundreds of years and have built up trust accordingly 3. I am OK with the transaction not being reversible through the typical legal system remedies is miniscule. The idea of a distributed ownership ledger is more interesting, but again, if you have any level of trust with your partners, it is cheaper to throw away the entire proof of X system and replace it with a regular append only database. That said, the provided service may very well be a better way to access existing web3 services. > FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me. Have you found any uses for blockchain that are not better served, or cannot be served, by traditional solutions?

IanCal 4y ago
I think you're arguing around something very different to what I'm saying. None of this is to do with whether web3 stuff is a good idea, or whether the underlying Blockchain setup/concept is useful or whether distributed whatevers are good. This is a product launch giving access to that data under a common API.
- com2kid 4y ago
  I agree that the product itself seems like a nicer way to access web3 stuff! And I'd actually be interested in some blog posts about how the authors got such dramatic speedups.