5 ms·

> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?

stingraycharles 8y ago
It’s difficult to search through, query, run projections. Also the API assumes you want to stream realtime data, rather than query historical data.
- lbradstreet 8y ago
  As a side note, we provide tooling to maintain Amazon Athena/Hive,Presto support over your Pyrostore archived data in S3, while maintaining its ability to be streamed in-order.
- linkmotif 8y ago
  I think you can use kafka streams to create whatever projections you want. Kafka can handle millions of small records per second. You can use the Kafka streams api to process all your data either into local rocks db stores or write the data to whatever database you want. This requires some work but then you have a good thing going if it works.
- sidlls 8y ago
  Use the correct tool for the job: hook an analytics DB up to the Kafka pipe and store the data for future queries. Kafka was never intended to support your use case. If your inbound data that you'd like to put to Kafka isn't large, by the way, just write straight to the DB. It's irritating to see Kafka used where it's not necessary. It adds complexity to an infrastructure and the cost for doing so has to be justifiable.
ryanworl 8y ago
One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data. An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something like “topic-partition-offset” where offset is the offset of the first message contained within that file. You can then read those forward starting from offset zero and go until you reach the end, then start reading from Kafka for recent data. The drawback is this isn’t integrated with Kafka so you’re now maintaining what is effectively two different systems for the same data. It also means the key-based compaction won’t work either and you’d have to re-implement that on top of the files in S3 as well.
- linkmotif 8y ago
  > or deal with a repartition of all data. Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?
  - ryanworl 8y ago
    That sounds possible, but it would both involve downtime or potentially ordering and data duplication issues if you mess it up. Dynamic expansion and contraction of partition count should be a feature that doesn’t require recreating the entire cluster like essentially every other data product in the world.
    - jbs40 8y ago
      The architectural pendulum is starting to swing away from co-location of storage and compute (the trend of the last 10+ years) to decoupling of storage and processing to avoid exactly these issues, but legacy architectures hang on for a while. In the streaming and messaging space, Apache Pulsar (pulsar.apache.org) is a more recent solution that has an architecture that decouples processing and storage. That gives you nice properties like independent scaling of storage and processing, infinite data retention, dynamic resizing and others.
      - gazarsgo 8y ago
        What pendulum do you see? From here, architectural patterns are clearly converging on a "distributed mainframe" model between containerization and lambda/kappa architectures... I think Joyent's Manta was ahead of its time in colocating compute and storage and I suspect we'll see more along this vein with the recent open sourcing of FoundationDB.
        jbs40 8y ago
        I was thinking more specifically of the internal architectures of data processing platforms, especially the categorizations that emerged from the MPP database world. The "shared nothing" architecture has been dominant in databases (and is also the core architecture of Hadoop), designed around "co-locating data and compute". Kafka largely follows that architecture as well, using local disk on the compute nodes as its persistent storage layer. A lot of new data processing platforms, from Snowflake in the data warehouse world to AWS Athena to Apache Pulsar in the broader data processing world, have moved to decoupled architectures. Containerization and container management frameworks (e.g. Kubernetes) certainly do change the meaning of "local" storage, will be interesting to see how that plays out.

It’s difficult to search through, query, run projections. Also the API assumes you want to stream realtime data, rather than query historical data.

lbradstreet 8y ago
As a side note, we provide tooling to maintain Amazon Athena/Hive,Presto support over your Pyrostore archived data in S3, while maintaining its ability to be streamed in-order.
linkmotif 8y ago
I think you can use kafka streams to create whatever projections you want. Kafka can handle millions of small records per second. You can use the Kafka streams api to process all your data either into local rocks db stores or write the data to whatever database you want. This requires some work but then you have a good thing going if it works.
sidlls 8y ago
Use the correct tool for the job: hook an analytics DB up to the Kafka pipe and store the data for future queries. Kafka was never intended to support your use case. If your inbound data that you'd like to put to Kafka isn't large, by the way, just write straight to the DB. It's irritating to see Kafka used where it's not necessary. It adds complexity to an infrastructure and the cost for doing so has to be justifiable.

ryanworl 8y ago

One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data. An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something like “topic-partition-offset” where offset is the offset of the first message contained within that file. You can then read those forward starting from offset zero and go until you reach the end, then start reading from Kafka for recent data. The drawback is this isn’t integrated with Kafka so you’re now maintaining what is effectively two different systems for the same data. It also means the key-based compaction won’t work either and you’d have to re-implement that on top of the files in S3 as well.

linkmotif 8y ago
> or deal with a repartition of all data. Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?
- ryanworl 8y ago
  That sounds possible, but it would both involve downtime or potentially ordering and data duplication issues if you mess it up. Dynamic expansion and contraction of partition count should be a feature that doesn’t require recreating the entire cluster like essentially every other data product in the world.
  - jbs40 8y ago
    The architectural pendulum is starting to swing away from co-location of storage and compute (the trend of the last 10+ years) to decoupling of storage and processing to avoid exactly these issues, but legacy architectures hang on for a while. In the streaming and messaging space, Apache Pulsar (pulsar.apache.org) is a more recent solution that has an architecture that decouples processing and storage. That gives you nice properties like independent scaling of storage and processing, infinite data retention, dynamic resizing and others.
    - gazarsgo 8y ago
      What pendulum do you see? From here, architectural patterns are clearly converging on a "distributed mainframe" model between containerization and lambda/kappa architectures... I think Joyent's Manta was ahead of its time in colocating compute and storage and I suspect we'll see more along this vein with the recent open sourcing of FoundationDB.
      - jbs40 8y ago
        I was thinking more specifically of the internal architectures of data processing platforms, especially the categorizations that emerged from the MPP database world. The "shared nothing" architecture has been dominant in databases (and is also the core architecture of Hadoop), designed around "co-locating data and compute". Kafka largely follows that architecture as well, using local disk on the compute nodes as its persistent storage layer. A lot of new data processing platforms, from Snowflake in the data warehouse world to AWS Athena to Apache Pulsar in the broader data processing world, have moved to decoupled architectures. Containerization and container management frameworks (e.g. Kubernetes) certainly do change the meaning of "local" storage, will be interesting to see how that plays out.