4 ms·

Thanks. That's kinda what I figured, but wanted to sounding board it out a bit as a sanity check. The link is a great reference by the way. > Your API should

by kortex 5y ago

Thanks. That's kinda what I figured, but wanted to sounding board it out a bit as a sanity check. The link is a great reference by the way. > Your API should use cloud storage (for example, AWS S3) and simply push a reference to S3 to Kafka or any other message broker. This is more or less what I figured. We already archive to S3 anyways so switching to using it as transport would be straightforward.

hardwaresofton 5y ago
> Thanks. That's kinda what I figured, but wanted to sounding board it out a bit as a sanity check. I'm by no means a Kafka expert or a video expert of course, but glad I could serve as a rubber duck. Maybe there's some lessons to be learned from Encore?[0] > The link is a great reference by the way. Yeah the amount of info in there is pretty good -- feels like Kafka could definitely be tuned to do the job but maybe it's better to just start with something better attuned. > This is more or less what I figured. We already archive to S3 anyways so switching to using it as transport would be straightforward. Yeah I figured this is what you were trying to avoid -- the round trips to S3 to get the data to the processing would be wasteful if the data is in this case small enough to flow along the processing route. Guess it really depends on your data. I could have sworn I saw some analysis of how kafka performs versus the size of messages it must deliver... Looks like DZone has some good content[1], LinkedIn of course[2]... Ah I finally found the one I was looking for and it's DZone[3]. All those links make mention of message size [0]: https://svt.github.io/encore-doc/ https://svt.github.io/encore-doc/ [1]: https://dzone.com/articles/processing-large-messages-with-apache-kafka https://dzone.com/articles/processing-large-messages-with-ap... [2]: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines https://engineering.linkedin.com/kafka/benchmarking-apache-k... [3]: https://dzone.com/articles/benchmarking-nats-streaming-and-apache-kafka https://dzone.com/articles/benchmarking-nats-streaming-and-a...

by kortex 5y ago

hardwaresofton 5y ago

> Thanks. That's kinda what I figured, but wanted to sounding board it out a bit as a sanity check. I'm by no means a Kafka expert or a video expert of course, but glad I could serve as a rubber duck. Maybe there's some lessons to be learned from Encore?[0] > The link is a great reference by the way. Yeah the amount of info in there is pretty good -- feels like Kafka could definitely be tuned to do the job but maybe it's better to just start with something better attuned. > This is more or less what I figured. We already archive to S3 anyways so switching to using it as transport would be straightforward. Yeah I figured this is what you were trying to avoid -- the round trips to S3 to get the data to the processing would be wasteful if the data is in this case small enough to flow along the processing route. Guess it really depends on your data. I could have sworn I saw some analysis of how kafka performs versus the size of messages it must deliver... Looks like DZone has some good content[1], LinkedIn of course[2]... Ah I finally found the one I was looking for and it's DZone[3]. All those links make mention of message size [0]: https://svt.github.io/encore-doc/ https://svt.github.io/encore-doc/ [1]: https://dzone.com/articles/processing-large-messages-with-apache-kafka https://dzone.com/articles/processing-large-messages-with-ap... [2]: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines https://engineering.linkedin.com/kafka/benchmarking-apache-k... [3]: https://dzone.com/articles/benchmarking-nats-streaming-and-apache-kafka https://dzone.com/articles/benchmarking-nats-streaming-and-a...