4 ms·

The amount of FUD thrown around about micro batching is really kind of silly. How many streaming analytics use cases are ok with the JVM, ok with 10s of millis

by kod 10y ago

The amount of FUD thrown around about micro batching is really kind of silly. How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?

jsmthrowaway 10y ago
Remove "analytics" from your statement and you'll arrive at the answer, because while analytics is driving the streaming space as we speak it is literally one of the most boring applications of streaming imaginable. I positively cannot get excited about engagement numbers from an event pipeline. Intrusion detection, fraud detection, about half of infrastructure monitoring tasks, event sourcing... You're also glossing over the impact of microbatching which looks more like seconds than milliseconds, as well as what microbatching does to your windowing abilities (such as having to double-process to work around the microbatch interval as applied to your desired windowing semantics). Microbatch latency immediately rules out several useful applications of streaming. I also didn't say anything about the JVM nor it being okay for my purposes. You did. I can back up my statements on streaming from Flink to Samza to Dataflow/Beam to Storm to MillWheel to Spark "Streaming" and back again because it has been my primary focus (literally thinking of nothing else) for a couple years. Please accuse someone else of FUD because that's a relatively veiled way to say "you don't know what you're talking about," and I assure you that you're (condescendingly) wrong. I think you're also coming at that angle from interpreting me as negative on Spark Streaming. Read carefully.
- kod 10y ago
  > You're also glossing over the impact of microbatching which looks more like seconds than milliseconds I've run spark streaming jobs at 250ms batch times. This comment you made, right here, is why I used the word FUD.

by kod 10y ago

jsmthrowaway 10y ago

Remove "analytics" from your statement and you'll arrive at the answer, because while analytics is driving the streaming space as we speak it is literally one of the most boring applications of streaming imaginable. I positively cannot get excited about engagement numbers from an event pipeline. Intrusion detection, fraud detection, about half of infrastructure monitoring tasks, event sourcing... You're also glossing over the impact of microbatching which looks more like seconds than milliseconds, as well as what microbatching does to your windowing abilities (such as having to double-process to work around the microbatch interval as applied to your desired windowing semantics). Microbatch latency immediately rules out several useful applications of streaming. I also didn't say anything about the JVM nor it being okay for my purposes. You did. I can back up my statements on streaming from Flink to Samza to Dataflow/Beam to Storm to MillWheel to Spark "Streaming" and back again because it has been my primary focus (literally thinking of nothing else) for a couple years. Please accuse someone else of FUD because that's a relatively veiled way to say "you don't know what you're talking about," and I assure you that you're (condescendingly) wrong. I think you're also coming at that angle from interpreting me as negative on Spark Streaming. Read carefully.

kod 10y ago
> You're also glossing over the impact of microbatching which looks more like seconds than milliseconds I've run spark streaming jobs at 250ms batch times. This comment you made, right here, is why I used the word FUD.