3 ms·
An opinionated map of incremental and streaming systems (2018)
- dochtman 6y agoThis is actually not (2018), it’s from a few days ago.
- tablespoon 6y agoYeah, seems like the OP goofed when parsing the date: > Last updated 2021-04-18
- BenoitP 6y agoI've been thinking a bit since the last article from the author on Internal Consistency [1] on HN [2] on how to stress test a framework, or an application written on it. The problem is: it is hard to know in advance what kind of pressures your production system will be submitted to. I think the right approach would be Jepsen-like: generate inputs and delay them by inconveniently crafted amounts. Maybe take a look at the execution graph to infer what to delay (for example Rows that will be joined further down the graph). One could implement dedicated operators for DataStreams; and for SQL some bytecode could be injected in Flink's Operator.processElement to add delays. [1] https://scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/ https://scattered-thoughts.net/writing/internal-consistency-... [2] https://news.ycombinator.com/item?id=26851803 https://news.ycombinator.com/item?id=26851803