> it should have all the necessary building blocks to do anything that can be done with the table API That's true in that the table api itself is built on top

> it should have all the necessary building blocks to do anything that can be done with the table API That's true in that the table api itself is built on top of the datastream api. But to run the example you'd have to implement your own joins and retraction-aware aggregates and then we'd be testing the consistency properties of that implementation, not of the datastream api itself, and we might as well test a mature implementation like the table api instead. > The file source can be used for testing streaming behavior, for example by using a directory and using a PROCESS_CONTINUOUSELY watch type. Ah, I was only looking at the table api version (https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/filesystem.html https://ci.apache.org/projects/flink/flink-docs-release-1.12...) which doesn't mention that option. I guess I could make stream and turn it into a table? > This is expected, that's how Flink deals with late data... I should clarify in the article that I was surprised not because it's wrong, but because it's very different from the behavior that I'm used to. > Also, it seems very odd to me that a watermark of 5s is used while the expected out-of-orderness is 10s. Why did the author choose to do that? To explore how late data is treated. There are some plausible sources of inconsistency that from late data handling that I didn't get to demonstrate in the article. For example, it sounds like even without allowed_lateness if two different windowed operations had different window sizes they might drop different subsets of late data, which could causes inconsistency between their outputs.