2 ms·

No idea where this fits on the priority list, but I think a lot of problems around stream processing still aren't solved, and it's holding us back from a really

by thom 7y ago

No idea where this fits on the priority list, but I think a lot of problems around stream processing still aren't solved, and it's holding us back from a really productive programming paradigm. Handling updates/retractions elegantly is hard or impossible on many platforms, handling late (sometimes _extremely_ late) data can be very inefficient. Working with complex dependencies between events (beyond just time-based windows), in realtime, can be really tough. As the saying goes, cache invalidation is one of the hardest problems in software engineering. Having a simple platform to represent processing as a DAG, but fully supporting both short and long term changes transparently would make event sourcing architectures trivial and extremely productive. The closest we've come seems to be: https://github.com/frankmcsherry/differential-dataflow https://github.com/frankmcsherry/differential-dataflow Lots of very active CS research in this area though.

btown 7y ago
When humans enter the mix, it becomes really complicated. Your event sourcing code can no longer look at events like "Human A changed the record with primary key B to have property C equal to D at time T" because that event is actually a function of the snapshot of how Human A saw the entire state of the interface at time T. And what if someone edited the record with primary key B to represent an entirely different entity (from the perspective of a rational human observer)? Is it still the case that a prior event on this record should be retroactively interpreted to affect the new identity? All code becomes recursively dependent on all prior code, and all events become recursively dependent on prior events. Figuring out how to code in a paradigm where you can't retire old code is very difficult and I'd welcome people pushing the envelope. Martin Kleppman's articles and research is a great place to start for anyone interested: https://martin.kleppmann.com/2015/02/11/database-inside-out-at-salesforce.html https://martin.kleppmann.com/2015/02/11/database-inside-out-...

by thom 7y ago

btown 7y ago

When humans enter the mix, it becomes really complicated. Your event sourcing code can no longer look at events like "Human A changed the record with primary key B to have property C equal to D at time T" because that event is actually a function of the snapshot of how Human A saw the entire state of the interface at time T. And what if someone edited the record with primary key B to represent an entirely different entity (from the perspective of a rational human observer)? Is it still the case that a prior event on this record should be retroactively interpreted to affect the new identity? All code becomes recursively dependent on all prior code, and all events become recursively dependent on prior events. Figuring out how to code in a paradigm where you can't retire old code is very difficult and I'd welcome people pushing the envelope. Martin Kleppman's articles and research is a great place to start for anyone interested: https://martin.kleppmann.com/2015/02/11/database-inside-out-at-salesforce.html https://martin.kleppmann.com/2015/02/11/database-inside-out-...