3 ms·
Better, simpler data flow frameworks. Represent all your business's logic declaratively, or at least succinctly. Connect up all the inputs, batch or streaming.
by thom 5y ago
Better, simpler data flow frameworks. Represent all your business's logic declaratively, or at least succinctly. Connect up all the inputs, batch or streaming. The platform automagically maintains your data, always consistent and up-to-date. Apps are just views on top of this that trigger more events later. Find a bug in some of your logic? Fix it and the entire dataset refreshes, with policies on how you want to deal with external integrations that trigger off the back of that. This is what a lot of enterprise data infrastructure already looks like, but cobbled together and slow and buggy. I don't think adding no/low code on top of that really moves the needle. Make it painless and joyful from beginning to end.
If you're thinking of wearables, I personally don't think there's anything interesting there until someone can work out a decent high-bandwidth way of communicating with your device. Gimme something like subvocalisation that doesn't require me to wear something stupid or talk out loud.
- truculent 5y agoThanks for sharing. Are there any dataflow frameworks you are particularly pleased with at the moment?
- thom 5y agoI think Materialize is a wonderful product for what it is, but is somewhat narrow. At the very least I'd love if it started to incorporate window functions as well as just lateral queries though. A gap in the market that I'd like to see filled is stateful stream processing where you care about sequences of events. Kafka Streams removes the need to care about windows and watermarks, for example, so you can accommodate extremely late data in a simple aggregate, but no frameworks that I'm aware of make it easy to work with state machines over sequential data where events in that sequence might arrive or be updated later. Give me the illusion of being able to run forward-only logic over an entire historical dataset on every update, while remaining fast, and I'd be a very happy camper.
- truculent 5y agoMaterialize looks fantastic. Thank you