3 ms·

The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure

The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data". State management is a tricky problem even in a single-threaded application. It's doubly so in distributed systems, where state can be inconsistent between all the moving pieces. This is the source of endless data integrity issues. I think data engineering is a great way to get some exposure to all of this.

darksaints 6y ago
> As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Exactly. You can't magically make a stateful problem stateless, you can merely move that state around. Sometimes moving state around means moving it somewhere that is appropriate and capable of expertly handling that data. But if you make those choices wrong, it makes every aspect of your application more complex. UI programming tried going down this idea of stateless programming, and for a while it was trendy to do so stuff like redux. The problem is that UIs are state machines. That's not an analogy, that is a literal statement. And it is true of all UI's...it's just as true of the transmission lever in your car as it is for your saas dashboard. You can't program stateless UIs...they would cease to be a UI. So at best, you can move that state around. And with most of these solutions (eg. redux), you end up pushing that state into a massive global singleton, where even simple things like the state of a single radio button needs to be fed through dozens of tightly coupled components in order to "statelessly" render. And even worse, you lose the extremely helpful distinction between UI state and domain state, mixing them both together into a gigantic shit stew.
jsinai 6y ago
>The other thing I'd emphasize here is dealing with "state". Data is effectively state. It gets even more complicated. It’s not just the current state that matters, but also the history (sometimes the entire history) up to that state.

darksaints 6y ago

> As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Exactly. You can't magically make a stateful problem stateless, you can merely move that state around. Sometimes moving state around means moving it somewhere that is appropriate and capable of expertly handling that data. But if you make those choices wrong, it makes every aspect of your application more complex. UI programming tried going down this idea of stateless programming, and for a while it was trendy to do so stuff like redux. The problem is that UIs are state machines. That's not an analogy, that is a literal statement. And it is true of all UI's...it's just as true of the transmission lever in your car as it is for your saas dashboard. You can't program stateless UIs...they would cease to be a UI. So at best, you can move that state around. And with most of these solutions (eg. redux), you end up pushing that state into a massive global singleton, where even simple things like the state of a single radio button needs to be fed through dozens of tightly coupled components in order to "statelessly" render. And even worse, you lose the extremely helpful distinction between UI state and domain state, mixing them both together into a gigantic shit stew.

jsinai 6y ago

>The other thing I'd emphasize here is dealing with "state". Data is effectively state. It gets even more complicated. It’s not just the current state that matters, but also the history (sometimes the entire history) up to that state.