5 ms·

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition be

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notification and save it to the list. "Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes. The nice thing about this is it doesn't stop you from also doing the event-based stuff (for "real-time" notifications, for example), so long as it's the secondary approach from design point of view. Ironically when everything is working fine, the event-driven stuff will actually be doing all the work -- but as soon as there's a failure (like, the system happens to reboot at 23:59) you'll be grateful to build this way. I could write paragraphs about the nuance, different approaches, failure modes and the trade-offs with all this -- suffice to say treating events as optional in event-based systems has served me well.

harlanji 6y ago
Apparently if you want spare cash, little books of this stuff could sell for $5-10. Just became aware myself, setting up to write again. That’s to say, this comment probably just saved or made me money. I’ve some hands on but clearly not as much as you exhibit. Thanks. Target audience: Sr SE, reporting to next level up and perhaps offering tradesoffs, or advising peers or reports working on those systems and need a refresher to give a complete answer to a question.
- pjc50 6y ago
  How would you market those? It's been a very long time since I bought a book on software.
- layoutIfNeeded 6y ago
  If this worked as a business model then everyone would be doing it. Everyone also includes people who don’t actually know this stuff, but want to make a quick buck, giving out bad advice. And if you don’t already have the experience, then you don’t have the knowledge to weed out bad advice.
randomsearch 6y ago
well today i just learned something very important, thanks! How can I find more wisdom like this?
- unixhero 6y ago
  I wish I didn't have to be so meme'ey, but to answer your question; By reading more Hacker News ;-)
- BatteryMountain 6y ago
  Stub your toe a few times until enlightenment forces its way into your field of view. Seriously though, if you want learn a different way of building applications where loads of these things pop up, look into Actor Model. Even if you are not going to use Actor Model, reading up about it will teach you a lot about "realtime" applications... My main epiphany was that realtime applications do not exist. Rather they cannot exist. Because our reality is not realtime (or our perception of it is always delayed because our brains/awareness is slow) - we can never truly capture the present moment. Data we see is always in the past. Reality is eventually consistent (from our perspective). All things in our reality runs concurrently, stateless and in their own little time bubble. You can also read up on the concepts behind Event Sourcing. Anyway, it is a huge rabbit hole, but reading up on the ideas behind Actor Model might just blow your mind. I'm convinced it will make a comeback because it is the natural way our reality works and Actor Model mimics it the closest, and it will also work better for multi-core cpu's. For non-tech reading, try The Power of Now by Eckhart Tolle. Then try to reflect on (relational databases) + (actor model) + (event sourcing) + (power of now) + (brain in a vat philosophy) + (try to to figure out at what speed the universe "renders", how fast our brains can think, how much delay there is, what limits do our body's sensors limits impose) + (try to figure out how closely Object Oriented programming can mimic reality (hint: all programming are tiny simulations that try to mimic some tiny part of our reality)) + (read up on the history of time keeping and the huge rabbit hole of calendars, and how most programming languages attempts to handle time (badly for the most part, it's a legit rabbit hole on it's own)). After a year or two of hard reading, reflection and teachings (and wrecking your head), you may have good insight into the nature of programming and the fallacies of trying to build realtime systems, or trying to mimic our reality in a computer in general. It is insanely complex, we as programmers merely touch a grain of sand of the beach of complexity. We have no idea how simple our most convoluted systems are in the face of nature, yet we convince ourselves that we KNOW how things work and that we are in control at all times. Good luck!
yosamino 6y ago
Is what you are describing not what they call "event-sourcing" ? Just keep a log of all the events that have already happened (crucial - I have seen "event logs" that were more like "request logs") and to find out the current state of the system - if the notification or report has has been sent - by examining this log. The lists you are describing kind of sound like an event log of sorts. It seems to me that this distinction between primary and secondary approach is not really neccessary.
- hvidgaard 6y ago
  The more structured way to do this is a CQRS framework with Event Sourcing as the underlying data store model. With that you create specialized readmodels that react to events and you can query them for the exact thing you need to know. Once you do something based on that query it will result in new events that update the readmodel. It's a powerful way of designing a system.
manigandham 6y ago
This is basically like going from imperative to declarative. Describe the final end/desired state and let the machine figure out how to get there, progressing through the work until it's caught up even across restarts or other boundaries.
- snidane 6y ago
  The outcome is the same. Except that it is not declarative, because 'you' are the machine who codes up that logic.
lloeki 6y ago
I cannot upvote this enough. One of the experiences I had was applying it to provisioning with idempotent shell scripts, combined with a trivial tool like https://github.com/lloeki/apply https://github.com/lloeki/apply Creating a VM in DO would add the basic SSH keys, then just run the thing against the new IP. Boom, VM ready. Want to make existing VMs config up to date? Apply. Boom. The whole thing ridiculously scaled up in a totally unexpected way, saving hundreds of hours of headaches and mistakes. Why not ansible/puppet? Check out the rationale in the readme.
- nanis 6y ago
  See also https://docs.chef.io/chef_solo/ https://docs.chef.io/chef_solo/
lathiat 6y ago
The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc. This was common when I was writing puppet. It was easy to have your entire puppet run take minutes - and want to run every few minutes - while consuming a LOT of cpu time including on a centralised server and not just the end host. There are of course ways and strategies to this and not every situational falls afoul of it. But it is something you very much need to be aware of. I otherwise whole heartedly endorse this.
- gregmac 6y ago
  > If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. Definitely! And on top of that, you probably want to apply some business logic to what you're doing. To expand on the example of "Report should run at midnight": If you run the check regularly, you can constrain the date range you're checking to just since the last time you checked, and store this date either in memory or persisted depending on your use case, technology, and how you want system failures/restarts handled. Similarly, if the system was down for several days, it might not be relevant to send all missing reports, but instead just the last one. On the other hand, if "daily reports" are required for auditing purposes, it probably does make sense to send all of them. But you have to be careful so on the very first run it doesn't go "Hey, there are 18,726 missing reports since Jan 1, 1970!" It's also important to write the report itself to be idempotent. You can't write it so it reports to "current" time with the assumption it runs at exactly midnight. Instead you have to actually constrain the dates to exactly what you want. Even aside from the idempotent invocation I'm talking about, as your system gets big you'll get thousands of reports that all need to run at "midnight" and that just isn't practical: maybe they run all sequentially, and some will not actually execute until several minutes after midnight.
  - 91aintprime 6y ago
    suppose you serve clients who want their reports to be generated at 02:30 local time. you could try to run a cron container for every time zone :bleh:, but with daylight savings, the 2:30 event can fire twice, or not at all! instead you simply see if there's a report for yesterday, and if there isn't make one. the edge cases are still there, see replies
mandeepj 6y ago
> "Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes. Ha! You are biased toward the word "Idempotent way". Above - you have totally different steps under "Idempotent way". So, your process is different for each of them. There's no tech differentiator and it should be - as that's how you implement "Idempotent way"
tareqak 6y ago
The idempotent way is definitely better, but I argue/add that it would be better to have the incoming business requirement phrased appropriately such that the idempotent approach is the first one that comes to mind. I am not sure how to perform that rewording/transformation in general though or even for the examples above.
xnx 6y ago
I've set up a bunch of Windows batch scripts similarly, but didn't know the name for it. It's beautiful because failures, crashes, bugs, restarts, etc. barely matter. The next time the system is able to do what it was supposed to do, it does.

harlanji 6y ago

Apparently if you want spare cash, little books of this stuff could sell for $5-10. Just became aware myself, setting up to write again. That’s to say, this comment probably just saved or made me money. I’ve some hands on but clearly not as much as you exhibit. Thanks. Target audience: Sr SE, reporting to next level up and perhaps offering tradesoffs, or advising peers or reports working on those systems and need a refresher to give a complete answer to a question.

pjc50 6y ago
How would you market those? It's been a very long time since I bought a book on software.
layoutIfNeeded 6y ago
If this worked as a business model then everyone would be doing it. Everyone also includes people who don’t actually know this stuff, but want to make a quick buck, giving out bad advice. And if you don’t already have the experience, then you don’t have the knowledge to weed out bad advice.

randomsearch 6y ago

well today i just learned something very important, thanks! How can I find more wisdom like this?

unixhero 6y ago
I wish I didn't have to be so meme'ey, but to answer your question; By reading more Hacker News ;-)
BatteryMountain 6y ago
Stub your toe a few times until enlightenment forces its way into your field of view. Seriously though, if you want learn a different way of building applications where loads of these things pop up, look into Actor Model. Even if you are not going to use Actor Model, reading up about it will teach you a lot about "realtime" applications... My main epiphany was that realtime applications do not exist. Rather they cannot exist. Because our reality is not realtime (or our perception of it is always delayed because our brains/awareness is slow) - we can never truly capture the present moment. Data we see is always in the past. Reality is eventually consistent (from our perspective). All things in our reality runs concurrently, stateless and in their own little time bubble. You can also read up on the concepts behind Event Sourcing. Anyway, it is a huge rabbit hole, but reading up on the ideas behind Actor Model might just blow your mind. I'm convinced it will make a comeback because it is the natural way our reality works and Actor Model mimics it the closest, and it will also work better for multi-core cpu's. For non-tech reading, try The Power of Now by Eckhart Tolle. Then try to reflect on (relational databases) + (actor model) + (event sourcing) + (power of now) + (brain in a vat philosophy) + (try to to figure out at what speed the universe "renders", how fast our brains can think, how much delay there is, what limits do our body's sensors limits impose) + (try to figure out how closely Object Oriented programming can mimic reality (hint: all programming are tiny simulations that try to mimic some tiny part of our reality)) + (read up on the history of time keeping and the huge rabbit hole of calendars, and how most programming languages attempts to handle time (badly for the most part, it's a legit rabbit hole on it's own)). After a year or two of hard reading, reflection and teachings (and wrecking your head), you may have good insight into the nature of programming and the fallacies of trying to build realtime systems, or trying to mimic our reality in a computer in general. It is insanely complex, we as programmers merely touch a grain of sand of the beach of complexity. We have no idea how simple our most convoluted systems are in the face of nature, yet we convince ourselves that we KNOW how things work and that we are in control at all times. Good luck!

yosamino 6y ago

Is what you are describing not what they call "event-sourcing" ? Just keep a log of all the events that have already happened (crucial - I have seen "event logs" that were more like "request logs") and to find out the current state of the system - if the notification or report has has been sent - by examining this log. The lists you are describing kind of sound like an event log of sorts. It seems to me that this distinction between primary and secondary approach is not really neccessary.

hvidgaard 6y ago
The more structured way to do this is a CQRS framework with Event Sourcing as the underlying data store model. With that you create specialized readmodels that react to events and you can query them for the exact thing you need to know. Once you do something based on that query it will result in new events that update the readmodel. It's a powerful way of designing a system.

manigandham 6y ago

This is basically like going from imperative to declarative. Describe the final end/desired state and let the machine figure out how to get there, progressing through the work until it's caught up even across restarts or other boundaries.

snidane 6y ago
The outcome is the same. Except that it is not declarative, because 'you' are the machine who codes up that logic.

lloeki 6y ago

I cannot upvote this enough. One of the experiences I had was applying it to provisioning with idempotent shell scripts, combined with a trivial tool like https://github.com/lloeki/apply https://github.com/lloeki/apply Creating a VM in DO would add the basic SSH keys, then just run the thing against the new IP. Boom, VM ready. Want to make existing VMs config up to date? Apply. Boom. The whole thing ridiculously scaled up in a totally unexpected way, saving hundreds of hours of headaches and mistakes. Why not ansible/puppet? Check out the rationale in the readme.

nanis 6y ago
See also https://docs.chef.io/chef_solo/ https://docs.chef.io/chef_solo/

lathiat 6y ago

The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc. This was common when I was writing puppet. It was easy to have your entire puppet run take minutes - and want to run every few minutes - while consuming a LOT of cpu time including on a centralised server and not just the end host. There are of course ways and strategies to this and not every situational falls afoul of it. But it is something you very much need to be aware of. I otherwise whole heartedly endorse this.

gregmac 6y ago
> If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. Definitely! And on top of that, you probably want to apply some business logic to what you're doing. To expand on the example of "Report should run at midnight": If you run the check regularly, you can constrain the date range you're checking to just since the last time you checked, and store this date either in memory or persisted depending on your use case, technology, and how you want system failures/restarts handled. Similarly, if the system was down for several days, it might not be relevant to send all missing reports, but instead just the last one. On the other hand, if "daily reports" are required for auditing purposes, it probably does make sense to send all of them. But you have to be careful so on the very first run it doesn't go "Hey, there are 18,726 missing reports since Jan 1, 1970!" It's also important to write the report itself to be idempotent. You can't write it so it reports to "current" time with the assumption it runs at exactly midnight. Instead you have to actually constrain the dates to exactly what you want. Even aside from the idempotent invocation I'm talking about, as your system gets big you'll get thousands of reports that all need to run at "midnight" and that just isn't practical: maybe they run all sequentially, and some will not actually execute until several minutes after midnight.
- 91aintprime 6y ago
  suppose you serve clients who want their reports to be generated at 02:30 local time. you could try to run a cron container for every time zone :bleh:, but with daylight savings, the 2:30 event can fire twice, or not at all! instead you simply see if there's a report for yesterday, and if there isn't make one. the edge cases are still there, see replies

mandeepj 6y ago

> "Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes. Ha! You are biased toward the word "Idempotent way". Above - you have totally different steps under "Idempotent way". So, your process is different for each of them. There's no tech differentiator and it should be - as that's how you implement "Idempotent way"

tareqak 6y ago

The idempotent way is definitely better, but I argue/add that it would be better to have the incoming business requirement phrased appropriately such that the idempotent approach is the first one that comes to mind. I am not sure how to perform that rewording/transformation in general though or even for the examples above.

xnx 6y ago

I've set up a bunch of Windows batch scripts similarly, but didn't know the name for it. It's beautiful because failures, crashes, bugs, restarts, etc. barely matter. The next time the system is able to do what it was supposed to do, it does.