8 ms·

Around 2010 I was working at Urban Airship (now just Airship) offering an API for mobile app developers to send push notifications. We put a ton of effort into

Around 2010 I was working at Urban Airship (now just Airship) offering an API for mobile app developers to send push notifications. We put a ton of effort into making sure we didn’t drop pushes and our API was HA. At one point very early on I was restoring pushes out of error emails in mutt (not great, I know, I know). The reason? Not some unrealistic view of ourselves as Google or Amazon, but because we knew we served pushes for a popular pill reminder app (among other reasons). So just because you don’t have Google or Amazons scale, the data you store and process may be as precious to your users as Amazon’s shopping carts are to them. Over 10 years later there is a lot more medication in my life, and I’m really proud of the sometimes-ridiculous lengths we went to to get the pushes delivered.

lbriner 5y ago
That isn't really the counter-argument to the OP. Their examples were all about large complexity for the gain in throughput. Clearly, having HA and reliable transport is reasonable for your use-case but that isn't the same as microservices, hadoop or Kafka.
- schmichael 5y ago
  Definitely a bit off topic from me, apologies! I think I was mainly writing in agreement with the article’s closing: > What we’re all imploring you to do is to think! And to actually understand the problem you are trying to solve.
- hn_throwaway_99 5y ago
  I would actually go farther and say the comment you are responding to supports the article's thesis. They talk about remediating issues with email alerts. That is exactly the kind of approach you can take when you are nowhere near Google's scale, instead of building a complex system to ensure all of these remediations can happen automatically without any human intervention. Sounds like they took exactly the right approach of not over-engineering a solution prematurely.
  - schmichael 5y ago
    I intentionally wrote the story without referring to the article to let people think for themselves about the meaning, and it makes me very happy that this was what you thought! Ironically we kind of relied on e-mail alerts for too long and would kinda sorta break Gmail. While it was probably just the web/IMAP interface or browser/client itself struggling, trying to bulk delete tens of thousands of emails while hundreds more per second were coming in often had incoherent results.
Beltalowda 5y ago
The main point of this article wasn't that these things are never needed or don't solve a real problem, but rather that a lot of people use them "because hype" without a good understanding of their problems, the strengths and weaknesses of these tools, etc. UNPHAT is a good approach. Any webshop essentially has Amazon's "a failed add to cart will lose us money"-problem and there are a lot of webshops out there, so it's even a common problem! Cassandra may be a good choice even for a smaller webshop (I'm not familiar with it myself, and may be a poor trade-off with respect of ROI, but that's another thing), but it's definitely not a good fit for a lot of other things like batch jobs. At my current $dayjob we run a low-volume high-profit B2B business; we have a few hundred customers, and it'll never be more than a few thousand – several ten thousand at the most if we end up wildly successful. It's a whole forest of microservices and will scale to the moon, but it's also pretty complex and difficult to work with. All that for a fairly small customer base using a fairly basic web UI they rarely log in to. We do actually have high-availability requirements in the core business, but that's separate from the management UI. It's a good example of not having understood the actual problems and just using something "because I read about it on Hacker News".
- bitexploder 5y ago
  Cargo cults everywhere.
CGamesPlay 5y ago
Man, not to take away from your point or efforts, but imagine missing your pill reminder because you’re on a road trip and you didn’t have cell coverage in Montana.
- schmichael 5y ago
  Ha, right? IIRC this was well before local scheduled pushes existed, so the Remote Montana problem was real! Mobile has come a long way for sure!
  - HWR_14 5y ago
    I've been able to set a repeating alarm forever
    - schmichael 5y ago
      It's entirely possible the app author made a poor choice. We have no control over their code. All we could do is try to deliver our meager little component as competently as possible.
  - brimble 5y ago
    My recollection is also that local notifications didn't exist yet in 2010 on iOS, or had just been added. I tried to confirm it but it's actually pretty hard to find that information, it seems. Could be wrong. But, I was heavy in mobile and push notifications just after 2010, so I'm fairly sure that's right. I distinctly remember that as a feature that was added, at some point.
commandlinefan 5y ago
> Urban Airship (now just Airship) Ironically, Urban Airship (now just Airship) itself is great example of something that people use even though they don't need it - due to hype. You have a mobile app and it needs to send notifications, but you don't have any in-house expertise on mobile notifications. Management says, "ok, let's outsource this to somebody who has some expertise!" They look around and find Urban Airship - the leader in mobile app notification handling! So you "integrate" with Urban Airship, per management dictates (the same reason you used Hadoop, Cassandra and Kafka). They give you this big complicated presentation but you discover that, in order to use their "service" you have to: 1) instrument your application to send their API a notification every time something happens in your application. 2) set up a workflow in their system that tells them when to tell you that something happened that a notification needs to be sent for 3) instrument your application to read a dataset from them telling you when to send a mobile notification 4) actually send the mobile notification You realize that they're charging your company a ton of money to do... absolutely nothing. You're still doing everything, responsible for everything, and they're nothing but completely empty overhead. They could have been replaced by a Postgres instance running on a single underpowered EC2 node (and it would be more reliable). But management sees "leader in mobile app notifications" the same as they saw "leader in distributed computing loads" and insisted that you use this thing that exists because it exists.
- dont__panic 5y ago
  Slightly unrelated, but can anybody explain why private RSS feeds are not a more popular choice for "client push" when push lag of a couple of hours isn't a big deal? The "pill reminder" app made me think about how easy it would be to host a private RSS feed for each client that the backend updates on a schedule, and clients just... check the RSS feed for an update. Maybe I'm missing something here, but it feels like you don't need some fancy realtime queue service for most push notifications, so why not use something that offloads the checks to the client?
  - deleted 5y ago
    [deleted]
  - 5y ago
HWR_14 5y ago
I'm confused why a pill reminder app is driven by the internet at all. Wouldn't it make more sense to have the pill reminder app send a local notification at the appropriate time?
- ryandrake 5y ago
  Excellent comment! A lot of projects already failed the "too much complexity" problem by just using the Internet when the Internet is unnecessary. I remember a project I worked on in the past where the requirement was to have the app display a notification to the user at a certain time, like an alarm clock. Pretty simple, and could be implemented with a line or two of code as a locally-generated notification. But no, the lead declared that we needed a push notification from a server in order to do this properly. Even after the difference between a local notification and a push notification was explained to him (he didn't know that local notifications existed), he insisted that notifications = push notifications, and we must have a server. AND that server needed to run a complex web service. AND it needed a way for devices to log in with credentials, so it needed a database. AND it needed to be fault tolerant since this was a critical user journey in the app. AND the clients had to check in with the server periodically, since their location might have changed to a different time zone, therefore the push notification time had to change. AND this AND that AND so on. All for what was essentially an alarm clock--something you could implement in the client code running on the device in 5 minutes.
  - A4ET8a8uTh0 5y ago
    Can we assume that the unstated purpose of push vs local was the ability to query various notifications without the need to separately upload queries somewhere else?
- schmichael 5y ago
  > app send a local notification at the appropriate time IIRC this didn't exist at the time but is indeed the ideal solution. (some more discussion of this downthread) If pill reminders were sync'd between devices (or between mobile and web verisons of the product), then push notifications would still be necessary to inform one side or the other when to resync data if nothing else. Still, 2010 was a wildly different time in mobile. We were all still figuring a lot out (clearly)!

lbriner 5y ago

That isn't really the counter-argument to the OP. Their examples were all about large complexity for the gain in throughput. Clearly, having HA and reliable transport is reasonable for your use-case but that isn't the same as microservices, hadoop or Kafka.

schmichael 5y ago
Definitely a bit off topic from me, apologies! I think I was mainly writing in agreement with the article’s closing: > What we’re all imploring you to do is to think! And to actually understand the problem you are trying to solve.
hn_throwaway_99 5y ago
I would actually go farther and say the comment you are responding to supports the article's thesis. They talk about remediating issues with email alerts. That is exactly the kind of approach you can take when you are nowhere near Google's scale, instead of building a complex system to ensure all of these remediations can happen automatically without any human intervention. Sounds like they took exactly the right approach of not over-engineering a solution prematurely.
- schmichael 5y ago
  I intentionally wrote the story without referring to the article to let people think for themselves about the meaning, and it makes me very happy that this was what you thought! Ironically we kind of relied on e-mail alerts for too long and would kinda sorta break Gmail. While it was probably just the web/IMAP interface or browser/client itself struggling, trying to bulk delete tens of thousands of emails while hundreds more per second were coming in often had incoherent results.

Beltalowda 5y ago

The main point of this article wasn't that these things are never needed or don't solve a real problem, but rather that a lot of people use them "because hype" without a good understanding of their problems, the strengths and weaknesses of these tools, etc. UNPHAT is a good approach. Any webshop essentially has Amazon's "a failed add to cart will lose us money"-problem and there are a lot of webshops out there, so it's even a common problem! Cassandra may be a good choice even for a smaller webshop (I'm not familiar with it myself, and may be a poor trade-off with respect of ROI, but that's another thing), but it's definitely not a good fit for a lot of other things like batch jobs. At my current $dayjob we run a low-volume high-profit B2B business; we have a few hundred customers, and it'll never be more than a few thousand – several ten thousand at the most if we end up wildly successful. It's a whole forest of microservices and will scale to the moon, but it's also pretty complex and difficult to work with. All that for a fairly small customer base using a fairly basic web UI they rarely log in to. We do actually have high-availability requirements in the core business, but that's separate from the management UI. It's a good example of not having understood the actual problems and just using something "because I read about it on Hacker News".

bitexploder 5y ago
Cargo cults everywhere.

CGamesPlay 5y ago

Man, not to take away from your point or efforts, but imagine missing your pill reminder because you’re on a road trip and you didn’t have cell coverage in Montana.

schmichael 5y ago
Ha, right? IIRC this was well before local scheduled pushes existed, so the Remote Montana problem was real! Mobile has come a long way for sure!
- HWR_14 5y ago
  I've been able to set a repeating alarm forever
  - schmichael 5y ago
    It's entirely possible the app author made a poor choice. We have no control over their code. All we could do is try to deliver our meager little component as competently as possible.
- brimble 5y ago
  My recollection is also that local notifications didn't exist yet in 2010 on iOS, or had just been added. I tried to confirm it but it's actually pretty hard to find that information, it seems. Could be wrong. But, I was heavy in mobile and push notifications just after 2010, so I'm fairly sure that's right. I distinctly remember that as a feature that was added, at some point.

commandlinefan 5y ago

> Urban Airship (now just Airship) Ironically, Urban Airship (now just Airship) itself is great example of something that people use even though they don't need it - due to hype. You have a mobile app and it needs to send notifications, but you don't have any in-house expertise on mobile notifications. Management says, "ok, let's outsource this to somebody who has some expertise!" They look around and find Urban Airship - the leader in mobile app notification handling! So you "integrate" with Urban Airship, per management dictates (the same reason you used Hadoop, Cassandra and Kafka). They give you this big complicated presentation but you discover that, in order to use their "service" you have to: 1) instrument your application to send their API a notification every time something happens in your application. 2) set up a workflow in their system that tells them when to tell you that something happened that a notification needs to be sent for 3) instrument your application to read a dataset from them telling you when to send a mobile notification 4) actually send the mobile notification You realize that they're charging your company a ton of money to do... absolutely nothing. You're still doing everything, responsible for everything, and they're nothing but completely empty overhead. They could have been replaced by a Postgres instance running on a single underpowered EC2 node (and it would be more reliable). But management sees "leader in mobile app notifications" the same as they saw "leader in distributed computing loads" and insisted that you use this thing that exists because it exists.

dont__panic 5y ago
Slightly unrelated, but can anybody explain why private RSS feeds are not a more popular choice for "client push" when push lag of a couple of hours isn't a big deal? The "pill reminder" app made me think about how easy it would be to host a private RSS feed for each client that the backend updates on a schedule, and clients just... check the RSS feed for an update. Maybe I'm missing something here, but it feels like you don't need some fancy realtime queue service for most push notifications, so why not use something that offloads the checks to the client?
- deleted 5y ago
  [deleted]
- 5y ago

HWR_14 5y ago

I'm confused why a pill reminder app is driven by the internet at all. Wouldn't it make more sense to have the pill reminder app send a local notification at the appropriate time?

ryandrake 5y ago
Excellent comment! A lot of projects already failed the "too much complexity" problem by just using the Internet when the Internet is unnecessary. I remember a project I worked on in the past where the requirement was to have the app display a notification to the user at a certain time, like an alarm clock. Pretty simple, and could be implemented with a line or two of code as a locally-generated notification. But no, the lead declared that we needed a push notification from a server in order to do this properly. Even after the difference between a local notification and a push notification was explained to him (he didn't know that local notifications existed), he insisted that notifications = push notifications, and we must have a server. AND that server needed to run a complex web service. AND it needed a way for devices to log in with credentials, so it needed a database. AND it needed to be fault tolerant since this was a critical user journey in the app. AND the clients had to check in with the server periodically, since their location might have changed to a different time zone, therefore the push notification time had to change. AND this AND that AND so on. All for what was essentially an alarm clock--something you could implement in the client code running on the device in 5 minutes.
- A4ET8a8uTh0 5y ago
  Can we assume that the unstated purpose of push vs local was the ability to query various notifications without the need to separately upload queries somewhere else?
schmichael 5y ago
> app send a local notification at the appropriate time IIRC this didn't exist at the time but is indeed the ideal solution. (some more discussion of this downthread) If pill reminders were sync'd between devices (or between mobile and web verisons of the product), then push notifications would still be necessary to inform one side or the other when to resync data if nothing else. Still, 2010 was a wildly different time in mobile. We were all still figuring a lot out (clearly)!