7 ms·

Extended Outage at Instapaper

58 points by lenova 10y ago

msh 10y ago
From the article : After spending multiple hours on the phone with our cloud service provider, it appears we hit a system limit for our hosted database that’s preventing new articles from being saved. At this time, our only option is to export all data from our old database and import it into a new one. Seems pretty bad that it takes hours to discover that. Anyone knows what cloud it is?
- nanaujal 10y ago
  instapaper.com seems to be hosted @ AWS
  - bhouston 10y ago
    Which DB at AWS: https://aws.amazon.com/products/databases/ https://aws.amazon.com/products/databases/ Also shouldn't AWS warn them that this was about to happen? Edit 1: DynamoDB says: "There is no practical limit on a table's size. Tables are unconstrained in terms of the number of items or the number of bytes." The SQL databases say this: "The Amazon Aurora engine will automatically grow the size of your database volume as your database storage needs grow, up to a maximum of 64 TB or a maximum you define. The MySQL, MariaDB, Oracle, and PostgreSQL engines allow you to provision up to 6 TB of storage on-the-fly with zero downtime." So they either hit 64TB or 6TB of data I assume? EDIT 2: According to this they are MySQL based: https://medium.com/making-instapaper/rebuilding-instapaper-search-7fae1340699e#.svigv748s https://medium.com/making-instapaper/rebuilding-instapaper-s...
    - gagabity 10y ago
      Think you need to set up CloudWatch to get alerts.
      - dhd415 10y ago
        And CloudWatch is free for RDS at 5-minute collection intervals which is more than adequate for alerting on things such as "Am I about to hit the space limit on my database?"
    - dexterdog 10y ago
      They could be on aurora though since many people just consider it to be mysql. It's a pretty painless migration from mysql as long as you are 5.6 or below. You can just replicate over to it live.
  - deleted 10y ago
    [deleted]
gagabity 10y ago
I'm guessing they mean space limit on a single DB, you would think a service like Instapaper would have some monitoring in place by now.
- j_s 10y ago
  Interesting to compare both the different approaches taken in sharing details by GitLab vs. Instapaper, and the long-term reaction to it here. The backlash against GitLab was obvious; I'll be watching the next Instapaper disucssion with interest. https://news.ycombinator.com/item?id=13608267 https://news.ycombinator.com/item?id=13608267 etc.
ycombinete 10y ago
I'd be interested to know what services like Instapaper are used by HN members. Personally I've always wanted to like Instapaper more. I really do. I've given it a few extended runs, on it's own, and parallel to Pocket; but I always end up moving back to Pocket.
- joekrill 10y ago
  I recently switched from Pocket to a self-hosted instance of Wallabag (https://wallabag.org https://wallabag.org) and I've been very happy with it. One thing I've noticed with these types of services is that they quickly start becoming intrusive and annoying. Ads are the obvious annoyance. But as an additional example, the Pocket extension injects code into certain sites (HackerNews being one) so that I can "more easily" "pocket" posted articles directly from the listing page itself. And in fairness it can be disabled, but it's enabled by default. And with things like this, I just find that simpler is better. I don't need all these extra features being pushed and modifying my browsing experience "by default".
  - dangoldin 10y ago
    I had some issues getting the Wallabag app working on a smartphone. Have you used that successfully?
    - joekrill 10y ago
      I've had no problems at all. I use it primarily on my Android tablet (Nvidia Shield K1), and occasionally on my phone (also Android, Moto X Pure 2015). I haven't noticed any problems. I use it for both reading and adding using the built-in sharing functionality in Android. For setup I simply set my URL, username, and password in the settings and used the "Autofill" option to handle populating the rest of the settings, and it works great. I'm running the server itself in a Docker container, with connections proxied through Nginx (also in a Docker container).
    - nicosomb 10y ago
      Hey dangoldin, Don't hesitate to ping us on gitter.im/wallabag/wallabag if you still have any problems.
dangoldin 10y ago
Very odd and surprising it's taking that long to recover. I got the following email from them this morning which seems to indicate they will only be fully normal a week from now. As we shared earlier today, Instapaper is experiencing an extended outage. After 31 hours of downtime, we were able to rebuild a database instance to get Instapaper back online! In the interest of coming back up as soon as possible, this instance only has the last six weeks of articles. For now, anything you’ve saved since December 20, 2016 is accessible. We’re working on getting the rest of your articles restored, but wanted to give you access to your most recently saved articles in the meantime. Again, there hasn’t been any data loss. The full database exports are taking much longer than anticipated. We’re aiming to get the full archives available by next Friday, February 17 at the latest. We appreciate your patience and understanding in the interim. If you've got any questions or thoughts, just reply to this email or let us know at @InstapaperHelp on Twitter. - Instapaper Team
macintux 10y ago
Post-mortem: https://medium.com/making-instapaper/instapaper-outage-cause-recovery-3c32a7e9cc5f#.g7jy6jtdg https://medium.com/making-instapaper/instapaper-outage-cause...

58 points by lenova 10y ago

msh 10y ago

From the article : After spending multiple hours on the phone with our cloud service provider, it appears we hit a system limit for our hosted database that’s preventing new articles from being saved. At this time, our only option is to export all data from our old database and import it into a new one. Seems pretty bad that it takes hours to discover that. Anyone knows what cloud it is?

nanaujal 10y ago
instapaper.com seems to be hosted @ AWS
- bhouston 10y ago
  Which DB at AWS: https://aws.amazon.com/products/databases/ https://aws.amazon.com/products/databases/ Also shouldn't AWS warn them that this was about to happen? Edit 1: DynamoDB says: "There is no practical limit on a table's size. Tables are unconstrained in terms of the number of items or the number of bytes." The SQL databases say this: "The Amazon Aurora engine will automatically grow the size of your database volume as your database storage needs grow, up to a maximum of 64 TB or a maximum you define. The MySQL, MariaDB, Oracle, and PostgreSQL engines allow you to provision up to 6 TB of storage on-the-fly with zero downtime." So they either hit 64TB or 6TB of data I assume? EDIT 2: According to this they are MySQL based: https://medium.com/making-instapaper/rebuilding-instapaper-search-7fae1340699e#.svigv748s https://medium.com/making-instapaper/rebuilding-instapaper-s...
  - gagabity 10y ago
    Think you need to set up CloudWatch to get alerts.
    - dhd415 10y ago
      And CloudWatch is free for RDS at 5-minute collection intervals which is more than adequate for alerting on things such as "Am I about to hit the space limit on my database?"
  - dexterdog 10y ago
    They could be on aurora though since many people just consider it to be mysql. It's a pretty painless migration from mysql as long as you are 5.6 or below. You can just replicate over to it live.
- deleted 10y ago
  [deleted]

gagabity 10y ago

I'm guessing they mean space limit on a single DB, you would think a service like Instapaper would have some monitoring in place by now.

j_s 10y ago
Interesting to compare both the different approaches taken in sharing details by GitLab vs. Instapaper, and the long-term reaction to it here. The backlash against GitLab was obvious; I'll be watching the next Instapaper disucssion with interest. https://news.ycombinator.com/item?id=13608267 https://news.ycombinator.com/item?id=13608267 etc.

ycombinete 10y ago

I'd be interested to know what services like Instapaper are used by HN members. Personally I've always wanted to like Instapaper more. I really do. I've given it a few extended runs, on it's own, and parallel to Pocket; but I always end up moving back to Pocket.

joekrill 10y ago
I recently switched from Pocket to a self-hosted instance of Wallabag (https://wallabag.org https://wallabag.org) and I've been very happy with it. One thing I've noticed with these types of services is that they quickly start becoming intrusive and annoying. Ads are the obvious annoyance. But as an additional example, the Pocket extension injects code into certain sites (HackerNews being one) so that I can "more easily" "pocket" posted articles directly from the listing page itself. And in fairness it can be disabled, but it's enabled by default. And with things like this, I just find that simpler is better. I don't need all these extra features being pushed and modifying my browsing experience "by default".
- dangoldin 10y ago
  I had some issues getting the Wallabag app working on a smartphone. Have you used that successfully?
  - joekrill 10y ago
    I've had no problems at all. I use it primarily on my Android tablet (Nvidia Shield K1), and occasionally on my phone (also Android, Moto X Pure 2015). I haven't noticed any problems. I use it for both reading and adding using the built-in sharing functionality in Android. For setup I simply set my URL, username, and password in the settings and used the "Autofill" option to handle populating the rest of the settings, and it works great. I'm running the server itself in a Docker container, with connections proxied through Nginx (also in a Docker container).
  - nicosomb 10y ago
    Hey dangoldin, Don't hesitate to ping us on gitter.im/wallabag/wallabag if you still have any problems.

dangoldin 10y ago

Very odd and surprising it's taking that long to recover. I got the following email from them this morning which seems to indicate they will only be fully normal a week from now. As we shared earlier today, Instapaper is experiencing an extended outage. After 31 hours of downtime, we were able to rebuild a database instance to get Instapaper back online! In the interest of coming back up as soon as possible, this instance only has the last six weeks of articles. For now, anything you’ve saved since December 20, 2016 is accessible. We’re working on getting the rest of your articles restored, but wanted to give you access to your most recently saved articles in the meantime. Again, there hasn’t been any data loss. The full database exports are taking much longer than anticipated. We’re aiming to get the full archives available by next Friday, February 17 at the latest. We appreciate your patience and understanding in the interim. If you've got any questions or thoughts, just reply to this email or let us know at @InstapaperHelp on Twitter. - Instapaper Team

macintux 10y ago

Post-mortem: https://medium.com/making-instapaper/instapaper-outage-cause-recovery-3c32a7e9cc5f#.g7jy6jtdg https://medium.com/making-instapaper/instapaper-outage-cause...