9 ms·

A Developer's Worst Nightmare (The TinyGrab Story)

48 points by nam3d 16y ago

philjackson 16y ago
"Like all premature babies, TinyGrab 2.0 just wasn’t ready for mainstream usage" What the fuck?
- markkanof 16y ago
  Yes, this seemed like a particularly offensive metaphor. I suppose it works at a logical level, but to compare a human life to a piece of software seems kind of extreme.
gm 16y ago
If anything this is a cautionary tale about how to screw up your startup. I have no idea what TinyGrab's architecture looks like, but I can say with full confidence that backups were an afterthought, and they find themselves in deep shit because they did not have a proper backup in place. It is inexcusable to have the current situation and still claim they have good backups. "The issues lays with the server being compromised and being forced to shut down. The old API and apps relied on a fixed IP address. It would have taken us months to get TinyGrab v1 back up to scratch" Ok, computer scientists reading this: To which of you does this make sense? It's hard to keep counts of all the fails in here: - If your problem is a corrupted server, replace it and assign the same IP address. - Why would anyone hard-code IP addresses for an API to work? Were you born yesterday? - It should take at the very most hours to recover from a truly catastrophic failure. "Months" you say? You do not know what you're doing, I say. - They have a bunch of backup and not one works, specially when there is no data loss? Smells to me like no source control; no off site backup, and absolutely no testing of the backup. If you have lived a single day in system admin work, you know that it is not enough to do a backup. You need to test the damned restore. Maybe this could be a story about when _not_ to self-finance your startup. Get outside funding and hire people that know more than you do about the technology involved. Sorry for being an asshole in my comments, but this situation is full of fail at every step.
- consultutah 16y ago
  No need to get outside funding for any of this. It can all be done on the cheap. I agree though, how can it take months to reset up a server? Late last year I had both of our supposedly redundant webservers fail at the same time - never happens, right? ;) We were able to get two entirely separate boxes setup with the same static addresses in under an hour due to maintaining all possible configuration in svn. Worse, this was on windows servers where we couldn't use something like puppet or one of the other 20 solutions to script your configuration.
- datasink 16y ago
  I believe there is a complete, one sentence explanation that would ring true for most developers, but is not being offered for one reason or another. My best guess: It would have been easier to get 1.0 back up and working, but this was viewed as an opportunity to finally push out the 2.0 and convert over the vast majority of the existing user-base in short order. The release was not tested well enough, and so this was a bad plan in hindsight.
  - gm 16y ago
    Well, the worst time to change technologies or perform version updates is when there is a fire under your ass to get the system working, but that is a minor screw up compared to all else going on. This whole thing stinks of lack of experience. No one should repeat the steps these guys are taking.
- markkanof 16y ago
  I guess a lot of this could be attributed to the fact that the TinyGrab developers are volunteers (at least according to the article). Like many open source projects they probably focused on the glamourous work of actually coding the app and not the boring stuff like backups. I'm not saying this is acceptable, I'm just suggesting an explanation. I could sort of see not having a completely bulletproof backup strategy in place, but with all the options available, I can't imagine not working with source control.
- samengland 16y ago
deleted 16y ago
[deleted]
cheez 16y ago
Why wouldn't they just put the old code base and fix the security hole? Why go to a new code base? Now you have two problems!
rll 16y ago
I must be missing something. How is it possible that they don't have umpteen copies of the sources checked out of whatever source control system they are using? Even if the central repository was destroyed, every developer should have a more or less recent version checked out somewhere.
- zacharypinter 16y ago
  Looks like a PHP site. I'm guessing they weren't using version control and were instead just editing the files over FTP.
  - enko 16y ago
    If that turns out to be the case .. I wouldn't go so far as to say they deserved it, but my sympathy would definitely be pretty limited.
    - nwp 16y ago
      I agree. There is absolutely no excuse for not having some sort of source control and deployment system in place at an organization with a commercially deployed solution. Why would anyone regard such a system as optional? It is irresponsible.
  - Androsynth 16y ago
    so php == ignorance? are we really going to make that assumption? thats just foolish.
    - deleted 16y ago
      [deleted]
    - zacharypinter 16y ago
      Not at all. Just trying to figure out how they might have a dev environment setup where there isn't a copy on local developer machines. Not all languages make that easy. PHP does.
michaelcampbell 16y ago
"TinyGrab was attacked and the attacker was able to gain access to one of their servers. Although the details are scarce, the attacker corrupted the codebase for TinyGrab 1. The TinyGrab team was forced to migrate their entire userbase to the newest version, a version which wasn’t completely ready for yet." I didn't find any mention of it in TFA, but do people just not do backups anymore? Or rolling, dated backups? My wife owns a small retail/web store, and I backup her Quickbooks, the database, and the source code nightly, and keep at least a few weeks of those backups on the drive where it's stored, 2 different backup drives, and and offsite place. Is this overkill? Perhaps, but it would have saved us from that.
- cocoacoding 16y ago
  I was a bit confused when I first read about it. I'm sorry if I worded it a bit weird in the article. My best guess is they broke something with how the TinyGrab client interacts with their internal API's. They did have backups, but it looks like it was deeper than that. That is my best guess.
- maukdaddy 16y ago
  Apparently part of Web 2.0 is deploy and forget? How could no one have at least a snapshot of the code and database?
randrews 16y ago
I used to use TinyGrab all the time, it was a pretty indispensable part of my workflow, which is why, after nearly a week of it being dead, I got fed up and wrote my own: http://dl.dropbox.com/u/651972/ScreenDuck-0.1a.zip http://dl.dropbox.com/u/651972/ScreenDuck-0.1a.zip It's obviously very rough right now, but it actually works. Later on I'll write a website for it and put it up on screenduck.com.
- pauldino 16y ago
  This is the first I've heard of TinyGrab, but for Mac users looking for something else Cloud seems to do the same thing (that is, it can automatically upload screenshots and put the URL in the clipboard for you). http://getcloudapp.com/ http://getcloudapp.com/
- paulbaumgart 16y ago
  A buddy of mine made this, works pretty well: http://grabbox.devsoft.no/ http://grabbox.devsoft.no/
- chrisleydon 16y ago
  A good alternative you'll find is FileShuttle. - http://getfileshuttle.com http://getfileshuttle.com
acangiano 16y ago
They did an extremely poor PR job. My account is no longer working. I would expect an email of apologies that explains the problem and buys them time. Instead, I got an account that mysteriously doesn't work anymore, and a password reset link that claims my account doesn't exist. I had to learn about what happened from a random HN post.
- chrisleydon 16y ago
  If you can shoot me an email, chris@tinygrab.com, I'll deal with your account and help get you set back up again.
  - acangiano 16y ago
    Thanks Chris. Just sent an email.
deleted 16y ago
[deleted]
chrisleydon 16y ago
We have more backups than you can shake a stick at. The issues lays with the server being compromised and being forced to shut down. The old API and apps relied on a fixed IP address. It would have taken us months to get TinyGrab v1 back up to scratch, or a few weeks to perfect 2.0. No data was lost, what would you choose? Chris Leydon TinyGrab Founder and Project Manager chris@tinygrab.com
- bcl 16y ago
  Thanks for the explanation, but I still don't understand the problem. You re-image the server, fix whatever holes they exploited and re-install from a backup. Applications should be designed for failure. Assume this will happen every week and plan for it. If not someone cracking your system then a hardware failure, or your ISP going bankrupt or any number of other disasters.
- troels 16y ago
  But .. why can't you `grep|sed` your source code for the ip address? I'm confused.
  - wvenable 16y ago
    I suspect it's hard-coded into all the clients -- thus they'd have to get everyone to download a new version of the client (which they are now doing, but straight to version 2.0). Although it should not have been hard to get a new server up at the same IP.
- Jabbles 16y ago
  Could you explain the decision process behind using fixed IP addresses? It's not my area of expertise, but surely using a URL would provide you just as much control with vastly more flexibility.
  - chrisleydon 16y ago
    It was an early and foolish decision we made back in the dearly days of TinyGrab. The original system wasn't built, or even created, to handle more than a handful of people using the service. Thanks to some great PR though we scaled to ver 400,000+ users almost over night. The code base was horrible, buggy and 2.0 was being written to fix all of that. It was supposed to be a smooth transition over, but unfortunately someone got there first and forced us to launch the new system.
aneth 16y ago
This situation makes no sense at all based on the explanations here by chrisleydon. You have backups of the code - can you explain why it isn't a pretty straightforward process to set up a new server? The fact that you have a fixed IP address has no importance as far as I can tell, and that's the only explanation offered. > It would have taken us months to get TinyGrab v1 back up to scratch Really?

48 points by nam3d 16y ago

philjackson 16y ago

"Like all premature babies, TinyGrab 2.0 just wasn’t ready for mainstream usage" What the fuck?

markkanof 16y ago
Yes, this seemed like a particularly offensive metaphor. I suppose it works at a logical level, but to compare a human life to a piece of software seems kind of extreme.

gm 16y ago

If anything this is a cautionary tale about how to screw up your startup. I have no idea what TinyGrab's architecture looks like, but I can say with full confidence that backups were an afterthought, and they find themselves in deep shit because they did not have a proper backup in place. It is inexcusable to have the current situation and still claim they have good backups. "The issues lays with the server being compromised and being forced to shut down. The old API and apps relied on a fixed IP address. It would have taken us months to get TinyGrab v1 back up to scratch" Ok, computer scientists reading this: To which of you does this make sense? It's hard to keep counts of all the fails in here: - If your problem is a corrupted server, replace it and assign the same IP address. - Why would anyone hard-code IP addresses for an API to work? Were you born yesterday? - It should take at the very most hours to recover from a truly catastrophic failure. "Months" you say? You do not know what you're doing, I say. - They have a bunch of backup and not one works, specially when there is no data loss? Smells to me like no source control; no off site backup, and absolutely no testing of the backup. If you have lived a single day in system admin work, you know that it is not enough to do a backup. You need to test the damned restore. Maybe this could be a story about when _not_ to self-finance your startup. Get outside funding and hire people that know more than you do about the technology involved. Sorry for being an asshole in my comments, but this situation is full of fail at every step.

consultutah 16y ago
No need to get outside funding for any of this. It can all be done on the cheap. I agree though, how can it take months to reset up a server? Late last year I had both of our supposedly redundant webservers fail at the same time - never happens, right? ;) We were able to get two entirely separate boxes setup with the same static addresses in under an hour due to maintaining all possible configuration in svn. Worse, this was on windows servers where we couldn't use something like puppet or one of the other 20 solutions to script your configuration.
datasink 16y ago
I believe there is a complete, one sentence explanation that would ring true for most developers, but is not being offered for one reason or another. My best guess: It would have been easier to get 1.0 back up and working, but this was viewed as an opportunity to finally push out the 2.0 and convert over the vast majority of the existing user-base in short order. The release was not tested well enough, and so this was a bad plan in hindsight.
- gm 16y ago
  Well, the worst time to change technologies or perform version updates is when there is a fire under your ass to get the system working, but that is a minor screw up compared to all else going on. This whole thing stinks of lack of experience. No one should repeat the steps these guys are taking.
markkanof 16y ago
I guess a lot of this could be attributed to the fact that the TinyGrab developers are volunteers (at least according to the article). Like many open source projects they probably focused on the glamourous work of actually coding the app and not the boring stuff like backups. I'm not saying this is acceptable, I'm just suggesting an explanation. I could sort of see not having a completely bulletproof backup strategy in place, but with all the options available, I can't imagine not working with source control.
samengland 16y ago

deleted 16y ago

[deleted]

cheez 16y ago

Why wouldn't they just put the old code base and fix the security hole? Why go to a new code base? Now you have two problems!

rll 16y ago

I must be missing something. How is it possible that they don't have umpteen copies of the sources checked out of whatever source control system they are using? Even if the central repository was destroyed, every developer should have a more or less recent version checked out somewhere.

zacharypinter 16y ago
Looks like a PHP site. I'm guessing they weren't using version control and were instead just editing the files over FTP.
- enko 16y ago
  If that turns out to be the case .. I wouldn't go so far as to say they deserved it, but my sympathy would definitely be pretty limited.
  - nwp 16y ago
    I agree. There is absolutely no excuse for not having some sort of source control and deployment system in place at an organization with a commercially deployed solution. Why would anyone regard such a system as optional? It is irresponsible.
- Androsynth 16y ago
  so php == ignorance? are we really going to make that assumption? thats just foolish.
  - deleted 16y ago
    [deleted]
  - zacharypinter 16y ago
    Not at all. Just trying to figure out how they might have a dev environment setup where there isn't a copy on local developer machines. Not all languages make that easy. PHP does.

michaelcampbell 16y ago

"TinyGrab was attacked and the attacker was able to gain access to one of their servers. Although the details are scarce, the attacker corrupted the codebase for TinyGrab 1. The TinyGrab team was forced to migrate their entire userbase to the newest version, a version which wasn’t completely ready for yet." I didn't find any mention of it in TFA, but do people just not do backups anymore? Or rolling, dated backups? My wife owns a small retail/web store, and I backup her Quickbooks, the database, and the source code nightly, and keep at least a few weeks of those backups on the drive where it's stored, 2 different backup drives, and and offsite place. Is this overkill? Perhaps, but it would have saved us from that.

cocoacoding 16y ago
I was a bit confused when I first read about it. I'm sorry if I worded it a bit weird in the article. My best guess is they broke something with how the TinyGrab client interacts with their internal API's. They did have backups, but it looks like it was deeper than that. That is my best guess.
maukdaddy 16y ago
Apparently part of Web 2.0 is deploy and forget? How could no one have at least a snapshot of the code and database?

randrews 16y ago

I used to use TinyGrab all the time, it was a pretty indispensable part of my workflow, which is why, after nearly a week of it being dead, I got fed up and wrote my own: http://dl.dropbox.com/u/651972/ScreenDuck-0.1a.zip http://dl.dropbox.com/u/651972/ScreenDuck-0.1a.zip It's obviously very rough right now, but it actually works. Later on I'll write a website for it and put it up on screenduck.com.

pauldino 16y ago
This is the first I've heard of TinyGrab, but for Mac users looking for something else Cloud seems to do the same thing (that is, it can automatically upload screenshots and put the URL in the clipboard for you). http://getcloudapp.com/ http://getcloudapp.com/
paulbaumgart 16y ago
A buddy of mine made this, works pretty well: http://grabbox.devsoft.no/ http://grabbox.devsoft.no/
chrisleydon 16y ago
A good alternative you'll find is FileShuttle. - http://getfileshuttle.com http://getfileshuttle.com

acangiano 16y ago

They did an extremely poor PR job. My account is no longer working. I would expect an email of apologies that explains the problem and buys them time. Instead, I got an account that mysteriously doesn't work anymore, and a password reset link that claims my account doesn't exist. I had to learn about what happened from a random HN post.

chrisleydon 16y ago
If you can shoot me an email, chris@tinygrab.com, I'll deal with your account and help get you set back up again.
- acangiano 16y ago
  Thanks Chris. Just sent an email.

deleted 16y ago

[deleted]

chrisleydon 16y ago

We have more backups than you can shake a stick at. The issues lays with the server being compromised and being forced to shut down. The old API and apps relied on a fixed IP address. It would have taken us months to get TinyGrab v1 back up to scratch, or a few weeks to perfect 2.0. No data was lost, what would you choose? Chris Leydon TinyGrab Founder and Project Manager chris@tinygrab.com

bcl 16y ago
Thanks for the explanation, but I still don't understand the problem. You re-image the server, fix whatever holes they exploited and re-install from a backup. Applications should be designed for failure. Assume this will happen every week and plan for it. If not someone cracking your system then a hardware failure, or your ISP going bankrupt or any number of other disasters.
troels 16y ago
But .. why can't you `grep|sed` your source code for the ip address? I'm confused.
- wvenable 16y ago
  I suspect it's hard-coded into all the clients -- thus they'd have to get everyone to download a new version of the client (which they are now doing, but straight to version 2.0). Although it should not have been hard to get a new server up at the same IP.
Jabbles 16y ago
Could you explain the decision process behind using fixed IP addresses? It's not my area of expertise, but surely using a URL would provide you just as much control with vastly more flexibility.
- chrisleydon 16y ago
  It was an early and foolish decision we made back in the dearly days of TinyGrab. The original system wasn't built, or even created, to handle more than a handful of people using the service. Thanks to some great PR though we scaled to ver 400,000+ users almost over night. The code base was horrible, buggy and 2.0 was being written to fix all of that. It was supposed to be a smooth transition over, but unfortunately someone got there first and forced us to launch the new system.

aneth 16y ago

This situation makes no sense at all based on the explanations here by chrisleydon. You have backups of the code - can you explain why it isn't a pretty straightforward process to set up a new server? The fact that you have a fixed IP address has no importance as far as I can tell, and that's the only explanation offered. > It would have taken us months to get TinyGrab v1 back up to scratch Really?