5 ms·

If anything this is a cautionary tale about how to screw up your startup. I have no idea what TinyGrab's architecture looks like, but I can say with full confi

by gm 16y ago

If anything this is a cautionary tale about how to screw up your startup. I have no idea what TinyGrab's architecture looks like, but I can say with full confidence that backups were an afterthought, and they find themselves in deep shit because they did not have a proper backup in place. It is inexcusable to have the current situation and still claim they have good backups. "The issues lays with the server being compromised and being forced to shut down. The old API and apps relied on a fixed IP address. It would have taken us months to get TinyGrab v1 back up to scratch" Ok, computer scientists reading this: To which of you does this make sense? It's hard to keep counts of all the fails in here: - If your problem is a corrupted server, replace it and assign the same IP address. - Why would anyone hard-code IP addresses for an API to work? Were you born yesterday? - It should take at the very most hours to recover from a truly catastrophic failure. "Months" you say? You do not know what you're doing, I say. - They have a bunch of backup and not one works, specially when there is no data loss? Smells to me like no source control; no off site backup, and absolutely no testing of the backup. If you have lived a single day in system admin work, you know that it is not enough to do a backup. You need to test the damned restore. Maybe this could be a story about when _not_ to self-finance your startup. Get outside funding and hire people that know more than you do about the technology involved. Sorry for being an asshole in my comments, but this situation is full of fail at every step.

consultutah 16y ago
No need to get outside funding for any of this. It can all be done on the cheap. I agree though, how can it take months to reset up a server? Late last year I had both of our supposedly redundant webservers fail at the same time - never happens, right? ;) We were able to get two entirely separate boxes setup with the same static addresses in under an hour due to maintaining all possible configuration in svn. Worse, this was on windows servers where we couldn't use something like puppet or one of the other 20 solutions to script your configuration.
datasink 16y ago
I believe there is a complete, one sentence explanation that would ring true for most developers, but is not being offered for one reason or another. My best guess: It would have been easier to get 1.0 back up and working, but this was viewed as an opportunity to finally push out the 2.0 and convert over the vast majority of the existing user-base in short order. The release was not tested well enough, and so this was a bad plan in hindsight.
- gm 16y ago
  Well, the worst time to change technologies or perform version updates is when there is a fire under your ass to get the system working, but that is a minor screw up compared to all else going on. This whole thing stinks of lack of experience. No one should repeat the steps these guys are taking.
markkanof 16y ago
I guess a lot of this could be attributed to the fact that the TinyGrab developers are volunteers (at least according to the article). Like many open source projects they probably focused on the glamourous work of actually coding the app and not the boring stuff like backups. I'm not saying this is acceptable, I'm just suggesting an explanation. I could sort of see not having a completely bulletproof backup strategy in place, but with all the options available, I can't imagine not working with source control.
samengland 16y ago
You said it - you are indeed being an arsehole in your comments. Firstly: "...this could be a story about when _not_ to self-finance your startup" That's easier said than done. There is absolutely no point in over analysing the situation and then concluding that TinyGrab should have got "outside funding" and "hired people that know more than they do" about the technology involved - because what is done, is done. You are seriously insulting the intelligence of everyone behind TinyGrab if you think they don't already know that version one was problematic. The point here is that the popularity of TinyGrab was never predicted or planned and that it grew from strength to strength entirely organically. At no point did they think about scalability from the outset - so they acknowledged the problem and were rectifying this with a full rewrite in TinyGrab 2. They we're simply unlucky in that they were unable to roll this update out quickly enough - they were hacked before they could complete TinyGrab 2, and therefore did the right thing and released TinyGrab 2 which was very nearly complete anyway. With the lack of income and voluntary manpower they had, they couldn't have changed anything in hindsight at all. So in that respect this was not "full of fail" as you so crudely put it. Secondly: What you need to do is put yourself in the shoes of those battling to actually keep TinyGrab online, and realise that they are actually trying their best to respond to every support request and resume the excellent level of service their users have come to expect - and frankly they've almost achieved that now. What you also need to realise is that those behind TinyGrab are not in it for commercial gain - they are in it because they love the product and their users. Since this is a completely free service they could have very easily - and might I add completely legally closed TinyGrab as of today, with no risk of litigation against them. They would also most probably refund existing premium users, but would not be legally obliged to. What they have instead chosen to do, is keep this service online, do their best to get TinyGrab 2 to the bare minimum in order to roll it out completely, and work day and night as a team to respond to every support request and @mention on Twitter to keep users informed and solve their issues individually. I think this is absolutely admirable given their 300,000+ user base and could never be described as "full of fail" especially since the vast proportion of their users do not pay them a penny - and especially since those behind TinyGrab work in their own free time on the project whilst receiving no commercial gain whatsoever. TinyGrab will easily recover from this over the coming days - all they need at this moment in time is some moral support and some understanding - a bit of slack if you like - not a bunch of anonymous, cynical, patronising and moronic nobodies trying to tell them how it should have been done in hindsight. As I said above, it's frankly just insulting the intelligence of those behind it - any reasonable person can see that they were fully aware of the pitfalls of TinyGrab version one and were trying their best to roll out version two. Give them a break.
lethain 16y ago
Regarding the hard-coded IP piece, I've run into a similar scenario wherein the early versions of the distributed counters patch for Cassandra relied on vector-clocks which used IPs as part of those clocks. Certainly not a robust design in the case of machine failure, but I've seen smart people make awkward decisions, and if someone had gone into production with that patch and lost a machine, it would have been remarkably annoying to recover (it would have required rewriting all the SSTables to a new IP, which isn't impossible or anything, but kind of a pita).

by gm 16y ago

consultutah 16y ago

No need to get outside funding for any of this. It can all be done on the cheap. I agree though, how can it take months to reset up a server? Late last year I had both of our supposedly redundant webservers fail at the same time - never happens, right? ;) We were able to get two entirely separate boxes setup with the same static addresses in under an hour due to maintaining all possible configuration in svn. Worse, this was on windows servers where we couldn't use something like puppet or one of the other 20 solutions to script your configuration.

datasink 16y ago

I believe there is a complete, one sentence explanation that would ring true for most developers, but is not being offered for one reason or another. My best guess: It would have been easier to get 1.0 back up and working, but this was viewed as an opportunity to finally push out the 2.0 and convert over the vast majority of the existing user-base in short order. The release was not tested well enough, and so this was a bad plan in hindsight.

gm 16y ago
Well, the worst time to change technologies or perform version updates is when there is a fire under your ass to get the system working, but that is a minor screw up compared to all else going on. This whole thing stinks of lack of experience. No one should repeat the steps these guys are taking.

markkanof 16y ago

I guess a lot of this could be attributed to the fact that the TinyGrab developers are volunteers (at least according to the article). Like many open source projects they probably focused on the glamourous work of actually coding the app and not the boring stuff like backups. I'm not saying this is acceptable, I'm just suggesting an explanation. I could sort of see not having a completely bulletproof backup strategy in place, but with all the options available, I can't imagine not working with source control.

samengland 16y ago

You said it - you are indeed being an arsehole in your comments. Firstly: "...this could be a story about when _not_ to self-finance your startup" That's easier said than done. There is absolutely no point in over analysing the situation and then concluding that TinyGrab should have got "outside funding" and "hired people that know more than they do" about the technology involved - because what is done, is done. You are seriously insulting the intelligence of everyone behind TinyGrab if you think they don't already know that version one was problematic. The point here is that the popularity of TinyGrab was never predicted or planned and that it grew from strength to strength entirely organically. At no point did they think about scalability from the outset - so they acknowledged the problem and were rectifying this with a full rewrite in TinyGrab 2. They we're simply unlucky in that they were unable to roll this update out quickly enough - they were hacked before they could complete TinyGrab 2, and therefore did the right thing and released TinyGrab 2 which was very nearly complete anyway. With the lack of income and voluntary manpower they had, they couldn't have changed anything in hindsight at all. So in that respect this was not "full of fail" as you so crudely put it. Secondly: What you need to do is put yourself in the shoes of those battling to actually keep TinyGrab online, and realise that they are actually trying their best to respond to every support request and resume the excellent level of service their users have come to expect - and frankly they've almost achieved that now. What you also need to realise is that those behind TinyGrab are not in it for commercial gain - they are in it because they love the product and their users. Since this is a completely free service they could have very easily - and might I add completely legally closed TinyGrab as of today, with no risk of litigation against them. They would also most probably refund existing premium users, but would not be legally obliged to. What they have instead chosen to do, is keep this service online, do their best to get TinyGrab 2 to the bare minimum in order to roll it out completely, and work day and night as a team to respond to every support request and @mention on Twitter to keep users informed and solve their issues individually. I think this is absolutely admirable given their 300,000+ user base and could never be described as "full of fail" especially since the vast proportion of their users do not pay them a penny - and especially since those behind TinyGrab work in their own free time on the project whilst receiving no commercial gain whatsoever. TinyGrab will easily recover from this over the coming days - all they need at this moment in time is some moral support and some understanding - a bit of slack if you like - not a bunch of anonymous, cynical, patronising and moronic nobodies trying to tell them how it should have been done in hindsight. As I said above, it's frankly just insulting the intelligence of those behind it - any reasonable person can see that they were fully aware of the pitfalls of TinyGrab version one and were trying their best to roll out version two. Give them a break.

lethain 16y ago

Regarding the hard-coded IP piece, I've run into a similar scenario wherein the early versions of the distributed counters patch for Cassandra relied on vector-clocks which used IPs as part of those clocks. Certainly not a robust design in the case of machine failure, but I've seen smart people make awkward decisions, and if someone had gone into production with that patch and lost a machine, it would have been remarkably annoying to recover (it would have required rewriting all the SSTables to a new IP, which isn't impossible or anything, but kind of a pita).