4 ms·

How Twitter Killed My Fish

26 points by dnewms 12y ago

Zigurd 12y ago
More like how not having a heartbeat message that confirms Twitter-based notifications are working killed the fish. And why Twitter? Seems like any of email, IM, or SMS would be a better choice. This is like not starting your emergency generators on a regular schedule and then being surprised that the fuel filter clogged.
- wpietri 12y ago
  This is one of those comments that relies on hindsight to sound smart, so it's frustrating to me. The author surely can also think up a solution now that he knows Twitter can't be trusted. In fact, as he explains in the comments, he switched to SMS months ago.
  - mikeash 12y ago
    Realizing that Twitter is not reliable is not hindsight! It is blatantly obvious! In terms of reliability, it's one step above putting a message in a bottle and tossing it in the ocean. If you didn't already know this then you're not paying attention.
    - wpietri 12y ago
      Here I take it "blatantly obvious" is code for "thing I know" or, less charitably, "thing I can claim to know so I can sound superior, now that it's obvious in hindsight". If you want to persuade me that it's really obvious, that only a fool would make this mistake, you'd have to show me where in the Twitter developer docs they say, "Hey, this is unreliable, so don't think we're going to actually deliver your messages". And then that a reasonable person would have had to see that page, read it, and ignore it before proceeding. Half of people are below average. Everybody is ignorant about most things, and they're always learning. I have no reason to think that this guy didn't make a reasonable novice mistake. I'd rather encourage novices to try things. Telling them after the fact, "Fool, you should have known!" only discourages them and other novices. Personally, I appreciate him producing an interesting article that will help other novices avoid his mistake.
      - mikeash 12y ago
        Twitter's terms of service are located here: https://twitter.com/tos https://twitter.com/tos Two choice bits: "We may suspend or terminate your accounts or cease providing you with all or part of the Services at any time for any reason...." "The Twitter Entities make no warranty and disclaim all responsibility and liability for: (i) the completeness, accuracy, availability, timeliness, security or reliability of the Services or any Content...." Emphasis added. Now I guess you'll say that a reasonable person won't read the TOS anyway so it doesn't count. And I say, that's all well and good when you're tweeting about last night's dinner, but when you're building something where you actually care about the consequences when it fails, you have to actually perform some form of research if you want reliability. I'm rather astonished that you're still acting like Twitter's unreliability is not only non-obvious, but something I wasn't aware of until I read this post, but am pretending that I knew before so I could act superior. Do you just assume that every random internet service out there is reliable until proven otherwise? Because that's the opposite of what's correct. Everything is completely unreliable until proven otherwise. This is, yes, obvious if you've been paying attention to the internet for any reasonable length of time. Twitter hasn't proven otherwise, therefore it is unreliable. I personally think that more than passes the threshold for "blatantly obvious". I disagree with you on the impact of pointing out that this was a bad decision. The fact that you can't rely on random internet services is an important lesson, and I don't think he learned that lesson. He just switched from one random unreliable internet service to another. When Zapier or Xively go down or change authentication requirements or decide he's a spammer and shut down his account, does he have some way to find out? I would much rather have someone tell me "that was pretty stupid of you, this was your real mistake here" than hide the truth from me because they're afraid of discouraging me. I want to be wrong for as little time as possible. Apparently this is an unusual sentiment, which I have a hard time coming to terms with.
        bryanh 12y ago
        Though I can't really comment on the discussion as a whole, I can say with good authority that if Zapier zaps fail (for any reason), we will try to send out alert emails to the best of our ability!
        12y ago
  - Zigurd 12y ago
    Rereading it it does sound a bit unfair in tone, but it is not wisdom in hindsight to think of a heartbeat that provides affirmative notification that notifications are working, especially in a system that's got a lot of other data acquisition that could be provided on a heartbeat. He shouldn't blindly trust that SMS will work for for his application, using his setup, either. What if he forgets to top up the account for the device sending out SMS? That would not be the fault of the SMS system.
mikeash 12y ago
More accurate title would be: How I Killed My Fish By Mistaking Twitter For A Reliable Messaging Platform.
allegory 12y ago
And that is a fine example of why you don't rely on such a Rube Goldberg machine for stuff like this... A poor engineering decision killed the fish. I see all sorts of crazy contraptions people have rigged up over the last 5 years since the appearance of the 'maker movement'. Surprised that fish are the only victim so far! Sure you can wire up your coffee machine to listen for a tweet and have coffee ready for when you get in... then when you get there your flat is on fire... Excuse the cynicism but as a professional engineer it scares me how dangerously and poorly thought out some of this sort of stuff is.
- noir_lord 12y ago
  On one level I agree with you however I think that the barn door is open and the horses are heading for the treeline. I trained as an industrial electrician before becoming a professional programmer (I programmed long before either) so I have an incredibly healthy respect for electricity and machinery. That said I also think the benefits of putting hardware and software into the hands of talented new users is well worth the (potential) downsides, I broke many many things to learn how to not break things, that is how you progress in the early days.
  - allegory 12y ago
    > On one level I agree with you however I think that the barn door is open and the horses are heading for the treeline. The horses are on fire. Agree entirely though.
- wpietri 12y ago
  The number of professional engineers we have is vastly too small to make good use of the enormous power that computers have handed us in the last 20 years. I see our options as: a) be condescending dicks every time a novice makes a novice mistake, b) leave enormous amounts of technical value untapped until professionals can get around to it, or c) use our professional engineering smarts to build tools that amateur engineers can succeed with. I favor C.
  - mikeash 12y ago
austinz 12y ago
From the Twitter terms of service: "The Twitter Entities make no warranty and disclaim all responsibility and liability for: (i) the completeness, accuracy, availability, timeliness, security or reliability of the Services or any Content; (ii) any harm to your computer system, loss of data, or other harm that results from your access to or use of the Services or any Content; (iii) the deletion of, or the failure to store or to transmit, any Content and other communications maintained by the Services; and (iv) whether the Services will meet your requirements or be available on an uninterrupted, secure, or error-free basis." Clearly it was never meant as a reliable communications channel for any sort of M2M application where failure could result in significant economic loss or a threat to human health or life.
- Houshalter 12y ago
  That just means you can't sue them. I'm sure any service can think of has a similar warning in their ToS. Did he have any reason to believe Twitter was more likely to fail than whatever alternative?
  - wpietri 12y ago
    Exactly. And I'd add: should have he expected it to fail where he gets silently blocked but everything else works as normal? One of the advantages of using something like Twitter is that if you're a regular user, you'll have a good idea of when it stops working altogether or is experiencing significant reliability problems. That there's a failure mode where they silently break your app but nothing else sure isn't mentioned in the TOS.
  - austinz 12y ago
    > Did he have any reason to believe Twitter was more likely to fail Whether or not Twitter is more or less reliable than any similar alternative doesn't really matter that much, especially if none of the alternatives offer anything more than best-effort service with no guarantee of reliability. The fact that Twitter was (at least superficially) suitable for this sort of application was probably not intended by Twitter's developers, nor was the existence of the pathological corner case that ruined OP's fish tank anything more than a unfortunate coincidence. It absolutely sucks that OP's fish died, but if Twitter explicitly disavows any sort of loss because of the possibility that their service might change or fail, it's doesn't seem reasonable to just ignore that and assume otherwise. Especially for an application where it doesn't matter if the service is up most of the time if it ends up failing during the brief period where it's needed the most. And if you are aware of the limitations of some aspect of your system, it then becomes possible to design redundancy into the other parts to compensate. I hope OP's next fishtank adventure keeps this in mind.
rbinv 12y ago
This is absolutely ridiculous. Why on earth would you rely on something like Twitter to get critical messages delivered?
- wpietri 12y ago
  Because you are somebody whose expertise is aquaculture, not software development? Because Twitter is the thing you use? Because the docs weren't clear? Because the APIs weren't really thought through for use cases like this?
  - kelnos 12y ago
    If the guy put together an Arduino with custom software to control his aquaponics system and interface with services on the internet, I would venture to guess that he has some expertise in software development. Enough not to make such a terrible mistake.
11thEarlOfMar 12y ago
This is just the beginning. Connecting things to the Internet is a whole new kettle of fish and there is lots for the novice developer to learn. There will be disasters of all scales. So we need a new school of thought.
deleted 12y ago
[deleted]
sgulseth 12y ago
So, you let a bird guard a fish?
ColdWeatherAqua 12y ago
Hi everyone. Author here. Gotta say I loved all the comments. My blog followers are all so nice. It feels good to get some real criticism for once :) Relying on Twitter was a truly bonehead move, to be sure. Truly a "bang head here" kind of moment. However, if trusting Twitter was the only "poor engineering decision" I had made I'd be a very happy man. It was just one of dozens of bonehead mistakes I made in building and operating my first aquaponics system. Aquaponics is one of those things where the path of least resistance is definitely following a set of plans that are known to work. You can find one of those on my site, and some other places too. But I've never one to follow the path of least resistance (eg. Screw the round wheel - I'm making mine with fractals). In case anyone's interested, I'm a former electrical engineer who used this as my first attempt at something with Arduino. More of a power-systems and lighting guy, software development was and still is a foreign land to me. I could have tested and researched the reliability of different notification platforms, but at the time I was building it I was already staying up late fixing plumbing problems and trying to get those d*&$ one-wire sensors to work consistently. Thankfully I eventually got my plumbing and sensors working, and bungled along into into Zapier which has been absolutely bulletproof. I think the truly useful lesson here is that if you ever meet anyone in aquaculture who can tell a good story at a party, plop yourself down and ask them if they've ever killed fish in any bizarre and interesting ways. Don't get up for drinks either because you won't want to miss anything. You'll be there all night.

26 points by dnewms 12y ago

More like how not having a heartbeat message that confirms Twitter-based notifications are working killed the fish. And why Twitter? Seems like any of email, IM, or SMS would be a better choice. This is like not starting your emergency generators on a regular schedule and then being surprised that the fuel filter clogged.

wpietri 12y ago
This is one of those comments that relies on hindsight to sound smart, so it's frustrating to me. The author surely can also think up a solution now that he knows Twitter can't be trusted. In fact, as he explains in the comments, he switched to SMS months ago.
- mikeash 12y ago
  Realizing that Twitter is not reliable is not hindsight! It is blatantly obvious! In terms of reliability, it's one step above putting a message in a bottle and tossing it in the ocean. If you didn't already know this then you're not paying attention.
  - wpietri 12y ago
    Here I take it "blatantly obvious" is code for "thing I know" or, less charitably, "thing I can claim to know so I can sound superior, now that it's obvious in hindsight". If you want to persuade me that it's really obvious, that only a fool would make this mistake, you'd have to show me where in the Twitter developer docs they say, "Hey, this is unreliable, so don't think we're going to actually deliver your messages". And then that a reasonable person would have had to see that page, read it, and ignore it before proceeding. Half of people are below average. Everybody is ignorant about most things, and they're always learning. I have no reason to think that this guy didn't make a reasonable novice mistake. I'd rather encourage novices to try things. Telling them after the fact, "Fool, you should have known!" only discourages them and other novices. Personally, I appreciate him producing an interesting article that will help other novices avoid his mistake.
    - mikeash 12y ago
      Twitter's terms of service are located here: https://twitter.com/tos https://twitter.com/tos Two choice bits: "We may suspend or terminate your accounts or cease providing you with all or part of the Services at any time for any reason...." "The Twitter Entities make no warranty and disclaim all responsibility and liability for: (i) the completeness, accuracy, availability, timeliness, security or reliability of the Services or any Content...." Emphasis added. Now I guess you'll say that a reasonable person won't read the TOS anyway so it doesn't count. And I say, that's all well and good when you're tweeting about last night's dinner, but when you're building something where you actually care about the consequences when it fails, you have to actually perform some form of research if you want reliability. I'm rather astonished that you're still acting like Twitter's unreliability is not only non-obvious, but something I wasn't aware of until I read this post, but am pretending that I knew before so I could act superior. Do you just assume that every random internet service out there is reliable until proven otherwise? Because that's the opposite of what's correct. Everything is completely unreliable until proven otherwise. This is, yes, obvious if you've been paying attention to the internet for any reasonable length of time. Twitter hasn't proven otherwise, therefore it is unreliable. I personally think that more than passes the threshold for "blatantly obvious". I disagree with you on the impact of pointing out that this was a bad decision. The fact that you can't rely on random internet services is an important lesson, and I don't think he learned that lesson. He just switched from one random unreliable internet service to another. When Zapier or Xively go down or change authentication requirements or decide he's a spammer and shut down his account, does he have some way to find out? I would much rather have someone tell me "that was pretty stupid of you, this was your real mistake here" than hide the truth from me because they're afraid of discouraging me. I want to be wrong for as little time as possible. Apparently this is an unusual sentiment, which I have a hard time coming to terms with.
      - bryanh 12y ago
        Though I can't really comment on the discussion as a whole, I can say with good authority that if Zapier zaps fail (for any reason), we will try to send out alert emails to the best of our ability!
      - 12y ago
- Zigurd 12y ago
  Rereading it it does sound a bit unfair in tone, but it is not wisdom in hindsight to think of a heartbeat that provides affirmative notification that notifications are working, especially in a system that's got a lot of other data acquisition that could be provided on a heartbeat. He shouldn't blindly trust that SMS will work for for his application, using his setup, either. What if he forgets to top up the account for the device sending out SMS? That would not be the fault of the SMS system.

mikeash 12y ago

More accurate title would be: How I Killed My Fish By Mistaking Twitter For A Reliable Messaging Platform.

allegory 12y ago

And that is a fine example of why you don't rely on such a Rube Goldberg machine for stuff like this... A poor engineering decision killed the fish. I see all sorts of crazy contraptions people have rigged up over the last 5 years since the appearance of the 'maker movement'. Surprised that fish are the only victim so far! Sure you can wire up your coffee machine to listen for a tweet and have coffee ready for when you get in... then when you get there your flat is on fire... Excuse the cynicism but as a professional engineer it scares me how dangerously and poorly thought out some of this sort of stuff is.

noir_lord 12y ago
On one level I agree with you however I think that the barn door is open and the horses are heading for the treeline. I trained as an industrial electrician before becoming a professional programmer (I programmed long before either) so I have an incredibly healthy respect for electricity and machinery. That said I also think the benefits of putting hardware and software into the hands of talented new users is well worth the (potential) downsides, I broke many many things to learn how to not break things, that is how you progress in the early days.
- allegory 12y ago
  > On one level I agree with you however I think that the barn door is open and the horses are heading for the treeline. The horses are on fire. Agree entirely though.
wpietri 12y ago
The number of professional engineers we have is vastly too small to make good use of the enormous power that computers have handed us in the last 20 years. I see our options as: a) be condescending dicks every time a novice makes a novice mistake, b) leave enormous amounts of technical value untapped until professionals can get around to it, or c) use our professional engineering smarts to build tools that amateur engineers can succeed with. I favor C.
- mikeash 12y ago

austinz 12y ago

From the Twitter terms of service: "The Twitter Entities make no warranty and disclaim all responsibility and liability for: (i) the completeness, accuracy, availability, timeliness, security or reliability of the Services or any Content; (ii) any harm to your computer system, loss of data, or other harm that results from your access to or use of the Services or any Content; (iii) the deletion of, or the failure to store or to transmit, any Content and other communications maintained by the Services; and (iv) whether the Services will meet your requirements or be available on an uninterrupted, secure, or error-free basis." Clearly it was never meant as a reliable communications channel for any sort of M2M application where failure could result in significant economic loss or a threat to human health or life.

Houshalter 12y ago
That just means you can't sue them. I'm sure any service can think of has a similar warning in their ToS. Did he have any reason to believe Twitter was more likely to fail than whatever alternative?
- wpietri 12y ago
  Exactly. And I'd add: should have he expected it to fail where he gets silently blocked but everything else works as normal? One of the advantages of using something like Twitter is that if you're a regular user, you'll have a good idea of when it stops working altogether or is experiencing significant reliability problems. That there's a failure mode where they silently break your app but nothing else sure isn't mentioned in the TOS.
- austinz 12y ago
  > Did he have any reason to believe Twitter was more likely to fail Whether or not Twitter is more or less reliable than any similar alternative doesn't really matter that much, especially if none of the alternatives offer anything more than best-effort service with no guarantee of reliability. The fact that Twitter was (at least superficially) suitable for this sort of application was probably not intended by Twitter's developers, nor was the existence of the pathological corner case that ruined OP's fish tank anything more than a unfortunate coincidence. It absolutely sucks that OP's fish died, but if Twitter explicitly disavows any sort of loss because of the possibility that their service might change or fail, it's doesn't seem reasonable to just ignore that and assume otherwise. Especially for an application where it doesn't matter if the service is up most of the time if it ends up failing during the brief period where it's needed the most. And if you are aware of the limitations of some aspect of your system, it then becomes possible to design redundancy into the other parts to compensate. I hope OP's next fishtank adventure keeps this in mind.

rbinv 12y ago

This is absolutely ridiculous. Why on earth would you rely on something like Twitter to get critical messages delivered?

wpietri 12y ago
Because you are somebody whose expertise is aquaculture, not software development? Because Twitter is the thing you use? Because the docs weren't clear? Because the APIs weren't really thought through for use cases like this?
- kelnos 12y ago
  If the guy put together an Arduino with custom software to control his aquaponics system and interface with services on the internet, I would venture to guess that he has some expertise in software development. Enough not to make such a terrible mistake.

11thEarlOfMar 12y ago

This is just the beginning. Connecting things to the Internet is a whole new kettle of fish and there is lots for the novice developer to learn. There will be disasters of all scales. So we need a new school of thought.

deleted 12y ago

[deleted]

sgulseth 12y ago

So, you let a bird guard a fish?

ColdWeatherAqua 12y ago

Hi everyone. Author here. Gotta say I loved all the comments. My blog followers are all so nice. It feels good to get some real criticism for once :) Relying on Twitter was a truly bonehead move, to be sure. Truly a "bang head here" kind of moment. However, if trusting Twitter was the only "poor engineering decision" I had made I'd be a very happy man. It was just one of dozens of bonehead mistakes I made in building and operating my first aquaponics system. Aquaponics is one of those things where the path of least resistance is definitely following a set of plans that are known to work. You can find one of those on my site, and some other places too. But I've never one to follow the path of least resistance (eg. Screw the round wheel - I'm making mine with fractals). In case anyone's interested, I'm a former electrical engineer who used this as my first attempt at something with Arduino. More of a power-systems and lighting guy, software development was and still is a foreign land to me. I could have tested and researched the reliability of different notification platforms, but at the time I was building it I was already staying up late fixing plumbing problems and trying to get those d*&$ one-wire sensors to work consistently. Thankfully I eventually got my plumbing and sensors working, and bungled along into into Zapier which has been absolutely bulletproof. I think the truly useful lesson here is that if you ever meet anyone in aquaculture who can tell a good story at a party, plop yourself down and ask them if they've ever killed fish in any bizarre and interesting ways. Don't get up for drinks either because you won't want to miss anything. You'll be there all night.