10 ms·

The most popular apps on one of the most popular operating systems just plainly don't even open. What has this industry become? How are we so goddamn inept at

The most popular apps on one of the most popular operating systems just plainly don't even open. What has this industry become? How are we so goddamn inept at writing software? This is an industry where we can automate testing - not many other industries have this capability - and we still don't have a simple test regime of "check if the app opens" for any of those apps. Somehow, at the most valuable software companies of the world, nobody has set up a system that makes sure that you can't introduce a change that renders these apps unusable? WHAT THE FUCK. I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess.

rachelbythebay 6y ago
Years ago, I had one of the employee builds on my company phone, and it picked up an update and would "instacrash" - tap the icon, and it'd die immediately. Crash. Crash. Crash. A couple of days later I caught up with the people allegedly responsible for this stuff in the bigger review meeting. "Shouldn't you open a SEV because you are shipping stuff that can't possibly work and aren't catching it pre-release?" The response was a lot of hand-waving and stupid manager-speak that boiled down to "no, go away". I basically wrote them off at that point. If they could act that way and get away with it by way of implicit acceptance from upper management, that's just the way it was going to be. I see the quality is as high as always.
- exikyut 6y ago
  Curiously, this type of handwavy response reveals a not-quite-1:1 mapping between the sort of assertive, pro-admission, fail-forward balance that "move fast, break things" implies for me, and the apparent reality which seems closer to "haha crowdsourced QC go brrrrt" :(
- fsociety 6y ago
  Yep at such a large company there is such huge variance in quality. Wish you and many others were still here to help fight against this mindset. There is lots of high quality work done here, but it doesn’t matter if we end up with issues like this.
yreg 6y ago
The apps are broken serverside by Facebook. No amount of testing (automated or not) on the side of the app developers would have prevented this. Facebook's own apps did not break since they are on the latest version of FBSDK. What Facebook needs to do is to test whether their changes don't break other people's apps.
- akerro 6y ago
  Are you telling me that Facebook Inc. has no capacity or ability to test their own, one of the most popular SDKs in the world, in the wild? Like have a single phone connected by USB and open spotify app and simply login after they make a backend change?
  - yreg 6y ago
    By app developers I mean the consumers of FBSDK, not Facebook.
- linux2647 6y ago
  I disagree. While, yes, Facebook needs to test their SDK, app developers need to build resiliency into their apps to ensure third party dependencies don’t break their app. The developer adopted it into their app; it’s their responsibility to do the proper checks in cases like these where Facebook broke things on their end.
  - corndoge 6y ago
    Ah yes. If hacker news goes down while you are writing this comment, it is your responsibility to ensure you can still post your comment despite the service breakage.
    - slipheen 6y ago
      Unless your app is designed for browsing Facebook, there's no reason that anything on Facebook's servers should affect if your app works. Many of the apps crashing are using Facebook for a portion of their app, such as one login option. Letting that take down the whole app is insufficiently defensive coding, whether in the app, sdk, or glue between them.
blantonl 6y ago
Bugs happen. Tests miss edge cases. Shit happens. This will be a fart in the wind and lessons will be learned.
- emerongi 6y ago
  Maybe lessons will be learned, maybe not. The industry seems to have very low standards. Let's let these mistakes happen over and over and let's just patch them as we go. No long-term thinking. Someone should collect all these horrible incidents and start a university course on just plainly what NOT to do.
  - blantonl 6y ago
    Some of the very brightest software engineers in the world work at Facebook. As much as I would love to rip on Facebook, I also understand that integrated software development is hard, and I guarantee you that Facebook is "doing it better" than most. Don't sit in an ivory tower and throw stones in glass houses. Can you guarantee that your own SDLC house is in order? Because it might be, right up until a sneaky edge case nips you in the butt, and then suddenly it isn't.
    - mattgreenrocks 6y ago
      Let's avoid lionizing/denigrating here. They're using static initialization and it bit them in the ass again. They should stop using it because it is prone to things like this which result in apps not launching at all. I'd like to see them commit to stop using this method to initialize themselves, and take responsibility for the crashes. Then, IMO, we'd be good. In a scenario like this, I'd hope they'd want to make this bug's recurrence impossible.
      - tdrp 6y ago
        It was horrifying to see how many config changes you have to do to basically beg facebook not to send back data to their server. Most of these were hidden in various corners of the documentation and I still suspect there's crap being sent back without permission. I would love nothing more than to disable the SDK entirely but we have a few hundred thousand users logged into our app via Facebook. Here is our Android config in case anyone's interested: <meta-data android:name="com.facebook.sdk.AutoLogAppEventsEnabled" android:value="false"/> <meta-data android:name="com.facebook.sdk.AdvertiserIDCollectionEnabled" android:value="false"/> <meta-data android:name="com.facebook.sdk.AutoInitEnabled" android:value="false"/>
vsareto 6y ago
>How are we so goddamn inept at writing software? Wow who would have thought no standards, free access to learning, fake-ass meritocracy, and exorbitant profits and market control would have led to such a thing Pick a standard, whether it's a degree program, a cert, or apprenticeships. Hurry up! Just give us a solid target to aim for.
babesh 6y ago
There is more complexity in this. Proximate Issues The app makers cannot easily prevent the crash since it is happening in Facebook provided code. Perhaps the only thing they could do is to code in kill switches. Another possibility would be to modify the Facebook code. That introduces maintainability issues. The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception. Objective-C is an old unsafe language. Ultimate Issues Client-side libraries have a poor history in the industry. They are often written by server engineers who aren’t expert on the platform. For instance, if it was really a static load method that called the server, then this was a poor design decision. Furthermore, weaker server engineers are notorious for breaking backward compatibility. They don’t design nor build in tests to ensure backward compatibility. The backward compatibility must include backward data compatibility. The above issues are indicative of organizational issues. Facebook is a little bit of an outlier. It was notorious for badly written client libraries stretching back to the Facebook platform days. The joke was that their libraries may have been written by interns even though so many companies were dependent on them. The Facebook libraries and the ilk are used for sign on through Facebook and ad attribution. The trade off an app maker needs to make it a whether to run ads on Facebook and whether to support sign in. Facebook is not the only big company with these issues. Google’s equivalent also crashed albeit at a much lower rate due to a much more convoluted issue. Apple introduced the Swift programming language to address the deficiencies in Objective-C. That these companies are running into these issues makes me wonder if they never transitioned to Swift. They are certainly way behind the latest version of the libraries. To summarize, it is a cascade of issues some technical and some organizational rather than a single coding error. These issues span companies.
- saagarjha 6y ago
  > The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception. > Objective-C is an old unsafe language. This was a safe crash, not an unsafe one. Swift can and would crash the same way.
  - babesh 6y ago
    You can’t safely call a method on a nil value in Swift. You need to explicitly force that in your code with an exclamation point to assume that the variable is not nil. Looks like the server passed empty value in JSON which got converted to NSNull in Objective-C. It's a common mistake to assume null is returned instead of NSNull. The Swift equivalent of the JSON library should return nil which you then need to explicitly check for non nilness before using unless you use the unsafe exclamation pint.
ljm 6y ago
The fact that so many popular apps have to pull in a full-blown SDK from Facebook to get support for even a subset of its functionality is insane to me, too. That 'log in with Facebook' button on your app comes with an entire suite of stuff that Facebook can use to slurp up any data it wants.
notJim 6y ago
These histrionics are ridiculous. There has never been an era in software engineering where people didn't write bugs. There has never been an era where people didn't miss bugs. You can scream and moan at the top of your lungs all you want, but shaming and calling people incompetent does not contribute anything. > I will have to go and get a degree Was not aware that having a degree made you immune to writing bugs.
jasonv 6y ago
This is the second time. Recently tested an enterprise marketing app with an SDK ... and we tested if the endpoint wasn't available. Nothing happened -- it worked just fine. Failed quietly and gracefully -- the user would never notice. "We" know how to do this. Facebook hasn't done anything to fix this problem since it happened a few weeks ago. But it's Facebook -- I hope they keep it up. Can't wait for them to AOL themselves.
- layoutIfNeeded 6y ago
  The problem is not that the endpoint is not available, rather that the endpoint is returning garbage on which the parser chokes. Of course you should always write parsing to be 100% resilient against any kind of garbage input (even malicious), but my point is that your marketing SDK could still be a ticking timebomb.
  - jasonv 6y ago
    True, yes. But this is for a financial services mobile app... one of the first concerns was the user experience. The org is well versed in the other back-end security and data issues that also have to be looked at. But if you're Facebook, and you're pushing your SDK as a great solution and it's embedded in high profile apps and you are letting this happen a couple of times, I get to make fun of you. (a dollop of sarcasm, to be sure..)
kobalsky 6y ago
>>> I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess. This would still happen because A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them, B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff. The engineering process was the issue here, and sadly the only way to prevent this is not to make personal barriers of entry higher but to slow down the process with more QA, testing or whatever you can throw at it. In other words, make software more expensive. And keep in mind that this would only give better numbers but it would not solve the issue either (see Boeing 737 MAX).
- jfkebwjsbx 6y ago
  > A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them Very unlikely. You probably don't realize how tough some professional exams are in some fields. We are talking dedicated months of study here. > B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff Most software out there is just CRUD stuff, not really new stuff...
brink 6y ago
> I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. There's a good chance that the one who wrote this bug had a degree. Surgeries in medicine still get botched despite all of their degrees and barriers. Degrees don't produce good code, good developers produce good code.
- mushbino 6y ago
  There's no standardized process for producing a "good developer". Every company has a different process and different standards. I think that's a part of the problem and I agree that it's not one academia can solve, but I wonder if there could potentially be another way.
StreamBright 6y ago
I try to avoid software in general as much as possible.
dang 6y ago
> WHAT THE FUCK Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
marcinzm 6y ago
>I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess. I can almost guarantee that the programmer who wrote this had a degree and passed a very rigorous testing process to get their job at Facebook. Doesn't mean anything when the company tells them to cut corners to iterate faster. Stop blaming engineers for the mistakes of management.
- stjohnswarts 6y ago
  Yeah it's not like most programmers wouldn't prefer to be writing good software but the "let the user test it" cycle of quick turnaround sprint culture is always going to have this issue.

rachelbythebay 6y ago

Years ago, I had one of the employee builds on my company phone, and it picked up an update and would "instacrash" - tap the icon, and it'd die immediately. Crash. Crash. Crash. A couple of days later I caught up with the people allegedly responsible for this stuff in the bigger review meeting. "Shouldn't you open a SEV because you are shipping stuff that can't possibly work and aren't catching it pre-release?" The response was a lot of hand-waving and stupid manager-speak that boiled down to "no, go away". I basically wrote them off at that point. If they could act that way and get away with it by way of implicit acceptance from upper management, that's just the way it was going to be. I see the quality is as high as always.

exikyut 6y ago
Curiously, this type of handwavy response reveals a not-quite-1:1 mapping between the sort of assertive, pro-admission, fail-forward balance that "move fast, break things" implies for me, and the apparent reality which seems closer to "haha crowdsourced QC go brrrrt" :(
fsociety 6y ago
Yep at such a large company there is such huge variance in quality. Wish you and many others were still here to help fight against this mindset. There is lots of high quality work done here, but it doesn’t matter if we end up with issues like this.

yreg 6y ago

The apps are broken serverside by Facebook. No amount of testing (automated or not) on the side of the app developers would have prevented this. Facebook's own apps did not break since they are on the latest version of FBSDK. What Facebook needs to do is to test whether their changes don't break other people's apps.

akerro 6y ago
Are you telling me that Facebook Inc. has no capacity or ability to test their own, one of the most popular SDKs in the world, in the wild? Like have a single phone connected by USB and open spotify app and simply login after they make a backend change?
- yreg 6y ago
  By app developers I mean the consumers of FBSDK, not Facebook.
linux2647 6y ago
I disagree. While, yes, Facebook needs to test their SDK, app developers need to build resiliency into their apps to ensure third party dependencies don’t break their app. The developer adopted it into their app; it’s their responsibility to do the proper checks in cases like these where Facebook broke things on their end.
- corndoge 6y ago
  Ah yes. If hacker news goes down while you are writing this comment, it is your responsibility to ensure you can still post your comment despite the service breakage.
  - slipheen 6y ago
    Unless your app is designed for browsing Facebook, there's no reason that anything on Facebook's servers should affect if your app works. Many of the apps crashing are using Facebook for a portion of their app, such as one login option. Letting that take down the whole app is insufficiently defensive coding, whether in the app, sdk, or glue between them.

blantonl 6y ago

Bugs happen. Tests miss edge cases. Shit happens. This will be a fart in the wind and lessons will be learned.

emerongi 6y ago
Maybe lessons will be learned, maybe not. The industry seems to have very low standards. Let's let these mistakes happen over and over and let's just patch them as we go. No long-term thinking. Someone should collect all these horrible incidents and start a university course on just plainly what NOT to do.
- blantonl 6y ago
  Some of the very brightest software engineers in the world work at Facebook. As much as I would love to rip on Facebook, I also understand that integrated software development is hard, and I guarantee you that Facebook is "doing it better" than most. Don't sit in an ivory tower and throw stones in glass houses. Can you guarantee that your own SDLC house is in order? Because it might be, right up until a sneaky edge case nips you in the butt, and then suddenly it isn't.
  - mattgreenrocks 6y ago
    Let's avoid lionizing/denigrating here. They're using static initialization and it bit them in the ass again. They should stop using it because it is prone to things like this which result in apps not launching at all. I'd like to see them commit to stop using this method to initialize themselves, and take responsibility for the crashes. Then, IMO, we'd be good. In a scenario like this, I'd hope they'd want to make this bug's recurrence impossible.
    - tdrp 6y ago
      It was horrifying to see how many config changes you have to do to basically beg facebook not to send back data to their server. Most of these were hidden in various corners of the documentation and I still suspect there's crap being sent back without permission. I would love nothing more than to disable the SDK entirely but we have a few hundred thousand users logged into our app via Facebook. Here is our Android config in case anyone's interested: <meta-data android:name="com.facebook.sdk.AutoLogAppEventsEnabled" android:value="false"/> <meta-data android:name="com.facebook.sdk.AdvertiserIDCollectionEnabled" android:value="false"/> <meta-data android:name="com.facebook.sdk.AutoInitEnabled" android:value="false"/>

vsareto 6y ago

>How are we so goddamn inept at writing software? Wow who would have thought no standards, free access to learning, fake-ass meritocracy, and exorbitant profits and market control would have led to such a thing Pick a standard, whether it's a degree program, a cert, or apprenticeships. Hurry up! Just give us a solid target to aim for.

babesh 6y ago

There is more complexity in this. Proximate Issues The app makers cannot easily prevent the crash since it is happening in Facebook provided code. Perhaps the only thing they could do is to code in kill switches. Another possibility would be to modify the Facebook code. That introduces maintainability issues. The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception. Objective-C is an old unsafe language. Ultimate Issues Client-side libraries have a poor history in the industry. They are often written by server engineers who aren’t expert on the platform. For instance, if it was really a static load method that called the server, then this was a poor design decision. Furthermore, weaker server engineers are notorious for breaking backward compatibility. They don’t design nor build in tests to ensure backward compatibility. The backward compatibility must include backward data compatibility. The above issues are indicative of organizational issues. Facebook is a little bit of an outlier. It was notorious for badly written client libraries stretching back to the Facebook platform days. The joke was that their libraries may have been written by interns even though so many companies were dependent on them. The Facebook libraries and the ilk are used for sign on through Facebook and ad attribution. The trade off an app maker needs to make it a whether to run ads on Facebook and whether to support sign in. Facebook is not the only big company with these issues. Google’s equivalent also crashed albeit at a much lower rate due to a much more convoluted issue. Apple introduced the Swift programming language to address the deficiencies in Objective-C. That these companies are running into these issues makes me wonder if they never transitioned to Swift. They are certainly way behind the latest version of the libraries. To summarize, it is a cascade of issues some technical and some organizational rather than a single coding error. These issues span companies.

saagarjha 6y ago
> The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception. > Objective-C is an old unsafe language. This was a safe crash, not an unsafe one. Swift can and would crash the same way.
- babesh 6y ago
  You can’t safely call a method on a nil value in Swift. You need to explicitly force that in your code with an exclamation point to assume that the variable is not nil. Looks like the server passed empty value in JSON which got converted to NSNull in Objective-C. It's a common mistake to assume null is returned instead of NSNull. The Swift equivalent of the JSON library should return nil which you then need to explicitly check for non nilness before using unless you use the unsafe exclamation pint.

ljm 6y ago

The fact that so many popular apps have to pull in a full-blown SDK from Facebook to get support for even a subset of its functionality is insane to me, too. That 'log in with Facebook' button on your app comes with an entire suite of stuff that Facebook can use to slurp up any data it wants.

notJim 6y ago

These histrionics are ridiculous. There has never been an era in software engineering where people didn't write bugs. There has never been an era where people didn't miss bugs. You can scream and moan at the top of your lungs all you want, but shaming and calling people incompetent does not contribute anything. > I will have to go and get a degree Was not aware that having a degree made you immune to writing bugs.

jasonv 6y ago

This is the second time. Recently tested an enterprise marketing app with an SDK ... and we tested if the endpoint wasn't available. Nothing happened -- it worked just fine. Failed quietly and gracefully -- the user would never notice. "We" know how to do this. Facebook hasn't done anything to fix this problem since it happened a few weeks ago. But it's Facebook -- I hope they keep it up. Can't wait for them to AOL themselves.

layoutIfNeeded 6y ago
The problem is not that the endpoint is not available, rather that the endpoint is returning garbage on which the parser chokes. Of course you should always write parsing to be 100% resilient against any kind of garbage input (even malicious), but my point is that your marketing SDK could still be a ticking timebomb.
- jasonv 6y ago
  True, yes. But this is for a financial services mobile app... one of the first concerns was the user experience. The org is well versed in the other back-end security and data issues that also have to be looked at. But if you're Facebook, and you're pushing your SDK as a great solution and it's embedded in high profile apps and you are letting this happen a couple of times, I get to make fun of you. (a dollop of sarcasm, to be sure..)

kobalsky 6y ago

>>> I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess. This would still happen because A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them, B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff. The engineering process was the issue here, and sadly the only way to prevent this is not to make personal barriers of entry higher but to slow down the process with more QA, testing or whatever you can throw at it. In other words, make software more expensive. And keep in mind that this would only give better numbers but it would not solve the issue either (see Boeing 737 MAX).

jfkebwjsbx 6y ago
> A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them Very unlikely. You probably don't realize how tough some professional exams are in some fields. We are talking dedicated months of study here. > B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff Most software out there is just CRUD stuff, not really new stuff...

brink 6y ago

> I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. There's a good chance that the one who wrote this bug had a degree. Surgeries in medicine still get botched despite all of their degrees and barriers. Degrees don't produce good code, good developers produce good code.

mushbino 6y ago
There's no standardized process for producing a "good developer". Every company has a different process and different standards. I think that's a part of the problem and I agree that it's not one academia can solve, but I wonder if there could potentially be another way.

StreamBright 6y ago

I try to avoid software in general as much as possible.

dang 6y ago

> WHAT THE FUCK Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.

marcinzm 6y ago

>I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess. I can almost guarantee that the programmer who wrote this had a degree and passed a very rigorous testing process to get their job at Facebook. Doesn't mean anything when the company tells them to cut corners to iterate faster. Stop blaming engineers for the mistakes of management.

stjohnswarts 6y ago
Yeah it's not like most programmers wouldn't prefer to be writing good software but the "let the user test it" cycle of quick turnaround sprint culture is always going to have this issue.