8 ms·

Bugzilla CVE-2015-4499: All Your Bugs Are Belong to Us

123 points by Chris911 11y ago

fein 11y ago
Incredibly interesting, and incredibly simple. Very cool stuff. I do wonder... Why on earth is domain based privileges ever a good idea? There should be a proper roles system behind the scenes which doesn't grant privileges based on an email domain.
- pilif 11y ago
  There is a proper roles system. But as a shortcut, roles are assigned by domain. It's either this or manually creating/accepting accounts if they need privileges other than the default. In this case, they could get away with the shortcut (if you exclude the MySQL misconfiguration), so that's what they have done.
- geofft 11y ago
  Because the business requirement is "Anyone from this company we are partnering with, which has signed a mutual NDA at the company level, should be able to see these bugs," and "The local Bugzilla admins shouldn't be involved in the hiring processes of that other company," and "No, they don't have a public facing identity management system we can integrate with."
ScottBurson 11y ago
You know, if you're going to use this meme, you should get it right. It should be "all your bug are belong to us" [0]. Yes, it's ungrammatical -- that's the point! [0] https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us
- rubbingalcohol 11y ago
  what you say?
- EpicDavi 11y ago
  I would argue that in the original, "base" was referring to a singular object and was not reduced down from "bases" (plural) to "base" (singular). Therefore, I do not think it is necessary to reduce "bugs" to "bug". I know you are joking but if we are being pedantic here, "all your swarm are belong to us", would be more apt. EDIT: Instead of swarm, a name for a group of bugs (errors) would be appropriate.
  - saint_fiasco 11y ago
    So "All your base" is Engrish for "The entirety of your base", not "All your bases"?
    - psykovsky 11y ago
      Are you guys really trying to "fix" a meme?
  - ben0x539 11y ago
    I don't speak Japanese, but according to whoever translated it for Wikipedia it was really "bases": https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us#Selected_transcript https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us...
    - gizmo686 11y ago
      Translation is an art. Japanese does not have plurals in the same way that English does. It does have an optional pluralizing suffix. However, the fact that it was not used with "base" does not imply that "base" is singular. Looking at the sentence structure, it seems to me that "base" was really intended to be singular (which also fits with the context provided by the script on your wikipedia link). Roughly the sentence is structured as follows: By means of the Federation Army's corporation, as for yall's base, entirely CATS has taken The word I translated into "entirely" could also be translated into "all". However, gramaticly, it still does not modify "base", so I can see no sense in which "all your bases" is a correct, literal translation.
kijin 11y ago
I'm not sure which is worse: not validating your data, or silently truncating invalid data. It seems that the fix [1] was to limit the length of email addresses to 127 characters. That'll do for the time being, but they really should pay more attention to database warnings. MySQL does all sorts of stupid things, but at least it tends to emit warnings when it's being stupid. Catching those warnings and turning them into exceptions should go a long way toward preventing bugs like this. [1] https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=commitdiff;h=9d64d15 https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=commitdif...
- DCoder 11y ago
  MySQL can turn those warnings to exceptions on its own, via sql_mode [1]. New installations of MySQL 5.6 have this behaviour enabled by default on InnoDB tables. [1] http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-default.html http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-d...
- harshal 11y ago
  It looks like the devs are doing exactly the right thing: Fixing the immediate problem quickly on released branches and following up by stricter checking in the development version (which looks like it will need a lot more changes) See: https://bugzilla.mozilla.org/show_bug.cgi?id=1202447#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1202447#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1202509 https://bugzilla.mozilla.org/show_bug.cgi?id=1202509
elchief 11y ago
1. This is mostly MySQL's fault for default silently truncating values longer than the requested length. Postgres and SQL Server don't do this, but that's all I tested 2. Lack of email validation by Bugzilla. Emails are max length 254, though this might not have been official when the code was written. See https://stackoverflow.com/questions/386294/what-is-the-maximum-length-of-a-valid-email-address https://stackoverflow.com/questions/386294/what-is-the-maxim... 3. Consider using an HMAC'd URL for registration email links instead of putting a token in the database, as per https://neosmart.net/blog/2015/using-hmac-signatures-to-avoid-database-writes/ https://neosmart.net/blog/2015/using-hmac-signatures-to-avoi...
- stinos 11y ago
  1. I copletely agree it isn't exactly the best design decision, on the contrary, but the developpers are still too blame as well in my opinion. They chose the datatype so they should have read the documentation, especially the caveats like this, and they should have done something to deal with this. This applies more generally. There are a ton of programming languages, standard libraries and APIs out there with weird/confusing/exceptional/undefined-behavior-invoking types and functions depedning on how they are used so a sane developper should alwyas check this and handle accordingly, even though it might lead to more code to handle edge cases than to the actual expected main code path.
pilif 11y ago
> Will the DB raise an exception? Will it crash? No – It automatically truncates the data so it fits the column size. If you write software, never be "helpful" and try to fix problems with input data you can't handle. You have no idea whether the fix you have in mind is going to be appropriate in all situations and there is no guarantee that your users are going to acknowledge the warning you might (or might not) issue in that case (case in point: MySQL does fire warnings, but many libraries don't even expose them to the user, so even if they wanted to check for warnings after every statement, they couldn't) "Fixing" broken input data is nothing but actually corrupting data because after the fact there is no way to know what the original was, nor is there a way to know whether the "fixed" data was actually fixed or whether it has been that way to begin with. If you are in a position where you can't do something based on the input data, then blow up. That way you help users to avoid ever having to deal with corrupted data (or all their private bugs exposed to the public like what happened here)
- deleted 11y ago
  [deleted]
- bkor 11y ago
  Not really the full story. Bugzilla explicitly disables strict mode (due to another problem). Pretty stupid to have done that! See for details https://bugzilla.mozilla.org/show_bug.cgi?id=321645#c20 https://bugzilla.mozilla.org/show_bug.cgi?id=321645#c20 PS: I wrote the patch to disable strict mode.
  - makomk 11y ago
    If I'm reading that correctly, the problem was that some MySQL versions couldn't actually restore database dumps they created using mysqldump if strict mode was enabled. That's a pretty ugly bug.
    - bkor 11y ago
      If you're on strict mode, TEXT and BLOB types in MySQL 5 don't support a default value. Bugzilla didn't set anything for those fields; relying on MySQL putting in the default value. At that time Bugzilla had loads of places where the database would insert things into the database. Fixing all of those would've been pretty daunting task and IMO MySQL not supporting those defaults anymore is a too invasive (backwards compatibility breaking) change. See http://dev.mysql.com/doc/refman/5.6/en/data-type-defaults.html http://dev.mysql.com/doc/refman/5.6/en/data-type-defaults.ht.... This was around the time that GNOME once again was upgrading its Bugzilla instance (IIRC), plus lots of people were moving to MySQL 4. Fixing this was pretty urgent and assumed that MySQL's behaviour would change back. Because of this MySQL change being backwards incompatible, it also broke restoring backups. This patch wouldn't help with that though (it just changes the setting for the database connection created by Bugzilla; it doesn't change the server setting). It took many years until the Bugzilla code was less of a mess. If the same bug would've been opened nowadays a proper fix could've been made. Back then, urgh.
derf_ 11y ago
It is important to point out that access to security bugs (the ones that contain descriptions of unpatched vulnerabilities) is not automatically granted to all Mozilla employees, so simply having an @mozilla.com e-mail is not enough to get information on how to exploit Firefox. It is enough to get access to potentially sensitive partner information (as can be seen from the posted screenshot), but those are usually a different class of bugs from security bugs. Further, we now classify security bugs by the area of the code they affect, and give engineers access to just the areas for which they are responsible, rather than blanket access to all security bugs. This is a process that began even before the recent attacks, but we have been pushing it more aggressively in their wake.
vog 11y ago
I never understood why people use MySQL for anything where correctness is important. The auto-truncaction is only one of many nasty surprises. Did you know that auto-truncation also kicks in if your columns values are small, but you run a CONCAT on them? That was a nasty surprise when a generated a CSV line on DB side, which by bad luck was truncated immediately before a comma, so the result was correst, just the jast few items were missing. And ever looked at the handling of invalid dates? Not to mention what happens if you forgot to specify a column in your GROUP BY clause, but use it in the SELECT part ... To anyone who is so brave to write their application using that database: Be sure you know all those issues and take care of them in your application. Or make sure you aren't affected by them. Or, switch to a high-quality database! I switched to PostgreSQL a long time ago and never looked back. These people care about exactness, quality and clean extensibility in all directions, even user-defined index structures are possible (GIN, GIST). The problem that PostgreSQL was slower than MySQL disappeared as soon as MySQL became transaction safe with InnoDB, because it was now a fair comparison as PostgreSQL was transaction safe from the very beginning. Also, the PostgreSQL project management is one of the best I have ever seen, meeting schedules, regular commit fests, quality-driven, honest beta releases, and so on.
- rimantas 11y ago
  > I never understood why people use MySQL for anything where > correctness is important. They probaly know how to configure it.
  - vog 11y ago
    This doesn't seem to be any harder than MySQL, it's just a bit different. Also note that PostgreSQL can perform "peer" auth, so you don't have to fiddle with DB passwords unless you really want to access your DB over a network. Install package: apt-get install postgresql-9.4 Become PostgreSQL admin: su - postgres Create a DB user who is allowed to create databases (assuming your app will run as user "someuser"): createuser -d someuser Login as that user: exit su - someuser Create your database: createdb somedb Play with your database: psql somedb CREATE TABLE ...
  - 11y ago
arjunseo 11y ago
They are the ones who are accountable for choosing right individuals for right job. This is done after rigorous scrutiny of the individuals depending on their academic qualification, skills and encounter of the individuals. Expert Packers and Movers Bangalore @ http://www.expert5th.in/packers-and-movers-bangalore/ http://www.expert5th.in/packers-and-movers-bangalore/
arjunseo 11y ago
Though they are not visible to the customers, they are straight contacted by the customers over online or phone, hence such individuals need to have excellent relationships and client managing skills and must be with the abilities of using the modern methods to create use of the same and assistance the client. Expert Packers and Movers in Pune @ http://www.expert5th.in/packers-and-movers-pune/ http://www.expert5th.in/packers-and-movers-pune/
arjunseo 11y ago
They are the ones whom the client would be in contact while stepping into the residence of packers and movers workplace. They are the ones who would be with excellent relationships, client managing skills with excellent overall look and have tolerance to cope with the client. Expert Packers and Movers Mumbai @ http://www.expert5th.in/packers-and-movers-mumbai/ http://www.expert5th.in/packers-and-movers-mumbai/ Expert Packers and Movers Hyderabad @ http://www.expert5th.in/packers-and-movers-hyderabad/ http://www.expert5th.in/packers-and-movers-hyderabad/
tomvangoethem 11y ago
For anyone interested in similar issues: here you can find a report for a vulnerability in Phabricator with exactly the same cause (truncation by MySQL), and pretty much the same result: https://hackerone.com/reports/2224 https://hackerone.com/reports/2224 If Bugzilla would allow non-ASCII characters in the email address, MySQL's truncation behaviour with astral symbols (e.g. 𝌆) would probably have lead to a similar vulnerability as well. (It did so in Phabricator: https://hackerone.com/reports/2233 https://hackerone.com/reports/2233)
bugmen0t 11y ago
The bug impact description is completely false. Mozilla has "corporate confidential" bugs behind the @mozilla.com email check, but everything with a security rating is restricted to specific accounts that have been explicitly vouched for.

123 points by Chris911 11y ago

fein 11y ago

Incredibly interesting, and incredibly simple. Very cool stuff. I do wonder... Why on earth is domain based privileges ever a good idea? There should be a proper roles system behind the scenes which doesn't grant privileges based on an email domain.

pilif 11y ago
There is a proper roles system. But as a shortcut, roles are assigned by domain. It's either this or manually creating/accepting accounts if they need privileges other than the default. In this case, they could get away with the shortcut (if you exclude the MySQL misconfiguration), so that's what they have done.
geofft 11y ago
Because the business requirement is "Anyone from this company we are partnering with, which has signed a mutual NDA at the company level, should be able to see these bugs," and "The local Bugzilla admins shouldn't be involved in the hiring processes of that other company," and "No, they don't have a public facing identity management system we can integrate with."

ScottBurson 11y ago

You know, if you're going to use this meme, you should get it right. It should be "all your bug are belong to us" [0]. Yes, it's ungrammatical -- that's the point! [0] https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us

rubbingalcohol 11y ago
what you say?
EpicDavi 11y ago
I would argue that in the original, "base" was referring to a singular object and was not reduced down from "bases" (plural) to "base" (singular). Therefore, I do not think it is necessary to reduce "bugs" to "bug". I know you are joking but if we are being pedantic here, "all your swarm are belong to us", would be more apt. EDIT: Instead of swarm, a name for a group of bugs (errors) would be appropriate.
- saint_fiasco 11y ago
  So "All your base" is Engrish for "The entirety of your base", not "All your bases"?
  - psykovsky 11y ago
    Are you guys really trying to "fix" a meme?
- ben0x539 11y ago
  I don't speak Japanese, but according to whoever translated it for Wikipedia it was really "bases": https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us#Selected_transcript https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us...
  - gizmo686 11y ago
    Translation is an art. Japanese does not have plurals in the same way that English does. It does have an optional pluralizing suffix. However, the fact that it was not used with "base" does not imply that "base" is singular. Looking at the sentence structure, it seems to me that "base" was really intended to be singular (which also fits with the context provided by the script on your wikipedia link). Roughly the sentence is structured as follows: By means of the Federation Army's corporation, as for yall's base, entirely CATS has taken The word I translated into "entirely" could also be translated into "all". However, gramaticly, it still does not modify "base", so I can see no sense in which "all your bases" is a correct, literal translation.

kijin 11y ago

I'm not sure which is worse: not validating your data, or silently truncating invalid data. It seems that the fix [1] was to limit the length of email addresses to 127 characters. That'll do for the time being, but they really should pay more attention to database warnings. MySQL does all sorts of stupid things, but at least it tends to emit warnings when it's being stupid. Catching those warnings and turning them into exceptions should go a long way toward preventing bugs like this. [1] https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=commitdiff;h=9d64d15 https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=commitdif...

DCoder 11y ago
MySQL can turn those warnings to exceptions on its own, via sql_mode [1]. New installations of MySQL 5.6 have this behaviour enabled by default on InnoDB tables. [1] http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-default.html http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-d...
harshal 11y ago
It looks like the devs are doing exactly the right thing: Fixing the immediate problem quickly on released branches and following up by stricter checking in the development version (which looks like it will need a lot more changes) See: https://bugzilla.mozilla.org/show_bug.cgi?id=1202447#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1202447#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1202509 https://bugzilla.mozilla.org/show_bug.cgi?id=1202509

elchief 11y ago

1. This is mostly MySQL's fault for default silently truncating values longer than the requested length. Postgres and SQL Server don't do this, but that's all I tested 2. Lack of email validation by Bugzilla. Emails are max length 254, though this might not have been official when the code was written. See https://stackoverflow.com/questions/386294/what-is-the-maximum-length-of-a-valid-email-address https://stackoverflow.com/questions/386294/what-is-the-maxim... 3. Consider using an HMAC'd URL for registration email links instead of putting a token in the database, as per https://neosmart.net/blog/2015/using-hmac-signatures-to-avoid-database-writes/ https://neosmart.net/blog/2015/using-hmac-signatures-to-avoi...

stinos 11y ago
1. I copletely agree it isn't exactly the best design decision, on the contrary, but the developpers are still too blame as well in my opinion. They chose the datatype so they should have read the documentation, especially the caveats like this, and they should have done something to deal with this. This applies more generally. There are a ton of programming languages, standard libraries and APIs out there with weird/confusing/exceptional/undefined-behavior-invoking types and functions depedning on how they are used so a sane developper should alwyas check this and handle accordingly, even though it might lead to more code to handle edge cases than to the actual expected main code path.

pilif 11y ago

> Will the DB raise an exception? Will it crash? No – It automatically truncates the data so it fits the column size. If you write software, never be "helpful" and try to fix problems with input data you can't handle. You have no idea whether the fix you have in mind is going to be appropriate in all situations and there is no guarantee that your users are going to acknowledge the warning you might (or might not) issue in that case (case in point: MySQL does fire warnings, but many libraries don't even expose them to the user, so even if they wanted to check for warnings after every statement, they couldn't) "Fixing" broken input data is nothing but actually corrupting data because after the fact there is no way to know what the original was, nor is there a way to know whether the "fixed" data was actually fixed or whether it has been that way to begin with. If you are in a position where you can't do something based on the input data, then blow up. That way you help users to avoid ever having to deal with corrupted data (or all their private bugs exposed to the public like what happened here)

deleted 11y ago
[deleted]
bkor 11y ago
Not really the full story. Bugzilla explicitly disables strict mode (due to another problem). Pretty stupid to have done that! See for details https://bugzilla.mozilla.org/show_bug.cgi?id=321645#c20 https://bugzilla.mozilla.org/show_bug.cgi?id=321645#c20 PS: I wrote the patch to disable strict mode.
- makomk 11y ago
  If I'm reading that correctly, the problem was that some MySQL versions couldn't actually restore database dumps they created using mysqldump if strict mode was enabled. That's a pretty ugly bug.
  - bkor 11y ago
    If you're on strict mode, TEXT and BLOB types in MySQL 5 don't support a default value. Bugzilla didn't set anything for those fields; relying on MySQL putting in the default value. At that time Bugzilla had loads of places where the database would insert things into the database. Fixing all of those would've been pretty daunting task and IMO MySQL not supporting those defaults anymore is a too invasive (backwards compatibility breaking) change. See http://dev.mysql.com/doc/refman/5.6/en/data-type-defaults.html http://dev.mysql.com/doc/refman/5.6/en/data-type-defaults.ht.... This was around the time that GNOME once again was upgrading its Bugzilla instance (IIRC), plus lots of people were moving to MySQL 4. Fixing this was pretty urgent and assumed that MySQL's behaviour would change back. Because of this MySQL change being backwards incompatible, it also broke restoring backups. This patch wouldn't help with that though (it just changes the setting for the database connection created by Bugzilla; it doesn't change the server setting). It took many years until the Bugzilla code was less of a mess. If the same bug would've been opened nowadays a proper fix could've been made. Back then, urgh.

derf_ 11y ago

It is important to point out that access to security bugs (the ones that contain descriptions of unpatched vulnerabilities) is not automatically granted to all Mozilla employees, so simply having an @mozilla.com e-mail is not enough to get information on how to exploit Firefox. It is enough to get access to potentially sensitive partner information (as can be seen from the posted screenshot), but those are usually a different class of bugs from security bugs. Further, we now classify security bugs by the area of the code they affect, and give engineers access to just the areas for which they are responsible, rather than blanket access to all security bugs. This is a process that began even before the recent attacks, but we have been pushing it more aggressively in their wake.

vog 11y ago

I never understood why people use MySQL for anything where correctness is important. The auto-truncaction is only one of many nasty surprises. Did you know that auto-truncation also kicks in if your columns values are small, but you run a CONCAT on them? That was a nasty surprise when a generated a CSV line on DB side, which by bad luck was truncated immediately before a comma, so the result was correst, just the jast few items were missing. And ever looked at the handling of invalid dates? Not to mention what happens if you forgot to specify a column in your GROUP BY clause, but use it in the SELECT part ... To anyone who is so brave to write their application using that database: Be sure you know all those issues and take care of them in your application. Or make sure you aren't affected by them. Or, switch to a high-quality database! I switched to PostgreSQL a long time ago and never looked back. These people care about exactness, quality and clean extensibility in all directions, even user-defined index structures are possible (GIN, GIST). The problem that PostgreSQL was slower than MySQL disappeared as soon as MySQL became transaction safe with InnoDB, because it was now a fair comparison as PostgreSQL was transaction safe from the very beginning. Also, the PostgreSQL project management is one of the best I have ever seen, meeting schedules, regular commit fests, quality-driven, honest beta releases, and so on.

rimantas 11y ago
> I never understood why people use MySQL for anything where > correctness is important. They probaly know how to configure it.
- vog 11y ago
  This doesn't seem to be any harder than MySQL, it's just a bit different. Also note that PostgreSQL can perform "peer" auth, so you don't have to fiddle with DB passwords unless you really want to access your DB over a network. Install package: apt-get install postgresql-9.4 Become PostgreSQL admin: su - postgres Create a DB user who is allowed to create databases (assuming your app will run as user "someuser"): createuser -d someuser Login as that user: exit su - someuser Create your database: createdb somedb Play with your database: psql somedb CREATE TABLE ...
- 11y ago

arjunseo 11y ago

They are the ones who are accountable for choosing right individuals for right job. This is done after rigorous scrutiny of the individuals depending on their academic qualification, skills and encounter of the individuals. Expert Packers and Movers Bangalore @ http://www.expert5th.in/packers-and-movers-bangalore/ http://www.expert5th.in/packers-and-movers-bangalore/

arjunseo 11y ago

Though they are not visible to the customers, they are straight contacted by the customers over online or phone, hence such individuals need to have excellent relationships and client managing skills and must be with the abilities of using the modern methods to create use of the same and assistance the client. Expert Packers and Movers in Pune @ http://www.expert5th.in/packers-and-movers-pune/ http://www.expert5th.in/packers-and-movers-pune/

arjunseo 11y ago

They are the ones whom the client would be in contact while stepping into the residence of packers and movers workplace. They are the ones who would be with excellent relationships, client managing skills with excellent overall look and have tolerance to cope with the client. Expert Packers and Movers Mumbai @ http://www.expert5th.in/packers-and-movers-mumbai/ http://www.expert5th.in/packers-and-movers-mumbai/ Expert Packers and Movers Hyderabad @ http://www.expert5th.in/packers-and-movers-hyderabad/ http://www.expert5th.in/packers-and-movers-hyderabad/

tomvangoethem 11y ago

For anyone interested in similar issues: here you can find a report for a vulnerability in Phabricator with exactly the same cause (truncation by MySQL), and pretty much the same result: https://hackerone.com/reports/2224 https://hackerone.com/reports/2224 If Bugzilla would allow non-ASCII characters in the email address, MySQL's truncation behaviour with astral symbols (e.g. 𝌆) would probably have lead to a similar vulnerability as well. (It did so in Phabricator: https://hackerone.com/reports/2233 https://hackerone.com/reports/2233)

bugmen0t 11y ago

The bug impact description is completely false. Mozilla has "corporate confidential" bugs behind the @mozilla.com email check, but everything with a security rating is restricted to specific accounts that have been explicitly vouched for.