5 ms·

Yet another security vulnerability caused by: 1. Using sequentially incremented integer sequences as object IDs, and 2. Failing to protect sensitive data usin

Yet another security vulnerability caused by: 1. Using sequentially incremented integer sequences as object IDs, and 2. Failing to protect sensitive data using some kind of authentication and authorization check. This is becoming a trend with data breaches. Several of Krebs' other reports on behalf of security researchers were originally identified by (trivially) walking across object IDs on public URLs. My cynical take is that Krebs couldn't go public before this afternoon because First American wanted it to hit the news at an opportune time, then get ahead of it with their own messaging. Krebs got in touch with First American on Monday May 19th. The story is only just breaking now on a Friday afternoon at 5 pm; markets are conveniently closed for the weekend. I expect them to issue a hollow PR statement about valuing security despite being unable to act on security reports until an investigative journalist threatens to go public.

eganist 7y ago
(It's apparent that my initial reply didn't resonate, so I've made substantial edits to my reply for clarity's sake. If you've read it once, give it another read; it's from the angle of an organization with much in the way of legacy impairment.) > Yet another security vulnerability caused by... I mean, yes, but these are also some of the easiest vulnerabilities to miss even with out-of-the-box static analysis (code scanning and data analysis), automated dynamic analysis (pentests [edit to clarify for tptacek: automated pentests]), and a basic code review process. They're usually identified in live environments during manual penetration tests or, in more security-mature environments, with custom static analysis checks and custom linting rules. As for best-case prevention: accomplished generally architecturally, e.g. language/framework decisions that enforce secure coding practices by design, or implementing certain patterns in development which whisks away some of the more risky coding decisions from engineers who may not be qualified to be making them, such as mandating authn/z and limiting exceptions only to roles and change processes qualified to make them. Checks including linting for specific privacy defects (direct object referencing using sensitive data or iterative identifiers as opposed to hashes/guids/etc) can help with catching them during development, and as you might've guessed, such checks tend to be custom for a given environment rather than out of the box. I distinctly recall a card issuer whose name starts with a C in the United States having an http endpoint which allowed for enumerating account details by iterating full PANs (16 digit card numbers)... around a decade ago. Here we are today, and you're seeing the same bugs continue to arise. Mitigation options in organizations with immature security practices typically rule out remediation simply because their existence might not be known, and practices traditionally reserved for defense-in-depth may need to be relied-upon instead (think monitoring web requests for anomalous behaviors and blocking traffic when detected) rather than trusting that one can fix all the defects, and even then you'll still lose a few records... but that might be the only solution available to you as a CTO, CIO, or CISO simply because of resource constraints and bureaucracy in an entrenched org e.g. in the financial or insurance space. -- tl;dr: these defects are among the harder ones to catch for legacy applications especially in environments with weaker security postures, and they're as old as time. What I'm saying is that as much as we can call companies out for making these mistakes in hindsight, their existence in larger legacy systems is to some extent inevitable and must be managed in other ways.
- tptacek 7y ago
  There are no effective static source code security analyzers. Static analyzers aren't a bad thing to add to a CI pipeline, because why not, but anyone depending on static analysis is playing to lose. This is absolutely not the kind of vulnerability that pentests tend to miss; rather, they're the first thing pentesters check for. You can miss bugs like this when they're in obscure backend features and your client or team didn't document the project adequately --- though you still shouldn't, and that's part of the point of getting an assessment, to find stuff like that --- but you generally don't miss them in an assessment where the bug is literally "edit a number in a URL". Web scanning tools will miss findings like this. But, regarding web scanners: see static source code security analyzers. As for code review: a competently constructed application shouldn't be relying on developers to catch every possible instance where numeric ids are used individually. In modern web frameworks, it should be obvious when you're looking an ID up without doing an authorization check; for instance, in a Rails or Django app, you can simply regex for lookups coming off the ORM class rather than the appropriate association instance. In sum: I dispute much of this analysis. People do miss things, even when they're things they shouldn't miss. Put 3 different test teams on the same application and you will get 3 overlapping but distinctive sets of vulnerabilities back. But this is not an instance of the kind of vulnerability that is hard to catch. see below
  - eganist 7y ago
    > This is absolutely not the kind of vulnerability that pentests tend to miss You're right; they don't. Which is why I called out automated dynamic analysis. I.e. the web scanning tools which you subsequently mentioned: > Web scanning tools will miss findings like this. --- > As for code review: a competently constructed application shouldn't be relying on developers to catch every possible instance where numeric ids are used individually. In modern web frameworks, it should be obvious when you're looking an ID up without doing an authorization check; for instance, in a Rails or Django app, you can simply regex for lookups coming off the ORM class rather than the appropriate association instance. Right, which I also stated: > As for best-case prevention: accomplished generally architecturally, e.g. language/framework decisions that enforce secure coding practices by design, or implementing certain patterns in development which whisks away some of the more risky coding decisions from engineers who may not be qualified to be making them, such as mandating authn/z and limiting exceptions only to roles and change processes qualified to make them. Checks including linting for specific privacy defects (direct object referencing using sensitive data or iterative identifiers as opposed to hashes/guids/etc) can help with catching them during development, and as you might've guessed, such checks tend to be custom for a given environment rather than out of the box.
mattmanser 7y ago
I once made an app not using sequential integers as object ids, as you suggest. It was an absolute nightmare. Maintenance was a nightmare, you're constantly having to generate or replicate these things that add an extra layer of complexity to everything, and almost always unnecessarily. It's also extremely bad for db performance, causes massive page fragmentation, indexes become useless almost straight after rebuilding them, etc. For almost everything, sequential int IDs are fine. It's the things you expose to the users that you need to be careful with, and then don't use the primary key to access them, add another unique key to them, but keep the id in there for the db to use and for your own use. My lesson was to go back to always using int ids, and on a few objects have a separate unique key column to expose to users for sensitive stuff.
- dvlsg 7y ago
  You can generate uuids that play nicer with database storage / indexing. NEWSEQUENTIALID() in MSSQL, for example. The keys will be easier to guess again, but if all you have to do is guess a primary key to get access to the underlying data, something else isn't right anyways.
  - mattmanser 7y ago
    I know, but as they're easier to guess, what's the point? Might as well just go back to ints.
  - prophesi 7y ago
    I think this gets to the crux of the issue. It's not about using hard-to-guess UUIDs[0], but restricting access to the underlying data[1]. [0] https://en.m.wikipedia.org/wiki/Security_through_obscurity https://en.m.wikipedia.org/wiki/Security_through_obscurity [1] https://en.m.wikipedia.org/wiki/Access_control https://en.m.wikipedia.org/wiki/Access_control
    - cm2187 7y ago
      It's not really security through obscurity. In these case I understand the ids where related to data that the company was making available to users through email links. A cryptographically secure 128bit UUID is impossible to guess, no more than a cryptographic access token. Now of course, you would probably rather want to have an authentication scheme on top of that, but that comes at a support cost in term of customers loosing their passwords, locking themselves out of their account, etc. And it is not clear you have increased security as people re-use passwords. Then of course there is the issue that email is for the most part un-encrypted (or encrypted without validating certificates).
scarface74 7y ago
There is nothing wrong with using sequential ids in and of themselves. The typical web app has the concept of a validated user session per request. How hard is it really to Select ... From Documents where documentid = ? and userid = ? So even if the user does a GET /Document/{id+1} No documents would be returned. Every web framework that I am aware of let’s you add one piece of middleware that validates a user session and won’t even route to the request if the user isn’t validated.
- birken 7y ago
  Maybe not "wrong", but there are some very obvious downsides to exposing sequential IDs vs a randomized token: - It exposes the count you have of a particular item - It exposes your growth rate of those items - If a developer accidentally breaks your authentication (or somebody hacks it), it becomes trivially easy to download all your items very quickly And it isn't like using a randomized token is hard. In the most common implementation, it is just one additional column that gets filled with a random string and an index on the column.
  - scarface74 7y ago
    In that simple scenario. What are some ways that a hacker could break your front end API to allow it to serve requests for multiple users without having access to multiple account logins? I understand that they could possibly get access to your database but that’s a different threat. If they could somehow change your code, all hope is already lost. But I do agree with it does allow someone to determine rate of growth which would be valuable more from a business intelligence side than a privacy violation. The larger issue is that a developer forgets to add the “and userid = ?” I guess the work around for that is to have a database that ties user authentication to records in the table/object store directly like DynamoDB or S3.
    - pointytrees 7y ago
      In my experience, many tables don't have a userid on the table that would be associated with the user. It would be a table join or two or three away. So the developer may think it is safe to say select value from stock positions left join account on account.id = stock position.id left join user_accounts on user_accounts.accountid == account.id left join users on user_accounts.userid == user.id where user.id == session.userid. Safe right? We checked userid. But then clicking on the position to drill in on the position data, they just select * from stock_position where stock_position.id = params.stock_id... there's no "and stock_position.userid" on that table, and the developer might be too lazy to spin up the entire join again especially if you don't need account data for this view. Whoops, suddenly a vulnerable page query. I imagine there are other ways to screw up. Like insecure cookies, and just checking cookie.userid, ah yes, you're the right user. Whoops, didn't realize cookies could be spoofed.

eganist 7y ago

(It's apparent that my initial reply didn't resonate, so I've made substantial edits to my reply for clarity's sake. If you've read it once, give it another read; it's from the angle of an organization with much in the way of legacy impairment.) > Yet another security vulnerability caused by... I mean, yes, but these are also some of the easiest vulnerabilities to miss even with out-of-the-box static analysis (code scanning and data analysis), automated dynamic analysis (pentests [edit to clarify for tptacek: automated pentests]), and a basic code review process. They're usually identified in live environments during manual penetration tests or, in more security-mature environments, with custom static analysis checks and custom linting rules. As for best-case prevention: accomplished generally architecturally, e.g. language/framework decisions that enforce secure coding practices by design, or implementing certain patterns in development which whisks away some of the more risky coding decisions from engineers who may not be qualified to be making them, such as mandating authn/z and limiting exceptions only to roles and change processes qualified to make them. Checks including linting for specific privacy defects (direct object referencing using sensitive data or iterative identifiers as opposed to hashes/guids/etc) can help with catching them during development, and as you might've guessed, such checks tend to be custom for a given environment rather than out of the box. I distinctly recall a card issuer whose name starts with a C in the United States having an http endpoint which allowed for enumerating account details by iterating full PANs (16 digit card numbers)... around a decade ago. Here we are today, and you're seeing the same bugs continue to arise. Mitigation options in organizations with immature security practices typically rule out remediation simply because their existence might not be known, and practices traditionally reserved for defense-in-depth may need to be relied-upon instead (think monitoring web requests for anomalous behaviors and blocking traffic when detected) rather than trusting that one can fix all the defects, and even then you'll still lose a few records... but that might be the only solution available to you as a CTO, CIO, or CISO simply because of resource constraints and bureaucracy in an entrenched org e.g. in the financial or insurance space. -- tl;dr: these defects are among the harder ones to catch for legacy applications especially in environments with weaker security postures, and they're as old as time. What I'm saying is that as much as we can call companies out for making these mistakes in hindsight, their existence in larger legacy systems is to some extent inevitable and must be managed in other ways.

tptacek 7y ago
There are no effective static source code security analyzers. Static analyzers aren't a bad thing to add to a CI pipeline, because why not, but anyone depending on static analysis is playing to lose. This is absolutely not the kind of vulnerability that pentests tend to miss; rather, they're the first thing pentesters check for. You can miss bugs like this when they're in obscure backend features and your client or team didn't document the project adequately --- though you still shouldn't, and that's part of the point of getting an assessment, to find stuff like that --- but you generally don't miss them in an assessment where the bug is literally "edit a number in a URL". Web scanning tools will miss findings like this. But, regarding web scanners: see static source code security analyzers. As for code review: a competently constructed application shouldn't be relying on developers to catch every possible instance where numeric ids are used individually. In modern web frameworks, it should be obvious when you're looking an ID up without doing an authorization check; for instance, in a Rails or Django app, you can simply regex for lookups coming off the ORM class rather than the appropriate association instance. In sum: I dispute much of this analysis. People do miss things, even when they're things they shouldn't miss. Put 3 different test teams on the same application and you will get 3 overlapping but distinctive sets of vulnerabilities back. But this is not an instance of the kind of vulnerability that is hard to catch. see below
- eganist 7y ago
  > This is absolutely not the kind of vulnerability that pentests tend to miss You're right; they don't. Which is why I called out automated dynamic analysis. I.e. the web scanning tools which you subsequently mentioned: > Web scanning tools will miss findings like this. --- > As for code review: a competently constructed application shouldn't be relying on developers to catch every possible instance where numeric ids are used individually. In modern web frameworks, it should be obvious when you're looking an ID up without doing an authorization check; for instance, in a Rails or Django app, you can simply regex for lookups coming off the ORM class rather than the appropriate association instance. Right, which I also stated: > As for best-case prevention: accomplished generally architecturally, e.g. language/framework decisions that enforce secure coding practices by design, or implementing certain patterns in development which whisks away some of the more risky coding decisions from engineers who may not be qualified to be making them, such as mandating authn/z and limiting exceptions only to roles and change processes qualified to make them. Checks including linting for specific privacy defects (direct object referencing using sensitive data or iterative identifiers as opposed to hashes/guids/etc) can help with catching them during development, and as you might've guessed, such checks tend to be custom for a given environment rather than out of the box.

mattmanser 7y ago

I once made an app not using sequential integers as object ids, as you suggest. It was an absolute nightmare. Maintenance was a nightmare, you're constantly having to generate or replicate these things that add an extra layer of complexity to everything, and almost always unnecessarily. It's also extremely bad for db performance, causes massive page fragmentation, indexes become useless almost straight after rebuilding them, etc. For almost everything, sequential int IDs are fine. It's the things you expose to the users that you need to be careful with, and then don't use the primary key to access them, add another unique key to them, but keep the id in there for the db to use and for your own use. My lesson was to go back to always using int ids, and on a few objects have a separate unique key column to expose to users for sensitive stuff.

dvlsg 7y ago
You can generate uuids that play nicer with database storage / indexing. NEWSEQUENTIALID() in MSSQL, for example. The keys will be easier to guess again, but if all you have to do is guess a primary key to get access to the underlying data, something else isn't right anyways.
- mattmanser 7y ago
  I know, but as they're easier to guess, what's the point? Might as well just go back to ints.
- prophesi 7y ago
  I think this gets to the crux of the issue. It's not about using hard-to-guess UUIDs[0], but restricting access to the underlying data[1]. [0] https://en.m.wikipedia.org/wiki/Security_through_obscurity https://en.m.wikipedia.org/wiki/Security_through_obscurity [1] https://en.m.wikipedia.org/wiki/Access_control https://en.m.wikipedia.org/wiki/Access_control
  - cm2187 7y ago
    It's not really security through obscurity. In these case I understand the ids where related to data that the company was making available to users through email links. A cryptographically secure 128bit UUID is impossible to guess, no more than a cryptographic access token. Now of course, you would probably rather want to have an authentication scheme on top of that, but that comes at a support cost in term of customers loosing their passwords, locking themselves out of their account, etc. And it is not clear you have increased security as people re-use passwords. Then of course there is the issue that email is for the most part un-encrypted (or encrypted without validating certificates).

scarface74 7y ago

There is nothing wrong with using sequential ids in and of themselves. The typical web app has the concept of a validated user session per request. How hard is it really to Select ... From Documents where documentid = ? and userid = ? So even if the user does a GET /Document/{id+1} No documents would be returned. Every web framework that I am aware of let’s you add one piece of middleware that validates a user session and won’t even route to the request if the user isn’t validated.

birken 7y ago
Maybe not "wrong", but there are some very obvious downsides to exposing sequential IDs vs a randomized token: - It exposes the count you have of a particular item - It exposes your growth rate of those items - If a developer accidentally breaks your authentication (or somebody hacks it), it becomes trivially easy to download all your items very quickly And it isn't like using a randomized token is hard. In the most common implementation, it is just one additional column that gets filled with a random string and an index on the column.
- scarface74 7y ago
  In that simple scenario. What are some ways that a hacker could break your front end API to allow it to serve requests for multiple users without having access to multiple account logins? I understand that they could possibly get access to your database but that’s a different threat. If they could somehow change your code, all hope is already lost. But I do agree with it does allow someone to determine rate of growth which would be valuable more from a business intelligence side than a privacy violation. The larger issue is that a developer forgets to add the “and userid = ?” I guess the work around for that is to have a database that ties user authentication to records in the table/object store directly like DynamoDB or S3.
  - pointytrees 7y ago
    In my experience, many tables don't have a userid on the table that would be associated with the user. It would be a table join or two or three away. So the developer may think it is safe to say select value from stock positions left join account on account.id = stock position.id left join user_accounts on user_accounts.accountid == account.id left join users on user_accounts.userid == user.id where user.id == session.userid. Safe right? We checked userid. But then clicking on the position to drill in on the position data, they just select * from stock_position where stock_position.id = params.stock_id... there's no "and stock_position.userid" on that table, and the developer might be too lazy to spin up the entire join again especially if you don't need account data for this view. Whoops, suddenly a vulnerable page query. I imagine there are other ways to screw up. Like insecure cookies, and just checking cookie.userid, ah yes, you're the right user. Whoops, didn't realize cookies could be spoofed.