4 ms·

Dorking is the technique of using public search engine indexes to uncover information that is presumed to be private. It has been used to uncover webcams, credi

Dorking is the technique of using public search engine indexes to uncover information that is presumed to be private. It has been used to uncover webcams, credit card numbers, confidential documents, and even spies. The problem is the website administers who are encoding authentication tokens into URL state, not the naive crawlers that find them.

shkkmo 3y ago
It can be OK to put authentication tokens in urls, but those tokens need to (at a bare minimum) have short expirations.
- knome 3y ago
  >It can be OK to put authentication tokens in urls When would this ever be necessary? URL session tokens have been a bad idea ever since they first appeared. The only things even near to auth tokens I can reasonably see stuffed into a URL are password reset and email confirmation tokens sent to email for one time short expiration use. Outside of that, I don't see any reason for it.
  - dylanowen 3y ago
    They're useful for images when you can't use cookies and want the client to easily be able to embed them.
  - albert_e 3y ago
    "presigned" URLs[1] are a pretty standard and recommended way of providing users access to upload/download content to Amazon S3 buckets without needing other forms of authentication like IAM credential pair, or STS token, etc Web Applications do utilize this pattern very frequently But as noted i previous comment these do have short expiry times (configurable) so that there is no permanent or long-term risk on the lines of the OP article [1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
    - vin10 3y ago
      You are right about short expiry times but another catch here is that if pre-signed URLs are being leaked in an automated fashion, these services also keep the downloaded content from these URLs around. I found various such examples where links no longer work, but PDFs downloaded from pre-signed URLs were still stored by scanning services. From https://urlscan.io/blog/2022/07/11/urlscan-pro-product-updates-for-q2-2022/ https://urlscan.io/blog/2022/07/11/urlscan-pro-product-updat... > In the process of scanning websites, urlscan.io will sometimes encounter file downloads triggered by the website. If we are able to successfully download the file, we will store it, hash it and make it available for downloading by our customers.
      - albert_e 3y ago
        Aha. That is an interesting issue indeed.
      - couchand 3y ago
        Indeed, the only valid operation with the magic URL is exchanging the URL-based token with something else (your PDF, a session token, etc.) and then expiring the URL, so by the time the scanner gets around to it the original URL is invalid.
        ta1243 3y ago
        That seems ripe for race condition class problems.
        couchand 3y ago
        If anybody but the intended recipient gets the magic URL first there's something more critical wrong with some assumption in your authentication scheme.
        fragmede 3y ago
        In the war against malware on the web, messenger services like Facebook messenger and google chat have been known to visit links passed through their service. the attacker hijacks an account, and then sends the malware link to all of that users contacts. coming from a trusted source, those contacts visit the link, and they get their account hijacked, and the cycle continues. in order to combat this, the platforms will visit the link to verify it isn't malware. Thus, the platform has the magic url. We trust them not to abuse this privilege, but that's one case where someone other than the intended recipient has the magic url.
layer8 3y ago
I wonder if there would be a way to tag such URLs in a machine-recognizable, but not text-searchable way. (E.g. take every fifth byte in the URL from after the authority part, and have those bytes be a particular form of hash of the remaining bytes.) Meaning that crawlers and tools in TFA would have a standardized way to recognize when a URL is meant to be private, and thus could filter them out from public searches. Of course, being recognizable in that way may add new risks.
- loa_in_ 3y ago
  We already have robots.txt in theory.
  - layer8 3y ago
    I didn’t think robots.txt would be applicable to URLs being copied around, but actually it might be, good point. Though again, collecting that robots.txt information could make it easier to search for such URLs.
- internetter 3y ago
  We already have a solution to this. It’s called not including authentication information within URLs Even if search engines knew to include it, would every insecure place a user put a link know it? Bad actors with their own indexes certainly wouldn’t care
  - layer8 3y ago
    How do you implement password-reset links otherwise? I mean, those should be short-lived, but still.
    - 1231232131231 3y ago
      A short-lived link that's locked down to their user agent/IP would work as well.
    - andersa 3y ago
FrustratedMonky 3y ago
"public search engine indexes" Then it should be the search engine at fault. If you leave your house unlocked is one thing. If there is a company trying everyone's doors, then posting a sign in the yard "this house is unlocked", has to account for something.
- pixl97 3y ago
  You've been appropriately downvoted for a terrible take. Imagine if you left your house unlocked it would be broken into seconds later. Even worse, the people that broke into it live in a different country with no extradition law and you'd never figure out who they are anyway. In this case your insurance company would tell you lock your damned doors and the police may even charge you under public nuisance laws.
  - FrustratedMonky 3y ago
    Yeah, it is a terrible take. It's a bad situation. Just like charging people for a crime for accessing private material, simply by browsing a public URL. Maybe Better take: It is like someone being charged for breaking and entering, simply by looking at a house from the street, when the door was left open. Your guilty by simply looking, and seeing inside. But you were just walking by, you saw inside before realizing it was a crime, now your guilty. If you are going to charge people for accessing private sites, potentially by accident, by simply being provided a public URL from a search engine. Then shouldn't the search engine have some culpability? Or. Better. Change the law so the onus is on the site to protect itself.
  - FrustratedMonky 3y ago
    "" Imagine a journalist finds a folder on a park bench, opens it, and sees a telephone number inside. She dials the number. A famous rapper answers and spews a racist rant. If no one gave her permission to open the folder and the rapper’s telephone number was unlisted, should the reporter go to jail for publishing what she heard? If that sounds ridiculous, it’s because it is. And yet, add in a computer and the Internet, and that’s basically what a newly unsealed federal indictment accuses Florida journalist Tim Burke of doing when he found and disseminated outtakes of Tucker Carlson’s Fox News interview with Ye, the artist formerly known as Kanye West, going on the first of many antisemitic diatribes."" https://arstechnica.com/tech-policy/2024/03/charges-against-journalist-tim-burke-are-a-hack-job/ https://arstechnica.com/tech-policy/2024/03/charges-against-... "According to Burke, the video of Carlson’s interview with Ye was streamed via a publicly available, unencrypted URL that anyone could access by typing the address into your browser."
thayne 3y ago
That isn't an inherent problem with having a secret in the url. The problem is the url was leaked somewhere where it could get indexed. And sometimes it isn't practical to require a POST request or a cookie. And the risk of a url leaking can be greatly mitigated if the url is only valid for a short period of time.
- ta1243 3y ago
  > That isn't an inherent problem with having a secret in the url. The problem is the url was leaked somewhere where it could get indexed. Technically you're right -- after all sending an authentication as a separate header doesn't make any difference. GET /endpoint/?Auth=token or GET /endpoint Auth: token Sends the same data over the wire. However software treats URLs differently to headers. They sit in browser histories, server logs, get parsed by MITM firewalls, mined by browser extensions, etc using https://user:pass@site.com/endpoint https://user:pass@site.com/endpoint or https://auth:token@site.com/endpoint https://auth:token@site.com/endpoint Would be better than https://site.com/endpoint/user/pass https://site.com/endpoint/user/pass or https://site.com/endpoint/?auth=token https://site.com/endpoint/?auth=token As the former is less likely to be stored, either on the client or on the server. I don't do front end (or backend authentication -- I just rely on x509 client certs or oidc and the web server passes the validated username)
  - paulgb 3y ago
    For better or worse, basic auth in the URL isn't really an option any more, (e.g. see https://stackoverflow.com/a/57193064 https://stackoverflow.com/a/57193064). I think the issue was that it reveals the secret to anyone who can see the URL bar, but the alternative we got still has that problem and also has the problem that the secret is no longer separable from the resource identifier.
    - thayne 3y ago
      The browser could hide the secret after it is entered.

shkkmo 3y ago

It can be OK to put authentication tokens in urls, but those tokens need to (at a bare minimum) have short expirations.

knome 3y ago
>It can be OK to put authentication tokens in urls When would this ever be necessary? URL session tokens have been a bad idea ever since they first appeared. The only things even near to auth tokens I can reasonably see stuffed into a URL are password reset and email confirmation tokens sent to email for one time short expiration use. Outside of that, I don't see any reason for it.
- dylanowen 3y ago
  They're useful for images when you can't use cookies and want the client to easily be able to embed them.
- albert_e 3y ago
  "presigned" URLs[1] are a pretty standard and recommended way of providing users access to upload/download content to Amazon S3 buckets without needing other forms of authentication like IAM credential pair, or STS token, etc Web Applications do utilize this pattern very frequently But as noted i previous comment these do have short expiry times (configurable) so that there is no permanent or long-term risk on the lines of the OP article [1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
  - vin10 3y ago
    You are right about short expiry times but another catch here is that if pre-signed URLs are being leaked in an automated fashion, these services also keep the downloaded content from these URLs around. I found various such examples where links no longer work, but PDFs downloaded from pre-signed URLs were still stored by scanning services. From https://urlscan.io/blog/2022/07/11/urlscan-pro-product-updates-for-q2-2022/ https://urlscan.io/blog/2022/07/11/urlscan-pro-product-updat... > In the process of scanning websites, urlscan.io will sometimes encounter file downloads triggered by the website. If we are able to successfully download the file, we will store it, hash it and make it available for downloading by our customers.
    - albert_e 3y ago
      Aha. That is an interesting issue indeed.
    - couchand 3y ago
      Indeed, the only valid operation with the magic URL is exchanging the URL-based token with something else (your PDF, a session token, etc.) and then expiring the URL, so by the time the scanner gets around to it the original URL is invalid.
      - ta1243 3y ago
        That seems ripe for race condition class problems.
        couchand 3y ago
        If anybody but the intended recipient gets the magic URL first there's something more critical wrong with some assumption in your authentication scheme.
        fragmede 3y ago
        In the war against malware on the web, messenger services like Facebook messenger and google chat have been known to visit links passed through their service. the attacker hijacks an account, and then sends the malware link to all of that users contacts. coming from a trusted source, those contacts visit the link, and they get their account hijacked, and the cycle continues. in order to combat this, the platforms will visit the link to verify it isn't malware. Thus, the platform has the magic url. We trust them not to abuse this privilege, but that's one case where someone other than the intended recipient has the magic url.

layer8 3y ago

I wonder if there would be a way to tag such URLs in a machine-recognizable, but not text-searchable way. (E.g. take every fifth byte in the URL from after the authority part, and have those bytes be a particular form of hash of the remaining bytes.) Meaning that crawlers and tools in TFA would have a standardized way to recognize when a URL is meant to be private, and thus could filter them out from public searches. Of course, being recognizable in that way may add new risks.

loa_in_ 3y ago
We already have robots.txt in theory.
- layer8 3y ago
  I didn’t think robots.txt would be applicable to URLs being copied around, but actually it might be, good point. Though again, collecting that robots.txt information could make it easier to search for such URLs.
internetter 3y ago
We already have a solution to this. It’s called not including authentication information within URLs Even if search engines knew to include it, would every insecure place a user put a link know it? Bad actors with their own indexes certainly wouldn’t care
- layer8 3y ago
  How do you implement password-reset links otherwise? I mean, those should be short-lived, but still.
  - 1231232131231 3y ago
    A short-lived link that's locked down to their user agent/IP would work as well.
  - andersa 3y ago

FrustratedMonky 3y ago

"public search engine indexes" Then it should be the search engine at fault. If you leave your house unlocked is one thing. If there is a company trying everyone's doors, then posting a sign in the yard "this house is unlocked", has to account for something.

pixl97 3y ago
You've been appropriately downvoted for a terrible take. Imagine if you left your house unlocked it would be broken into seconds later. Even worse, the people that broke into it live in a different country with no extradition law and you'd never figure out who they are anyway. In this case your insurance company would tell you lock your damned doors and the police may even charge you under public nuisance laws.
- FrustratedMonky 3y ago
  Yeah, it is a terrible take. It's a bad situation. Just like charging people for a crime for accessing private material, simply by browsing a public URL. Maybe Better take: It is like someone being charged for breaking and entering, simply by looking at a house from the street, when the door was left open. Your guilty by simply looking, and seeing inside. But you were just walking by, you saw inside before realizing it was a crime, now your guilty. If you are going to charge people for accessing private sites, potentially by accident, by simply being provided a public URL from a search engine. Then shouldn't the search engine have some culpability? Or. Better. Change the law so the onus is on the site to protect itself.
- FrustratedMonky 3y ago
  "" Imagine a journalist finds a folder on a park bench, opens it, and sees a telephone number inside. She dials the number. A famous rapper answers and spews a racist rant. If no one gave her permission to open the folder and the rapper’s telephone number was unlisted, should the reporter go to jail for publishing what she heard? If that sounds ridiculous, it’s because it is. And yet, add in a computer and the Internet, and that’s basically what a newly unsealed federal indictment accuses Florida journalist Tim Burke of doing when he found and disseminated outtakes of Tucker Carlson’s Fox News interview with Ye, the artist formerly known as Kanye West, going on the first of many antisemitic diatribes."" https://arstechnica.com/tech-policy/2024/03/charges-against-journalist-tim-burke-are-a-hack-job/ https://arstechnica.com/tech-policy/2024/03/charges-against-... "According to Burke, the video of Carlson’s interview with Ye was streamed via a publicly available, unencrypted URL that anyone could access by typing the address into your browser."

thayne 3y ago

That isn't an inherent problem with having a secret in the url. The problem is the url was leaked somewhere where it could get indexed. And sometimes it isn't practical to require a POST request or a cookie. And the risk of a url leaking can be greatly mitigated if the url is only valid for a short period of time.

ta1243 3y ago
> That isn't an inherent problem with having a secret in the url. The problem is the url was leaked somewhere where it could get indexed. Technically you're right -- after all sending an authentication as a separate header doesn't make any difference. GET /endpoint/?Auth=token or GET /endpoint Auth: token Sends the same data over the wire. However software treats URLs differently to headers. They sit in browser histories, server logs, get parsed by MITM firewalls, mined by browser extensions, etc using https://user:pass@site.com/endpoint https://user:pass@site.com/endpoint or https://auth:token@site.com/endpoint https://auth:token@site.com/endpoint Would be better than https://site.com/endpoint/user/pass https://site.com/endpoint/user/pass or https://site.com/endpoint/?auth=token https://site.com/endpoint/?auth=token As the former is less likely to be stored, either on the client or on the server. I don't do front end (or backend authentication -- I just rely on x509 client certs or oidc and the web server passes the validated username)
- paulgb 3y ago
  For better or worse, basic auth in the URL isn't really an option any more, (e.g. see https://stackoverflow.com/a/57193064 https://stackoverflow.com/a/57193064). I think the issue was that it reveals the secret to anyone who can see the URL bar, but the alternative we got still has that problem and also has the problem that the secret is no longer separable from the resource identifier.
  - thayne 3y ago
    The browser could hide the secret after it is entered.