6 ms·

If the user submits a GET request to DDG from the commandline with Javascript disabled, then all results links are prefixed with the address of a DDG server, e

by 3xblah 8y ago

If the user submits a GET request to DDG from the commandline with Javascript disabled, then all results links are prefixed with the address of a DDG server, e.g., https://example.com https://example.com becomes https://duckduckgo.com/l/uddg=https%3A%2F%2Fexample.com https://duckduckgo.com/l/uddg=https%3A%2F%2Fexample.com Whatever the reasons may be for this decision, it allows DDG to track all links that the user follows from the results page. That is no better than Google. It is not a privacy improvement in that regard. If the user submits the request via POST, then the results are not prefixed. For example, if the request is submitted from an html form with method set to "post". I submit that allowing users to GET (clean) results without using Javascript would be a privacy improvement over Google. Not to mention that it would please many users who get frustrated when trying to copy links from a Google results page only to find they have been prefixed and loaded with Google url parameters. Users deserve clean result links. To see the difference between prefixed and non-prefixed result links: curl -o a.htm https://duckduckgo.com/lite/?q=example.com grep -m1 .=result-link a.htm curl -o b.htm -d "q=example.com" https://duckduckgo.com/lite/ grep -m1 .=result-link b.htm

jolmg 8y ago
> Whatever the reasons may be for this decision, it allows DDG to track all links that the user follows from the results page. That is no better than Google. It is not a privacy improvement in that regard. It's meant so that the sites you click on can't see your search queries via the Referer HTTP header. You can turn that off in DDG's settings[1] under the privacy tab. It's the "Redirect" switch. They explain more about it[2]. Basically, it's a privacy choice they give you between preventing sites you visit from seeing your search queries or preventing DDG from seeing what you clicked on. If you want, you can prevent both by enabling their settings option to use POST instead of GET in their search. Then you can disable the redirect and the sites won't be able to see your search query from Referer, because it would have been passed through the request body instead of URL parameters. However, that's going to prevent you from being able to have your searches in your browser history. EDIT: I now note why you mentioned javascript and why the Referer option talks about being needed on old browsers. It seems that when javascript is enabled, they do something else to ensure that Referer doesn't leak search queries without passing info on what you clicked on to DDG. That means that, by default on a typical browser, both leaks are prevented. That's pretty neat. [1] https://duckduckgo.com/settings# https://duckduckgo.com/settings# [2] https://duck.co/help/results/rduckduckgocom https://duck.co/help/results/rduckduckgocom
- 3xblah 8y ago
  Here is another perspective from a non-developer user on those reasons and the resulting tradeoff (of allowing DDG to trivially track which result links are followed): 1. Privacy conscious users can control whether a Referer header is sent and if so, the content of that header. They might choose to use a browser that does not make it easy to control the Referer header. But DDG does not account for those privacy conscious users who may be using browsers that do allow this control. (Bias discosure: I use one and not for privacy reasons.) The DDG decision seems to encourage users to choose browsers that have privacy-unfriendly default settings or that do not make it easy or even possible to control what if any Referer is sent. At the very least, it tries to resolve an issue that is a clearly client-side one and is best resolved there. 2. Correct me if I am wrong but storing "settings" with DDG requires a cookie and/or Javascript to be enabled. If this is true, then again this is not an improvement in privacy. Privacy users may prefer to keep these browser "features" disabled wherever possible. Users can of course control whether they send GET or POST by using a client such as curl or the like, or sending requests from a local copy of a DDG html search page with the method changed from GET to POST. One might wonder why DDG cannot provide an alternative search page with its form method set to POST. Users can then choose which search page they prefer. No cookie needed. Instead it sets the default method on its search pages to GET (perhaps leaking the query in the URL). How many users will bother to change the default? A minority of them, no doubt. There is an opportunity to let users turn off cookies and Javascript and let them perform searches using POST, the method usuaully advocated by those advancing internet privacy. I cannot see that it would cost DDG anything substantial to provide it, except not promoting GET as a method by default, not requiring users who want to change defaults to enable cookies or javascript to "keep state" and reliquinshing the ability to easily track which result links users follow. To reiterate, control over the Referer header is easily done by the http client. Clients that do not allow easy user control over this are not improving internet privacy. In my experience sending Referer headers is almost never necessary. It is very rare that I find a site that requires a Referer header, let alone a specific one. Sending gratuitous information seems like a clear departure from the internet privacy playbook. Then again, I am not a privacy expert and perhaps the experts have decided there must be a carve-out for the Referer header and users should err on the side of sending too much information rather than too little. The only reason I am aware of this stuff is because I use the web heavily without javascript or cookies and I take an active role in what headers I send. I generally do not do this for privacy, but for speed and control. I can strip out the prefix client-side if I must use GET.
rlue 8y ago
Here it is with piping so you don't have to litter your working directory: curl -s https://duckduckgo.com/lite/?q=example.com | grep -m1 "'result-link'" curl -sd "q=example.com" https://duckduckgo.com/lite/ | grep -m1 "'result-link'"
- 3xblah 8y ago
  Then you cannot do diff a.htm b.htm Send the output to /tmp or some directory mounted as tmpfs. My working directory is always mounted tmpfs. This is how I keep things fast and clean.
- agumonkey 8y ago
  And for xpath lovers curl -sd "q=example.com" https://duckduckgo.com/lite/ | xmllint -html -xpath '//a[@class = "result-link"]/@href' - curl -s https://duckduckgo.com/lite/?q=example.com | xmllint -html -xpath '//a[@class = "result-link"]/@href' -
aphextron 8y ago
>Whatever the reasons may be for this decision, it allows DDG to track all links that the user follows from the results page. That is no better than Google. It is not a privacy improvement in that regard. I don't think that DDG is trying to have independently auditable level of privacy guarantees. What they are selling is trust. The trust that major tech companies have squandered over the last decade. I trust a couple guys sitting in an office in Pennsylvania somewhere way more than the faceless behemoth that is Google at this point. Who knows if that trust is worthwhile, or if things won't end up going the same way, but I'm on their side for now.
pmoriarty 8y ago
Try adding &kd=-1 to the end of your search URL. For example: https://duckduckgo.com/lite/?q=test&kd=-1 https://duckduckgo.com/lite/?q=test&kd=-1
- 3xblah 8y ago
  Thank you! Problem solved.

by 3xblah 8y ago

jolmg 8y ago

> Whatever the reasons may be for this decision, it allows DDG to track all links that the user follows from the results page. That is no better than Google. It is not a privacy improvement in that regard. It's meant so that the sites you click on can't see your search queries via the Referer HTTP header. You can turn that off in DDG's settings[1] under the privacy tab. It's the "Redirect" switch. They explain more about it[2]. Basically, it's a privacy choice they give you between preventing sites you visit from seeing your search queries or preventing DDG from seeing what you clicked on. If you want, you can prevent both by enabling their settings option to use POST instead of GET in their search. Then you can disable the redirect and the sites won't be able to see your search query from Referer, because it would have been passed through the request body instead of URL parameters. However, that's going to prevent you from being able to have your searches in your browser history. EDIT: I now note why you mentioned javascript and why the Referer option talks about being needed on old browsers. It seems that when javascript is enabled, they do something else to ensure that Referer doesn't leak search queries without passing info on what you clicked on to DDG. That means that, by default on a typical browser, both leaks are prevented. That's pretty neat. [1] https://duckduckgo.com/settings# https://duckduckgo.com/settings# [2] https://duck.co/help/results/rduckduckgocom https://duck.co/help/results/rduckduckgocom

3xblah 8y ago
Here is another perspective from a non-developer user on those reasons and the resulting tradeoff (of allowing DDG to trivially track which result links are followed): 1. Privacy conscious users can control whether a Referer header is sent and if so, the content of that header. They might choose to use a browser that does not make it easy to control the Referer header. But DDG does not account for those privacy conscious users who may be using browsers that do allow this control. (Bias discosure: I use one and not for privacy reasons.) The DDG decision seems to encourage users to choose browsers that have privacy-unfriendly default settings or that do not make it easy or even possible to control what if any Referer is sent. At the very least, it tries to resolve an issue that is a clearly client-side one and is best resolved there. 2. Correct me if I am wrong but storing "settings" with DDG requires a cookie and/or Javascript to be enabled. If this is true, then again this is not an improvement in privacy. Privacy users may prefer to keep these browser "features" disabled wherever possible. Users can of course control whether they send GET or POST by using a client such as curl or the like, or sending requests from a local copy of a DDG html search page with the method changed from GET to POST. One might wonder why DDG cannot provide an alternative search page with its form method set to POST. Users can then choose which search page they prefer. No cookie needed. Instead it sets the default method on its search pages to GET (perhaps leaking the query in the URL). How many users will bother to change the default? A minority of them, no doubt. There is an opportunity to let users turn off cookies and Javascript and let them perform searches using POST, the method usuaully advocated by those advancing internet privacy. I cannot see that it would cost DDG anything substantial to provide it, except not promoting GET as a method by default, not requiring users who want to change defaults to enable cookies or javascript to "keep state" and reliquinshing the ability to easily track which result links users follow. To reiterate, control over the Referer header is easily done by the http client. Clients that do not allow easy user control over this are not improving internet privacy. In my experience sending Referer headers is almost never necessary. It is very rare that I find a site that requires a Referer header, let alone a specific one. Sending gratuitous information seems like a clear departure from the internet privacy playbook. Then again, I am not a privacy expert and perhaps the experts have decided there must be a carve-out for the Referer header and users should err on the side of sending too much information rather than too little. The only reason I am aware of this stuff is because I use the web heavily without javascript or cookies and I take an active role in what headers I send. I generally do not do this for privacy, but for speed and control. I can strip out the prefix client-side if I must use GET.

rlue 8y ago

Here it is with piping so you don't have to litter your working directory: curl -s https://duckduckgo.com/lite/?q=example.com | grep -m1 "'result-link'" curl -sd "q=example.com" https://duckduckgo.com/lite/ | grep -m1 "'result-link'"

3xblah 8y ago
Then you cannot do diff a.htm b.htm Send the output to /tmp or some directory mounted as tmpfs. My working directory is always mounted tmpfs. This is how I keep things fast and clean.
agumonkey 8y ago
And for xpath lovers curl -sd "q=example.com" https://duckduckgo.com/lite/ | xmllint -html -xpath '//a[@class = "result-link"]/@href' - curl -s https://duckduckgo.com/lite/?q=example.com | xmllint -html -xpath '//a[@class = "result-link"]/@href' -

aphextron 8y ago

>Whatever the reasons may be for this decision, it allows DDG to track all links that the user follows from the results page. That is no better than Google. It is not a privacy improvement in that regard. I don't think that DDG is trying to have independently auditable level of privacy guarantees. What they are selling is trust. The trust that major tech companies have squandered over the last decade. I trust a couple guys sitting in an office in Pennsylvania somewhere way more than the faceless behemoth that is Google at this point. Who knows if that trust is worthwhile, or if things won't end up going the same way, but I'm on their side for now.

pmoriarty 8y ago

Try adding &kd=-1 to the end of your search URL. For example: https://duckduckgo.com/lite/?q=test&kd=-1 https://duckduckgo.com/lite/?q=test&kd=-1

3xblah 8y ago
Thank you! Problem solved.