6 ms·
Intercepting t.co links using DNS rewrites
- jeoqn 4y ago[flagged]
- djhworld 4y agoAuthor here, thanks for reading.
- netanbing 4y agoHave you considered using services like https://wheregoes.com/ https://wheregoes.com/ to fetch the final destination and navigate?
- djhworld 4y agoAuthor here, yeah I mention this at the start, there's quite a few of these link uncloakers. The annoying part is having to copy the link, navigate to the website, paste in the link etc. I was looking for something more "seamless" and works cross device (e.g. on phone, in the Twitter app etc) not just browsers. With this you just click the t.co link and the result is there instantly. It's a dumb solution but was fun to write.
- netanbing 4y agoGot it. The process of pipelining and piecing it all together is interesting. Thank you for the post.
- deleted 4y ago[deleted]
- woodruffw 4y agoNice writeup, and a nice demonstration of one of WebPKI's limitations! I understand why both HPKP and Expect-CT have been obsoleted, but it's a bummer that we still don't have a good enforcement mechanism for CA/cert pinning for a particular site. CT itself does a reasonable job of mitigating the "globally visible mis-issuring CA" problem, but does nothing to help users whose certificate stores contain all kinds of mystery enterprise or application-installed CAs.
- bourgeoismedia 4y agoIs your argument that it shouldn't be possible for a user to intercept t.co in this way? Seems like a perfectly valid use case (sidecar process to unwrap 9 layers of redirects from an anonymous browsing context). If the sidecar is validating the original t.co certs and you trust it then what's the problem?
- woodruffw 4y ago> Is your argument that it shouldn't be possible for a user to intercept t.co in this way? Not necessarily; the argument is that it's indistinguishable from a malicious MiTM. I think this is a great and legitimate use, but it's also probably something that website providers should be able to make themselves resilient against (or, at the least, be able to audit when it happens).
- ndriscoll 4y agoIf I configure my computer that I own to use my CA and proxy certain traffic, I don't see how that's any website's business. They have as much right to audit how I've set up my computers as I do theirs.
- woodruffw 4y agoThere are different models here: for every person like you who's trying to reasonably proxy their local traffic, there's a nation state, overbearing educational software provider, &c. who's trying to get access to sensitive and potentially life-affecting communications. When it comes to things like finances, private chats, &c. I think there's a reasonable argument to be made that it's in the website's (and my!) interests to be able to detect and prevent these kinds of man-in-the-middling.
- ndriscoll 4y agoThe problem is that in practice, at least in the US, the most realistic threats are from websites you visit delivering drive-by malware (e.g. spyware and adware), which they actually do constantly. It's such a common practice that it's not even usually phrased that way, but just imagine if you exploited eBay's web servers to port scan their internal network, which is exactly what they did to customers. The responsible employees should be criminally charged for that. It doesn't matter if it's in the websites interests. The client computer does not belong to them, and it's definitely not in the owner's interests to let others "audit" them just like it's not in web hosts interests to let us "audit" their nginx configs.
- chamik 4y agoAwesome writeup! It's short, but I've learned a lot.
- quesomaster9000 4y agoI wish there was an easier way to uniformly rewrite links across applications, one which annoys me is Reddit - they seem incapable of consistently linking to content on their own site, some could be fixed with a greasemonkey script but that doesn't work on mobile or across apps. But this is part of a wider pattern of the internet becoming deep fried, instead of a link we get a short code to a facebook page with a bot reposting tiktok videos of a phone screen-recording of an editorialized livestream of somebody watching a screen-recording of the video on youtube - and ofcourse it starts half way through then loops round again and plays twice with three sets of black bands, content creator @names, watermarks, wifi & signal indicators etc. If we could all do one thing to help stop the spread of this cancer, that would be great: de-duplication via content addressable links / tags. But it'll never happen.
- nonrandomstring 4y ago> the internet becoming deep fried Good phrase. I've frequently heard the term "Kentucky Fried Education" applied to the decay of schools and universities under the influence of big-tech mediocrity. My Digital Vegan response would be that "heavily processed" content harms your intellectual health because it's not fresh and has no vitamins :) Ted Nelson anticipated this "death by indirection". Some of his early writings on hypertext still seem like they're from a future we might one day get to. The idea that links should be bi-directional, and how different the Internet would be, is still mind-blowing. Most of the problems you describe stem from the mediating platforms being fundamentally controlling and dishonest. Everything is a work-around to around a work-around... all the way down. > will never happen I get the shrugging cynicism. But to me it's a good thing they're too broken to change. Let these festering sewers of low-quality content digest themselves and make room for new growth.
- ianai 4y agoCouldn't a bi-directional link still be written into HTML? Like any other tag, it could simply go unused while being available so nobody feels obligated.
- 4y ago
- jeroenhd 4y agoJust installed this in my own network. Already had my own CA so I just took all the supported domains and generate a certificate for it. The list kind of pollutes my Pihole domain overrides but that's alright by me. It works well! I'm running it with Nginx in a Proxmox LXC container where I've allocated a meager 64MB of RAM for it and it still has RAM to spare. I wish I could say the same of other "small" tools from across the web that I'm running. I like the minimalist web pages and the fact it auto-resolves multiple redirects on its own (bit.ly -> msft.it -> aka.ms -> ...) without making you wait for the page to load and the fact it removes tracking parameters for you. I know there are online tools and extensions that the same but those are a pain to install on mobile.
- djhworld 4y agoThis is awesome, thanks for taking the time to try it out! I honestly threw it together over a few hours this weekend so it's in a very rough state, and there's no unit tests for it, and none of the code is commented or structured that well so there's probably a lot of edge cases it does not account for. But still, glad it worked!
- logicallee 4y agoFor anyone who wants to check a specific link, https://wheregoes.com/ https://wheregoes.com/ does a good job of tracing where a Twitter link (or any other redirected link) goes. I just tried it and it works on t.co links.
- h43z 4y agoWhat is this trying to protect from? You see the actual URL in the tweet. Are you worried about redirects, if so why?
- deleted 4y ago[deleted]
- jojobas 4y agoYour clicks being logged by Twitter.
- b112 4y agoAnd google, facebook, reddit, everyone. I suppose it is another thing to push back against, though.
- brirec 4y agoIt’s not like the original URL is somehow encoded in the short URL, so this bot still has to request from t.co (and everywhere in between) to get the target.
- jojobas 4y agoThe proxy is resolving all links, therefore denying twitter the information on which links you actually click (see also ad nauseum).
- jacobn 4y ago? It only resolves the links navigated to, right? It shows you the interstitial page, but only when you try to navigate to the link. It does enable you to get the clean link and share that.
- djhworld 4y agoYou don't always see the URL, sometimes it's truncated or just the t.co link, it's not very consistent. Additionally if I want to share a URL with friends it's usually polluted with all sorts of nonsense query parameters, this tool strips them and gives you a nice clean one. As for the tracking element yeah, the link shortening services will still see my IP connecting to the service, doing a head request, just with a different user agent (whatever the one go's stdlib sets). One extra layer could be to move the go service out of the network, but tbh I didn't start this project with these considerations in mind
- mteam88 4y agoSimple, elegant. Love it. I wonder if there would be some way to redirect kinda even if you weren't on your home network... You mention using a VPN? Which one?
- xpressvideoz 4y agoThat 9-hop shortening example is disgusting, I wonder if that could be alleviated by introducing some protocol: 1. Make all shortening services append a `This-is-a-shortening-service: true` header to all the responses they send. 2. When a link is added to a shortening service, check if the response from the link has the header above and resolve the destination, recursively.
- westurner 4y agoawesome-url-shortener: https://github.com/738/awesome-url-shortener https://github.com/738/awesome-url-shortener /? shorturl api OpenAPI https://www.google.com/search?q=shorturl+api+openapi https://www.google.com/search?q=shorturl+api+openapi - TinyURL OpenAPI: https://tinyurl.com/app/dev https://tinyurl.com/app/dev - GH topic: url-shortener: https://github.com/topics/url-shortener https://github.com/topics/url-shortener A https://schema.org/Thing https://schema.org/Thing may have zero or more https://schema.org/url https://schema.org/url and/or https://schema.org/identifier https://schema.org/identifier ; and then first the ?s subject URI that's specified with the `@id` property in JSONLD RDF. You can add string, schema:Thing, or URI tags/labels with the https://schema.org/about https://schema.org/about property.
- citrin_ru 4y agoIt's disgusting but unfortunately it's not a bug - it a feature which allows each redirect hop to collect (and sell/use) some data about you.
- modeless 4y agoWill this show up in Certificate Transparency logs?
- 8organicbits 4y ago> I run my own self-signed Certificate Authority (CA) and my reverse proxy uses certs signed by this CA There's no need to track certificates from private CAs. In order for your browser to trust the certificates you need to manually trust the CA (or your system admin will configure it). Everyone else will see it as invalid.
- donalhunt 4y agoUnlikely as the logs are appended to by CAs when they issue certificates. Browsers could in theory contribute data but the infrastructure to support that would likely to be orders of magnitude bigger. I'm scared even thinking about going down that rabbit hole due to the expectation of what would be found (MITM evidence). :_(
- modeless 4y agoI thought browsers did contribute. Otherwise any CA could issue as many certificates as they like and simply not report them. Is that really possible?
- AviationAtom 4y agoIf you want to hear of one of the worst URL forwarding "services," Facebook is it. They got so bothered by people stripping all the tracking data from links they were sharing that they actually embedded all their tracking within the primary portion of the URL. If you want the truly original URL the only way to get it is to abuse one of their APIs.
- pgl 4y ago> I only really noticed this properly when my DNS sinkholing server (Adguard home) started blocking t.co links and I was getting an error when say, clicking a linked news article Mission accomplished!
- 1vuio0pswjnm7 4y agoIf (hypothetically at least) the "go service" is running on the device, then if one controls DNS on the device, one can set the A record for the t.co domainname to the address of the go service. In that case, does one still need a "reverse proxy".
- djhworld 4y agoYeah, I was running the service on a different port though, the proxy was just to work around that.
- 1vuio0pswjnm7 4y agoWhy does service need to run on a different port, i.e., not 80 or 443. What about running the service on the loopback, 127.0.0.2 or whatever. I ask because I have run into similar problems.
- TheHappyOddish 4y agoUsing nginx, caddy, apache etc means you can offload the SSL handling there and let the application just be an application. In the age of containers, setting up a reverse proxy to handle it is minimal effort.
- 1vuio0pswjnm7 4y agoI use forward proxy, not reverse proxy. It requires minimal effort. No containers needed. Not a caddy user, but I believe caddy, with a plugin, can function as a forward proxy.
- gigel82 4y agoThere is the FastForward browser extension if you don't want to compromise your device's security with self-signed certificates.
- meatjuice 4y agoJust quit twitter and it's all good huh? Why is everyone not leaving Twitter?
- human 4y agoIt’s not just Twitter. Maybe this tool could be made to uncloack a list of known url shorteners.
- djhworld 4y agoIt actually does work this way, you just need to setup your DNS for the other shorteners too :) List of supported ones is here: https://github.com/djhworld/theunwrapper/blob/main/config/unwrappers.json https://github.com/djhworld/theunwrapper/blob/main/config/un...
- senorqa 4y agoFriiter https://github.com/jonjomckay/fritter https://github.com/jonjomckay/fritter frontend for twitter automatically unwraps all t.co URLs and displays the original ones
- arnath 4y agoMaybe I’m missing something, but it seems like this doesn’t need to be a DNS rewrite at all. Couldn’t you just set a proxy server for outgoing requests that passes through for ones that aren’t a redirect? Or is the trade off there performance (since you’re proxying every request instead of just certain DNS names)?
- deleted 4y ago[deleted]
- TheHappyOddish 4y agoIf you're already running a squid style proxy server, you'd need to MITM the TLS request regardless. If you're just doing it for a single site, the DNS solution is a lot more light weight.
- justusthane 4y agoClever solution! The author keeps emphasizing how hacky and what a terrible idea this is, but it's really not provided they trust their own infrastructure. This is exactly how SSL decryption in corporate environments works as well - MITM traffic using a cert issued by your own CA.
- djhworld 4y agoI think I was more bracing myself for a deluge of disapproving comments, so just wanted to make sure to emphasise I know the drawbacks of the approach and understand the risks etc :)
- TheHappyOddish 4y agoWell, it seems you've been disappointed! I thought it was odd you kept calling it "bad" and "awful". It's exactly as secure as any other certificate on the web. Arguably more, as you're aware of any access to the keys. The only differentiation is a commercial interest, nothing "bad" at all.
- throwaway81523 4y agoI have a messy python script that does something like this for a bunch of sites. I've been wanting to rewrite it into a browser extension someday. Doing it at DNS level hadn't occurred to me. The python script is able to clean up a lot of tracking parameters though, while leaving the base domain the same, as well as bypassing some redirectors like t.co.