11 ms·
Apple: We're not handing over Safari URLs to Tencent, just IP addresses
- ethansinjin 7y agoIf the list of hashed URLs is relatively static, what’s to prevent Apple from caching this information on their own CDN? If this not possible in China due to the Great Firewall, at least it could be done in other areas. Unless it’s against Google/Tencent’s TOS’es?
- innagadadavida 7y agoThe other option is to just proxy the request and hide IP address. In fact they should just offer that as a service to their users.
- dannyw 7y agoThey do this for Siri and bing.
- shuckles 7y ago(i) It is not static. It actually changes very frequently. (ii) You would get a fair amount of cache misses, so a proxy if anything would be a better approach. (iii) The providers do not allow it (source: have tried to work with at least one of them). I am not sure exactly why they want IP, but the sense I got was it did have something to do with training their models.
- codedokode 7y agoSo basically, it means that using Tencent's database Chinese Government can block access to unwanted sites in user's browser?
- ethansinjin 7y agoChina has many other more effective means of blocking websites for users within China mainland. Since Tencent is used for a malware list instead of Google for all users who have their iOS region set to China (even if they aren’t physically in China) this would technically allow them to falsely flag websites as malware even for some people outside of the country. But, this is a user configurable setting, and you can always bypass the warning.
- gruez 7y agoThat would be the most incompetent implementation of censorship ever. All you'd have to do to bypass this is disable the safe browsing feature. At least school firewalls require you to use some sort of proxy service like google translate.
- stordoff 7y agoIf you want to condition people that a site is bad but more subtlety than a outright block, it is a viable method. If you can visit it, but every time you do you are misleadingly told it is malicious, then that may have a subconscious effect (assuming you don't disable it outright), and you don't have to stop everyone from visiting to influence society as a whole. In fact, there may be cases where being blocked by the Chinese authorities lends credibility to site, whereas being flagged for malware does not. These are definitely edge cases, and no doubt the China government would to completely block a site in most circumstances, but it is another tool they could use. It also increases their coverage - the current censorship tools only cover China, but this covers any _device_ where the region is set to China (meaning it can apply to some people outside of China, and continues to apply to people who have temporarily left China).
- aclsid 7y agoThey don't really need to do that since they have the great firewall in place.
- bigiain 7y agoI wonder haw many pages on a typical website you'd need to browse to let Tencent de-anonymise that k-anonymous hash-prefix scheme? If Tencent added ihatetheccp.com and ihatetheccp.com/about and ihatetheccp.com/news and ihatetheccp.com/blog and ihatetheccp.com/donate to their "unsafe page" list - I suspect the pattern of fetching of the full list of urls for some of those specific hash prefixes in a short time would allow Tencent (or Google) to make reasonably accurate assumptions about which site you're actually visiting.
- Operyl 7y agoIf I recall correctly hashes are based only on the domain though, not the full path.
- dpkonofa 7y agoIf they're using the same method that they use with Google, then this is not the case: https://developers.google.com/safe-browsing/v4/urls-hashing https://developers.google.com/safe-browsing/v4/urls-hashing The URLs are canonicalized first and then the full URL is hashed.
- gruez 7y agoThat's true, but if you scroll down further it mentions that they try several permutations of the url, up to 30. So while grabbing 1 of the 2^32 buckets won't say much, grabbing 5 buckets (once for each permutation) in a specific order may very well indicate that you browsed a specific url.
- dpkonofa 7y agoThat’s on the client, not on the server. The server would just provide the responses for the specific hash prefix. While I agree that, in time, it would eventually get good enough to categorize people, all it would take is a change in the hash and most of the old model would be trash.
- musicale 7y agoWell, they're handing over the URL prefix hashes, which Tencent uses to look up the real URL list that it returns to Safari. Conveniently, Tencent can also log this hash/list identifier and use whatever algorithms it likes to build a user profile. I doubt Apple is actually "handing over IP addresses" - rather it's a feature of connecting directly to Tencent via the IP internet. Though they could conceivably provide some sort of proxying, caching, and batching if they wanted to improve privacy. Similarly Apple isn't "handing over the timestamps" - Tencent can just look at the clock.
- jkp56 7y agoThese hashes also reveal domain names. Most users visit many URLs on a small set of domains. If a user requests 1000 hashes that all can map to Reddit, it's very likely that the user is indeed reading Reddit. Another way to look at it: if the same person appears in a crowd on hundreds of photos, it's trivial to notice that there is something special about this person, even though in all cases the person was k-anonymous.
- dpkonofa 7y agoExcept that the problem with that is that these hashes likely won't reveal domain names. For example, the hash for reddit.com/r/gifs would be different from reddit.com/r/funny and so the prefixes would be different for both of them. Unless the requested hashes are saved for every single user, it would be way too computationally expensive for them to get anything useful out of that. Not to mention the fact that hashes would return the same prefix for any thousands of URLs. Narrowing down which domains those URLs are rooted on would be incredibly hard.
- deleted 7y ago[deleted]
- kccqzy 7y agoNo, you are wrong. You don't understand how safebrowsing URL hashes are produced. Read this first: https://developers.google.com/safe-browsing/v4/urls-hashing https://developers.google.com/safe-browsing/v4/urls-hashing under the section "Suffix/prefix expressions"
- est 7y agoI am a bit skeptical about hashing URLs. Can malware sites just change few parameters and the hash will change right?
- dannyw 7y agoThe hash is the domain name, I.e. ycombinator.com for https://news.ycombinator.com/reply?id=21254732&goto=item%3Fid%3D21254166%2321254732 https://news.ycombinator.com/reply?id=21254732&goto=item%3Fi...
- dpkonofa 7y agoUnless I'm mistaken, that's not true. The hash is computed from the full path and the prefix is the first 32-bits of the hash (or 48-bits if using Safe Browsing).
- bigiain 7y agoLooks right. From your links upthread to Google's Safe Browsing API docs: Canonicalize("http://www.evil.com/blah#frag" http://www.evil.com/blah#frag") = "http://www.evil.com/blah"; http://www.evil.com/blah"; Canonicalize("http://evil.com/foo?bar;" http://evil.com/foo?bar;") = "http://evil.com/foo?bar;"; http://evil.com/foo?bar;"; So fragments get dropped (as expected) buy query params do not (also, in retrospect, what I'd expect to make it work at all...) So https://news.ycombinator.com/reply?id=21254732 https://news.ycombinator.com/reply?id=21254732 will not end up hashing "https://ycombinator.com" https://ycombinator.com", but the whole thing including the path and query string.
- tialaramex 7y agoSafe Browsing has a canonicalization step. Given a URL the Safe Browsing implementation calculates several hashes, each of which is checked. They're basically cutting the URL up into gradually less specific URLs, so e.g. maybe the site malware.example.com is a friendly site discussing Malware, no harm there, but https://malware.example.com/samples/ https://malware.example.com/samples/ is entirely full of sample malware, don't want that, so when you try to go to https://malware.example.com/samples/windowsmacker.exe https://malware.example.com/samples/windowsmacker.exe the browser hashes not only malware.example.com but also malware.example.com/samples and malware.example.com/windowsmacker.exe This lets Google flag the actual problem, without freaking out for users who aren't in harms way. Millions of users visiting the discussion site see nothing, but everybody directed to an actual malware installer gets a warning. So a "prefix" operation happens in the code twice, once in processing the SHA-256 hash to get a 32-bit prefix which is all Google / TenCent are ever shown but also before that during canonicalisation to figure out a list of hashes (not a single hash) for each URL visited.
- justinzollars 7y agoI may finally move to a Linux laptop.
- newnewpdro 7y agoUnfortunately Firefox and Chrome both do some annoying things on Linux too, but it's probably the best option.
- netfl0 7y agoWhat annoying things are you referring to Firefox on Linux? I am assuming Pocket is one, are there others?
- newnewpdro 7y agoOut of the box Firefox, even as-installed on Debian through apt, has sponsored content on the home page and new tabs. It's basically advertising. There are other annoyances, I forget, every time I setup a new user or new profile with Firefox I'm reminded how much worse things are and have to go change a bunch of obnoxious stuff. I miss Iceweasel. Edit: https://wiki.debian.org/Firefox#Disabling_automatic_connections https://wiki.debian.org/Firefox#Disabling_automatic_connecti...
- netfl0 7y agoThanks!
- Torwald 7y agoThe problem with a Linux laptop is that you are then bound to shitty UI. A problem which already begins, if you merely switch browsers. For instance the only browser other than Safari, who has the close buttons on a tab right is Opera. I think you can tweak Firefox to do it, but you are out of luck with Brave or Chrome. That is just one example.
- deleted 7y ago
- chj 7y agoA prefix of URL hash more or less gives away the site you are visiting, given a large enough database. And in China, an IP address can be tracked down to a person, given that most sim cards can be identified due to regulation.
- dpkonofa 7y ago>A prefix of URL hash more or less gives away the site you are visiting, given a large enough database. That database would be ridiculously huge. If they truly are using the same method for Tencent as they are for Google, the prefix has would give you several thousand different domains potentially. According to Google's docs, the full URL is hashed, not just the domain. (https://developers.google.com/safe-browsing/v4/urls-hashing https://developers.google.com/safe-browsing/v4/urls-hashing)
- bigiain 7y agoI just did some ballpark math... The top return for a google search "how many pages does google have in it's index" says Google has "30 trillion web pages" (that's a 2013 article, but let's roll with that number for now). The Google Safe Browsing docs says to send the first 32 bits of a 256 bit SHA256 hash. So there's 2^32 = 4e9 possible prefixes (And 2^224 = 2e67possible hashes for each prefix.) 30 trillion = 3e13 webpages. If you assume they're evenly distributed across all the hash prefixes (a reasonable assumption for a cryptographically strong hash function) there's about 1e4 or 10,000 urls matching each prefix. (And that's a lower bound, using 2013 vintage estimates of the number of known urls...) I _think_ that means visiting 13-14 unique urls from a site in Tencent's lists would be enough to guarantee they could tell which site and pages you'd just visited? (since 2^14 > 1e4)
- thenewnewguy 7y agoI think there's a few flaws with your process: 1: I'm pretty sure browsers keep a local list of known malicious hashes, and only request a list if the URL matches it. So those 13-14 URLs would all have to happen to have a prefix on the safe browsing list (presumably quite unlikely). I guess tencent could just advertise every hash as being on the list, but I feel like that would've been discovered if that was the case. 2: Even with 13-14 hashes I don't think that _guarantees_ a match, it's just the average you'd expect to need in order to find a match. 3: This becomes significantly harder when a user browses multiple websites at a time (which most people do, even if its just because they click a link from one website to go to another).
- judge2020 7y agoIn case you only read the headline, it was confirmed to only happen if the device's region was set to China. From the statement: > To accomplish this task, Safari receives a list of websites known to be malicious from Google, and for devices with their region code set to mainland China, it receives a list from Tencent. From Apple's statement, Tencent's API functions exactly the same as Google's k-anonymity model, which you can read more about the api[0] and how it works[1] (also via this JAMIA paper[2]). 0: https://developers.google.com/safe-browsing/v4/update-api#checking-urls https://developers.google.com/safe-browsing/v4/update-api#ch... 1: https://blog.cloudflare.com/validating-leaked-passwords-with-k-anonymity/ https://blog.cloudflare.com/validating-leaked-passwords-with... 2: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2528029/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2528029/
- viraptor 7y agos/device's language/region/ They seem to be different settings and you can have Chinese language with (for example) US region.
- judge2020 7y agoThanks, done.
- bri3d 7y agoLocale region, not language. Important distinction. For example I don’t believe Tencent would be used in Hong Kong (zh_HK).
- uto 7y agoThey already started censoring the emoji keyboard in HK, this would be even easier to justify as it is "protecting the user".
- fishywang 7y agoThere's no zh_US (or zh_UK, etc.), so any mainlander living overseas and still use Chinese as the preferred language on their iPhones will get their IP address sent to Tencent. There's not much difference here.
- orasis 7y agoThis could easily be fronted by an Apple controlled CDN to avoid Tencent from getting the IP addresses.
- baroffoos 7y agoUnless there is government regulation to block that.
- gruez 7y agoYou'd think that if the CCP wants the browsing history of its citizens, it will make that mandatory (ie. all browsers have to upload their histories to the CCP), rather than through some sort of convoluted scheme like this.
- Ayesh 7y agoI'm not vouching for Brave, but this browser is doing exactly that with Google. Safe Browsing APIs can be easily cached at a proxy. However, I suppose China has rules against that and wants Safe Browsing requests made directly to tencent.
- deleted 7y ago[deleted]
- segfaultbuserr 7y agoIt's not a backdoor, it's just safe browsing. And safe browsing is controversial since the beginning because this (but now most people have forgotten it). But don't forget that cryptographic techniques are already used to protect the privacy of users, Mathew Green just wrote a good analysis. * https://blog.cryptographyengineering.com/2019/10/13/dear-apple-safe-browsing-might-not-be-that-safe/ https://blog.cryptographyengineering.com/2019/10/13/dear-app... > To address these concerns, Google quickly came up with a safer approach to, um, “safe browsing”. The new approach was called the “Update API”, and it works like this: > * Google first computes the SHA256 hash of each unsafe URL in its database, and truncates each hash down to a 32-bit prefix to save space. > * Google sends the database of truncated hashes down to your browser. > * Each time you visit a URL, your browser hashes it and checks if its 32-bit prefix is contained in your local database. > * If the prefix is found in the browser’s local copy, your browser now sends the prefix to Google’s servers, which ship back a list of all full 256-bit hashes of the matching URLs, so your browser can check for an exact match. > At each of these requests, Google’s servers see your IP address, as well as other identifying information such as database state. It’s also possible that Google may drop a cookie into your browser during some of these requests. The Safe Browsing API doesn’t say much about this today, but Ashkan Soltani noted this was happening back in 2012. > It goes without saying that Lookup API is a privacy disaster. The “Update API” is much more private: in principle, Google should only learn the 32-bit hashes of some browsing requests. Moreover, those truncated 32-bit hashes won’t precisely reveal the identity of the URL you’re accessing, since there are likely to be many collisions in such a short identifier. This provides a form of k-anonymity. > The weakness in this approach is that it only provides some privacy. The typical user won’t just visit a single URL, they’ll browse thousands of URLs over time. This means a malicious provider will have many “bites at the apple” (no pun intended) in order to de-anonymize that user. A user who browses many related websites — say, these websites — will gradually leak details about their browsing history to the provider, assuming the provider is malicious and can link the requests. (Updated to add: There has been some academic research on such threats.) > And this is why it’s so important to know who your provider actually is. > What does this mean for Apple and Tencent? > That’s ultimately the question we should all be asking. > The problem is that Safe Browsing “update API” has never been exactly “safe”. Its purpose was never to provide total privacy to users, but rather to degrade the quality of browsing data that providers collect. Within the threat model of Google, we (as a privacy-focused community) largely concluded that protecting users from malicious sites was worth the risk. That’s because, while Google certainly has the brainpower to extract a signal from the noisy Safe Browsing results, it seemed unlikely that they would bother. (Or at least, we hoped that someone would blow the whistle if they tried.) > But Tencent isn’t Google. While they may be just as trustworthy, we deserve to be informed about this kind of change and to make choices about it. At very least, users should learn about these changes before Apple pushes the feature into production, and thus asks millions of their customers to trust them.
- AtomicOrbital 7y agoAs an aside ... Let's remember a Chinese outfit bought the Opera browser a while ago
- AtomicOrbital 7y agohttps://www.digitaltrends.com/computing/opera-sold-600-million-chinese-consortium/ https://www.digitaltrends.com/computing/opera-sold-600-milli...
- m463 7y agoI had camscanner plus and tencent bought it. When the link to the privacy document worked, it basically said we collect everything, but in a fuzzy jumble of poorly worded english that looked like it took a few laps through google translate.
- AtomicOrbital 7y agoA consortium of Chinese companies led by Qihoo 360 purchased the browser and several other Opera assets for $600 million https://www.digitaltrends.com/computing/opera-sold-600-million-chinese-consortium/ https://www.digitaltrends.com/computing/opera-sold-600-milli...
- cjensen 7y agoCouldn't this be solved if Apple sent encrypted requests to their own proxy server to hide where the requests are coming from?
- thanatos_dem 7y agoBut then they’d need to revise all the headlines to be outraged at Apple’s mass data collection.
- a3n 7y ago"It's just metadata."
- taneq 7y agoAh, I see you come from the Australian government school of thought. -_-
- abjKT26nO8 7y agoThe laws of mathematics are very commendable, but the only law that applies in Australia is the law of Australia.
- taneq 7y agoOh, well in THAT case... flaps my arms and flies away
- drawkbox 7y agoThe web is straight up wack today. It has gone from anti-authoritarian to authoritarian and a permanent record.
- kmlx 7y agoit is actually people and/or governments that have decided they want to fully control the web.
- mister_hn 7y agoGiving out IPs is also bad, they can fingerprint users and track them by find correlations between what Apple gives them vs what ISPs give them
- __m 7y agoThen why bother what Apple gives them?
- jjcm 7y agoKinda pointless that they're hashing the urls, as to tell if that url was malicious google/tencent would have had to generate that hash themselves as well. The associations are already there, so the hashing step adds nothing (except to prevent a MITM attack where the bad actor didn't have a lookup table). If Apple wants to create a privacy-oriented safe browsing experience, they need to ship that dataset with the phones themselves.
- niij 7y agoBy using the prefix of the hash, it adds some uncertainty about which URL you're requesting
- alienallys 7y agoThis assumes there can be many prefix matches for any given url hash. How many times (probability) do you think a 32-bit hash prefix matches more than one 256-bit hash? Very very unlikely; it's one-to-one practically speaking.
- niij 7y agoNearly 100% probability. Someone else[0] already did the ballpark math and came up with ~10,000 unique URLs per 32-bit hash prefix if you use the number of pages indexed by Google. My math has it coming out to be ~10 billion per 32-bit hash prefix. I simply did: 30,000,000,000,000/(36*8), but I might be doing something wrong, happy to be corrected. The issue is, as mentioned in that comment, with visiting a handful of pages you can get a clearer picture of similar URLs that are being returned among these hash prefixes, which can be used to build a profile of your browsing history. [0]: https://news.ycombinator.com/item?id=21255223 https://news.ycombinator.com/item?id=21255223
- dwild 7y agoWhy are you doing 36*8? It look likes 26 + 10? You got 32 bits, thus 2^32 = 4 294 967 296 possibilities 30 000 000 000 000 / 4 294 967 296 = 6984.9193 That doesn't seems to me like a 100% probability at all.
- TicklishTiger 7y agoApple insists that Safari doesn't reveal a different bit of information, the webpages Safari users visit Insists how? Was there a public statement? A phone call? With whom? An email? From whom? Is it normal journalism to just say "Company X says ..." without stating who of the company stated it and how? If it is true what the article claims, that they give hashes to Tencent, then this statement is false: Apple insists that Safari doesn't reveal a different bit of information, the webpages Safari users visit. Because a hash is an id for a set of pages. That is not what you mean when you say it doesn't reveal the pages.
- filleduchaos 7y ago> If it is true what the article claims, that they give hashes to Tencent, then this statement is false Is today really the first day all of y'all are hearing about Safe Browsing and how it works? The _prefix_ of the hash is sent to the blacklist service, not the entire thing.
- TicklishTiger 7y agoThe prefix is just a shorter hash. It specifies a set of urls. The urls that have a hash which begins with that prefix.
- filleduchaos 7y ago> The prefix is just a shorter hash With all due respect, this makes no sense whatsoever. You can't make any meaningful conclusion (in isolation, or even with small enough/random enough datasets) about the content that was hashed from only a part of the resultant hash, that's the whole point of how hashing works.
- LeoPanthera 7y ago> Insists how? Was there a public statement? A phone call? With whom? An email? From whom? Before bitching, actually read the article. "In a statement emailed to The Register (!), an Apple spokesperson said:"
- wonnage 7y agoI feel like the entire tech world is overthinking this. First, the Great Firewall already blocks the known "dangerous" sites. If you manage to access them via VPN, then chances are Tencent is getting your VPN's IP, and it's not your problem. If it's a hot new URL that hasn't been firewalled yet, then it's also not going to be in the safe browsing database, which means that your browser won't make that second request (with the full hash) to verify it. You can think of various other what-if scenarios. But the other thing is that all internet providers within China are in bed with the government and can cough up your network history on demand. Why bother going through Tencent? People are pillorying Apple over a vague theoretical risk that is only really practical in the free world.
- PavlovsCat 7y ago> Why bother going through Tencent? Indeed, especially considering this: https://news.ycombinator.com/item?id=21241712 https://news.ycombinator.com/item?id=21241712 I think the only difference is that it also applies to the region set, so don't do that if you're not in China.
- criddell 7y ago> Why bother going through Tencent? It's a way of reporting the use of a VPN.
- lordlimecat 7y agoVPN use is pretty obvious to the ISP.
- dwild 7y agoEDIT: I found out that there isn't 2 requests, only one and only with the prefix. The prefix is first lookup in the client database, and if it's there, a request is made to get the URL related to that prefix. My idea still works but then they would need to give theses specific prefix in the database already, which will be pretty obvious. > If it's a hot new URL that hasn't been firewalled yet, then it's also not going to be in the safe browsing database, which means that your browser won't make that second request (with the full hash) to verify it. Who say you need the full hash? Keep all prefix with the timestamp and their IP address and believe me, you can get a pretty good picture. All you need is to keep a bunch of URL you don't block but consider harmful, and a bunch of of others URL from theses same domains. Here's a partial hash that I'm pretty sure would be quite useful: > 18ec It map to this URL: > https://www.reddit.com/api/submit https://www.reddit.com/api/submit From your database, find everyone that called for 18ec. It may matches many other URL, but use your backup for Reddit and the timestamps. You now got the username linked to their IP. Someone doesn't post? Use 1483, B5E7, 13F9, 5B58, 0859, etc... The funniest is that you could probably find the URL yourself using theses prefixes... but yeah the goal is to find out who went to theses URL, not really which URL they went to. You don't need that many matches to confirm they truly went there.
- deleted 7y ago[deleted]
- Havoc 7y ago>Party of China's "Study the Great Nation" – through browser telemetry. Wow. As the corporate types would say - the optics on this are not great
- lcnmrn 7y agoApple should rely on its own server to check for safe browsing lists with Tencent, Google and other providers. An Apple API that encapsulates these external APIs.
- Hitton 7y agoPrivacy minded people I know disable safebrowsing even when it's provided by Google, and although Google dropped its "Don't be evil" long time ago, it is still more trustworthy than Tencent backed by Chinese government.
- 345218435 7y agoit appears to me their current leadership also would have said <<they‘re one of our biggest platform drivers. let‘s just put flash on idevices. screw „the right thing“. also, money money money!>> when adobe was bitching and mourning. sure, there have been issues under jobs. but those were of different quality. cook counts beens and has no balls. schiller goes on stage and says „courage“. ive has nobody measuring his and his teams ridiculous design decissions — flat it must be! i would bet money steve would have put the iphone 6 down, notice it wobbles, pull up his eye brows and sent the team back down the design bunker telling them not to emerge until that bullcrap is fixed.
- ryuukk_ 7y agowhen will these websites be finally banned pseudo buzz driven journalism, they are from the same species of crapzilla employees, the worst, the one that deserve to be put on fire because buzz drizen behavior is cancer
- hmans 7y ago苹果好人
- paulcarroty 7y ago> Apple may deny users in China VPN protection, it may deny Hong Kong democracy protesters an app used to avoid police, and it may remove references to Taiwan in localized versions of iOS 13 for the Chinese-controlled Hong Kong and Macau. With recent HKmap.live deleting this is EPIC FAIL, amigos. Seems like now China just bought most Apple shares and make new Huawei. Just imagine it 3-4 years ago.