13 ms·
Excessive load on NTP servers
- deleted 10y ago[deleted]
- easytiger 10y agoWait.. they are saying the app itself is making NTP requests? > Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs. Hmm. Is that a fraud prevention thing or something? No way on earth a user app should be getting its own time
- nkcmr 10y agoThe report does say "app" specifically. So I assume yes.
- acqq 10y ago> Is that a fraud prevention thing or something? See my other post here, and the problems in the (third party) iOS NTP library "features" and its use. There "createAssociations" in that library without any parameters contacts all the IPs behind the big domain list of the NTP servers! And according to the forum every IP is contacted -- behind one server name there are 3-4 servers in the DNS in this case, I get 31 server! Real "distributed denial of service" attack.
- dan1234 10y agoJust to be clear, this is a 3rd party library not part of iOS itself.
- acqq 10y agoYes, thank you, the third party library that Snapchat used without even thinking what it does by using some (wrong!) defaults. I thought it was obvious from the links I've given. I've edited my posts to name it as such.
- cuonic 10y agoSnapchat have a lot of "fraud protection" in the form of time sensitive tokens hashed with secret keys generated by strange .so libraries. This is used to keep third party apps from using their API, obviously a lot of user's devices have incorrect clocks, so when they reduced the secret token time frame lots of users probably started getting API errors, so this is their attempt at a solution.
- matt_wulfeck 10y agoThank you! This is the most reasonable answer I've seen in the thread so far.
- brainfire 10y agoI've never used Snapchat but I believe one of its features is time-expiring photos. If they do the expiration in the client then this may be a way to check if a user is getting around it by setting the system clock backwards.
- jrockway 10y agoNTP doesn't cryptographically verify the time. All you do is have your router redirect these NTP requests to your own server, which is set to serve the wrong time. Ironically, part of an HTTPS handshake involves sharing the server time in a cryptographically-verifiable manner. I am not sure why they don't use that! https://github.com/ioerror/tlsdate https://github.com/ioerror/tlsdate
- brainfire 10y agoWell, I didn't say it was a good way ;)
- viraptor 10y agoThat only works for current TLS. Version 1.3 makes sending the server date optional.
- Dylan16807 10y agoOptional, but does anyone turn it off? They control their own servers, anyway.
- viraptor 10y agoOptional for the implementations. So basically your ssl library is unlikely to do it. And at that point having a "getTime()" API call in your service is simpler than having a custom patched SSL implementation.
- Dylan16807 10y ago
- conradev 10y ago> No way on earth a user app should be getting its own time This practice is becoming increasingly common for "time-sensitive applications": https://eng.lyft.com/freezing-time-6ebe8ffe3321 https://eng.lyft.com/freezing-time-6ebe8ffe3321
- acqq 10y agoJust checked, at least they use "time.apple.com" as default, and the default "maximum of servers" is 5. https://github.com/lyft/Kronos/blob/master/Sources/NTPClient.swift https://github.com/lyft/Kronos/blob/master/Sources/NTPClient... The third-party library used by Snapchat didn't have any "maximum of servers" (using 30 at once(!)) and defaulted to many in the ntp.org pool, across all the continents!
- foota 10y agoThis was an error on their behalf. http://mailman.nanog.org/pipermail/nanog/2016-December/089620.html http://mailman.nanog.org/pipermail/nanog/2016-December/08962...
- deleted 10y ago[deleted]
- gbrown_ 10y agoFor all of Apple's App Store vetting one would think this kind of behavior would have thrown up a flag at some point no?
- profmonocle 10y agoThis really isn't the sort of thing that would show up on an App Store review. The tests aren't done by engineers, so something's only going to get caught if it's noticeable by the user or gets caught by automated tools. (e.g. use of private APIs.) Excessive NTP queries aren't going to cause noticeable issues on the device or on Apple's test network, especially with only one or two people testing simultaneously.
- AlphaWeaver 10y agoApp Store vetting varies wildly and tends to trend towards more close introspection on smaller apps. Well known apps such as Facebook Messenger and Snapchat for example can get an update reviewed and pushed out faster than a standalone developer.
- pooper 10y agoI think Apple should require app vendors to just submit source code and build instructions and have Apple just build it. It'd be harder to pull off on Google Play Store but I think Apple could make this happen if they wanted to.
- acqq 10y agoAccording to the forum, the pattern matched this third-party library: https://github.com/jbenet/ios-ntp https://github.com/jbenet/ios-ntp Specifically, all the servers(!) from here are contacted: https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-lib/NetworkClock.m#L121 https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-lib/Ne... Note that the library author wrote: "ios-ntp is often (mostly?) used to make sure someone hasn't fiddled with the system clock. The complications involved in using multiple servers and averaging time offsets is overkill for this purpose. The following skeleton code is all that is needed to check the time." And that "skeleton" contacts just "time.apple.com" But the library really has the default possibility of contacting a lot of the ntp.org servers from a big list ("createAssociations" with no parameters!) and it's bad. As we know, the developers like to just "copy-paste" whatever is where. Or use any defaults. "Hey it works."
- mrweasel 10y agoI think it's pretty safe to assume that the developers have no idea that things like vendor zones exists. NTP, like DNS or SMTP, is ubiquitous infrastructure, not something the average developer sets aside time to understand.
- acqq 10y agoThey didn't have to use ntp.org pool at all. They obviously wanted to check if somebody changed the time on the phone and to still use "the internet time." And for that contacting one Apple's server (time.apple.com) were enough. It seems that the iOS library author "helpfully" provided the default of contacting 30 servers from the ntp.org pool.
- Fnoord 10y agoIs it possible to spoof the reply? Or block the request? This smells like security by obscurity to me.
- Godel_unicode 10y ago
- lima 10y agoWorst part is that they did not bother to use a vendor zone.
- profmonocle 10y agoIndeed. This is a pretty clear misuse of the NTP pool. > You must absolutely not use the default pool.ntp.org zone names as the default configuration in your application or appliance. - http://www.pool.ntp.org/en/vendors.html#vendor-zone http://www.pool.ntp.org/en/vendors.html#vendor-zone Hopefully they were just unaware of the vendor zone policy.
- acqq 10y ago> Hopefully they were just unaware of the vendor zone policy. It seems they didn't know, or didn't care, how both the third party iOS library they used and the NTP worked, see my other posts here. They surely didn't need ntp.org pool at all.
- Declanomous 10y agoFor whatever reason, ntppool.org is blocked at my work. And of course, you don't get the page that states why when the website is served via https. Not that I need to see the page to know it was either blocked for "hacking" or "entertainment", and I'm guessing it's not entertainment. Edit: This probably explains why our clocks have been off by 45 minutes since Monday. I guess it will be entertaining to see how long it takes for IT to figure this one out.
- ryanlol 10y agoGambling would be my guess.
- Twirrim 10y ago> I guess it will be entertaining to see how long it takes for IT to figure this one out. Why not just tell them. What have you got to lose? Hell, blame your charitable spirit on the holiday season if you must.
- Declanomous 10y agoI got told off for diagnosing issues in the past. The IT director is a megalomaniac and interprets it as a challenge to his power. The only time I offer suggestions now is when one of his employees specifically asks me for help. Edit: I realize "got told off" didn't really capture what happened. I came in early one day and noticed we were having a dns issue. I manually refreshed my DNS cache and it started to work. I sent him an email to let him know that the DNS cache was expired. He told me I was out of line and complained to HR. I had to go meet with HR, which was pointless since they think he is on a power trip as well. Anyways he added a line to the IT policy that specifically prohibits "performing a diagnosis on the network or any of IT managed systems."
- leesalminen 10y agoI encountered a similar IT manager in high school. I kept telling him that netsend wasn't locked down and that any user could run a .bat. He told me I was wrong. So, I wrote a .bat with a netsend command and emailed it to all staff. Multiple staff clicked on the attachment. Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business was locking down .bat execution.
- sschueller 10y agoWhy on earth would you do that? If you want to prevent users from altering their time use your server and do a time compare with your server. NTP can be easily intercepted and altered so it would make a lot more sense to do this via a encrypted certificate pinned communication path increasing my work load drastically to alter the time. I snapchat going to pay for the DDOS they created?
- sleepychu 10y agoThe way they achieved their goal was misguided but what you've described skips over large parts of the problem. Why would the server time and phone time be in sync?
- jstanley 10y agoThey could just use their own canonical time (from their server) instead of hammering NTP. It doesn't say anything about synchronisation between phone and server. EDIT: In fact it is easier to implement it this way than using NTP. I've implemented something similar and I found it easier to add an API endpoint that returns time() than to ship an NTP client...
- Diamons 10y agoBut then you need to manage a server and ensure that the time is running accurately. Why add such a large level of responsibility for little to no gain?
- jstanley 10y agoHow is that extra responsibility? Do you think Snapchat aren't already running servers? And you don't care whether the time on the server is running accurately if you're just using it to generate tokens which are checked against the time on the server. It just needs to be consistent, it could be an arbitrary counter and would still work fine as long as it counted up reliably at consistent intervals.
- 10y ago
- coleca 10y agoFWIW my teenage daughter has been complaining about this latest Snapchat update for iOS the past couple days. It constantly crashes and causes the phone to reboot itself. Looking at Twitter, there's tons and tons of people reporting the same issue, so it seems pretty widespread. Wonder if it's related to this NTP issue.
- skeptic2718 10y agoCan apps cause iOS to reboot? That's a bit shocking. I don't own an iPhone.
- ayuvar 10y agoIt's not isolated to iOS. Snapchat does something funky in userland. I think on the Nexus 4, Snapchat still ships with a warning that it doesn't work properly. Mine would reboot about every second time I took a picture.
- ansible 10y agoI haven't investigated the issue, but I've heard that it was indeed a bug with the Nexus 4 drivers that caused the crash. Still, I've got to wonder, what are they doing that's so different than other camera apps that seem to work fine?
- stonecodegump 10y agoInteresting. I had similar problems on my Samsung several years ago when I was trying Snapchat out. I uninstalled and never returned..
- 1_2__3 10y agoI for one am shocked - shocked! - that Snapchat would be the kind of company to be cavalier about this kind of thing.
- thejosh 10y agoYeah, it's been really hit and miss here in AU for a few people I know.
- scoobydont3 10y agoThis is an example of the engineering "genius" within crapchat. Only yesterday we read an article about how the but chief keeps everyone apart and in the dark, lest they should discuss precious stuff, like maybe code reviews where spamming NTP is deemed OK.
- mark-r 10y agoThis happens often enough that Wikipedia has a page devoted to it: https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse The first one I had heard of was Netgear vs. UW-Madison.
- known 10y agoCaptcha should fix it
- _RPM 10y agoAnd to think that SC's engineering is praised among college kids is laughable.
- nxtrafalgar 10y agoOn my device, at various points, Snapchat and Uber have both been completely nonfunctional for days on end. I didn't think building phone apps could be so difficult for these large companies.
- sateesh 10y agoIt is interesting to read through the whole thread in a chronological order starting from the first message: http://mailman.nanog.org/pipermail/nanog/2016-December/089525.html http://mailman.nanog.org/pipermail/nanog/2016-December/08952... It took 4 days, to zero on the root cause. As is usual in a complex scenario like this there are a few false positives, some suspects abusing the protocol and alas final redemption. Amazing work by a dedicated group of technical folks in coordinating (just via emails, I suppose) and tracing the root cause.
- Faaak 10y agoI wondered why I was seeing so much packet loss on my IP: http://mrtg.vi-di.fr/krootservers.ping.html http://mrtg.vi-di.fr/krootservers.ping.html Guess I know why now..
- deleted 10y ago[deleted]
- sstevo66 10y agoI do some work for the Network Time Foundation and we were not contacted by snapchat as far as I know. Anyone have a contact there, they probably need our help.
- askbjoernhansen 10y agoPeople from the NTP Pool community were talking to them (including myself, briefly). Given the available information I'm not sure why you think they need help from NTF ...