5 ms·
In short, if you are ever going to make an embedded device, or an operating system distribution, or anything with NTP default on, please make sure, 1. NEVER, e
by bcaa7f3a8bbc 8y ago
In short, if you are ever going to make an embedded device, or an operating system distribution, or anything with NTP default on, please make sure,
1. NEVER, ever hardcode an individual NTP server (in form of a IP or domain). DO NOT just go to a list of NTP servers, then copy a few into your code. DON'T ping pool.ntp.org and get its IP address written down. DON'T DO ANY OF THESE! PLEASE!
2. DO NOT use Stratum 0 and Stratum 1 servers. Please use Tier-2 and lower. Practically, if you follow Rule No.1, then you are always following this rule.
3. If the scale of your system is small, in hundreds, or in a few thousands, PLEASE USE pool.ntp.org, this is the NTP community cluster backed by DNS load balancer. Always request the DNS, and make sure the IP is not cached locally for too long. If you need more than one servers, use 0.pool.ntp.org, 1.pool.ntp.org, 2.pool.ntp.org, etc (3 is often enough).
4. If the scale of your system is large, such as tens of thousand, or you are making a new system, you SHOULD request a customized prefix from pool.ntp.org, such as debian.pool.ntp.org, it helps the community to manage the traffic. If your system is a large commercial one, you ARE REQUIRED to donate some servers to the NTP Pool to compensate the community. Another option is running your own private NTP cluster. The policy is here: https://www.ntppool.org/en/vendors.html https://www.ntppool.org/en/vendors.html
5. If possible run a standard NTP implementation, like NTPd, chrony, or something else as long as it's written professionally. Nowadays even lightbulbs run Linux, then why don't you run a standard NTPd?
But If you can't, then make sure...
(a) implement NTPv4, DO NOT use NTPv1.
(b) Read the new SNTP RFC if you are implementing an SNTP client. http://www.faqs.org/rfc/rfc4330.txt http://www.faqs.org/rfc/rfc4330.txt
(c) DO NOT synchronize time on the beginning of an hour, or 00:00 UTC! Select a minute in a hour randomly for synchronization.
(d) Use an exponentially-increase retrial interval, DO NOT keep retrying when the server is unreachable, you are launching a DDoS attack!
(e) Support Kiss of Death packet, your client should immediately stop requesting a server, cease and desist, once a KoD packet is received.
(f) Make sure the client will stop requesting the builtin list of servers, once an alternative server is set by the user.
These should have been written in all textbooks related to practical networking lectures, but apparently, there aren't. People don't even realize that their actions are harmful, and we have all the problems...
--------------------------------
The NTP community is a complete tragedy of the commons. Even many government institutions cannot keep up with the abusive traffic, and stopped providing public NTP servers.
Today, if we don't count Microsoft and Apple's NTP, almost all public NTP servers are provided by the volunteers from https://www.pool.ntp.org https://www.pool.ntp.org. By using DNS, it forms a NTP cluster to distribute the load. These people provide time for the entire Internet, and they are the people who withstand all the abuses day by day.
People just assume they are some random super servers that always work, without being responsible for their actions, such as hardcoding IP addresses, writing abusive retry code (without exponential increment of timeout), and making a cronjob that initialize a synchronization exactly at midnight (without randomization), effectively a DDoS.
Usually, if a device comes with hardcoded NTP addresses, it, in fact, usually indicates their program is poorly-written, and the manufacturers are irresponsible. Those devices have the worst homebrew NTP implementation on the planet,
1. They send ancient NTPv1 packets, while the latest version is NTPv4.
2. They synchronize their time on the beginning of an hour, effectively making a flooding attack. Another larger flooding attack starts at 00:00 UTC.
3. They retry interval is around 3 minutes, if fails to reach the server, make even more traffic to the server, rather than an exponentially-increase interval.
4. They still try to talk to the default hardcoded servers, even if an alternative server list is set.
5. They don't support the Kiss of Death packet, nothing can stop them if they became wild.
Stratum 0/1 servers are the most vulnerable: they have highest accuracy, with reference clock. Despite the acceptable usage of ST-1 is only passing time to downstream, or for scientific purposes, since there's only a handful of them and often listed publicly, they are often spotted by those manufacturers, and put in their devices by default.
Stratum 0/1 are usually provided by universities, or unpaid volunteers for the public good of the Internet. If a single server got hardcoded in those mass-manufactured devices, serious consequences can happen, the volunteer may literally bankrupt: your whole institute/school will be kicked out from the Internet [0]; when you came to the manufacture asking to pay the damage they are responsible for, you are threatened by a lawyer from California. [1] The whole Internet community should honor the spirit of self-sacrifice of these NTP volunteers.
The NTP community pool is Stratum 2+, suitable for general use. It has similar issues of abuses - once you're in and became well-known on the net, there's no way out and you keep receiving bad-traffic, because some clueless people have hardcoded your IP address, or has put it in a cache that never expires. Fortunately given a reasonable bandwidth, it is often a negligible issue and safe to ignore. But there are exceptions. [2] One of my NTP server became DDoSed one day, because an ISP cached the IP address for pool.ntp.org with a large TTL, and the IP address happened to be mine! The traffic was 40 Mbps...
In contrast, NTPd has proper rate-limit mechanism built-in, such as KoD and good pooling interval, blocking NTP does NOT causes more user traffic. What increased is the abuser traffic. The damage caused by a standard NTPd and silly sysadmin is much less significant and is negligible compared to the Internet of Scary Things.
By the way, not only hardware devices can contains dangerous NTP code, but also software.
As long as manufactures still write broken code and unaware of the proper way to use NTP, nothing can be done to solve this issue. Many involved in these misuses and abuses are totally unaware what they are doing. The proper way to use NTP should have been written in all textbooks related to practical networking lectures.
[0]: Flawed Routers Flood University of Wisconsin Internet Time Server
http://pages.cs.wisc.edu/~plonka/netgear-sntp/ http://pages.cs.wisc.edu/~plonka/netgear-sntp/
[1]: Open Letter to D-Link about their NTP vandalism
https://web.archive.org/web/20060423012837/http://people.freebsd.org/~phk/dlink/ https://web.archive.org/web/20060423012837/http://people.fre...
[2]: Recent NTP pool traffic increase
https://mailman.nanog.org/pipermail/nanog/2016-December/089588.html https://mailman.nanog.org/pipermail/nanog/2016-December/0895...
- wpietri 8y agoOn the one hand, I completely appreciate this. When I set up NTP, I'm careful to think through consequences and tread lightly. After all, I learned my sysadmin skills at a university before the September that Never Ended, so it really was a collegial place back then. On the the other hand, the internet is a much bigger place. Things are orders of magnitude more complex. The feedback loops that made NTP work well in a 1990s university environment are mostly absent. When a problem happened then, I'd see something in the logs or in packet captures, figure out what was happening, and get the responsible person quickly on the phone. That's not even hazily possible these days. As much as I'd love to think putting a stern warning in textbooks would fix this, I doubt that would matter at all. What we really need is a major increase in observability or traceability. And failing that, what we'll get is common resources getting sliced up so they fit within domains of accountability.
- bcaa7f3a8bbc 8y agoI do agree. A complete solution is designing a new generation/revision of time protocol with accountability and anti-misuse as parts of its design, just like how people are implementing ASLR and NX in C programs and starting to use memory-safe languages. Or just see how TLSv1.3 removed all unsafe algorithms, so it will be little damage even under the worse case scenario. Also, I think NTP needs more publicity. We need people to be aware of it before we could get feedback. The community then can have a watchdog team that spots misuses and publishes alerts.
- Animats 8y agodesigning a new generation/revision of time protocol with accountability and anti-misuse as parts of its design It would be "time.google.com" and you would need a Google account.
- bcaa7f3a8bbc 8y agoOr perhaps RoughTime when security is needed? https://roughtime.googlesource.com/roughtime https://roughtime.googlesource.com/roughtime