10 ms·
Why is my NTP server costing $500 per year? Part 1 (2014)
- preinheimer 10y agoReminds me of when Netgear decided to use the University of Wisconsin NTP servers as the default in their consumer products: http://pages.cs.wisc.edu/~plonka/netgear-sntp/ http://pages.cs.wisc.edu/~plonka/netgear-sntp/
- zump 10y agoHow is that even legal?
- Elrac 10y agoOnce you put an NTP server on the 'net, it's public - pretty much like most Web sites. Sure, there are reasonable expectations of decency like for anything in the Commons, but I don't think there's any legal defense against skunks at the picnic. IIRC, the university called Netgear out for doing something stupid and disruptive, and Netgear stopped doing it. The second best possible scenario, I guess.
- preinheimer 10y agoI believe an "agreement was forged", no public details, but I might assume some money changed hands.
- DanBC 10y ago$350,000. https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#NETGEAR_and_the_University_of_Wisconsin.E2.80.93Madison https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#NE... > NETGEAR has donated $375,000 to the University of Wisconsin–Madison's Division of Information Technology for their help in identifying the flaw. Although that appears to be uncited.
- DanBC 10y ago> Netgear stopped doing it. Netgear issued patches for the devices. Most people never update their server firmware, and we're talking about over 700,000 devices. The university still gets considerable traffic. https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#NETGEAR_and_the_University_of_Wisconsin.E2.80.93Madison https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#NE...
- agumonkey 10y agoCan't the university throttle non local connections ? (netgear could have provided the hardware ;)
- zamfi 10y agoNot necessary -- fortunately these routers all used source port 23457 for their NTP packets, making them trivially easy to block. I do recommend reading the incident report posted above if you have an interest in network operations, it's quite interesting!
- feld 10y agoYou still have to receive the traffic to block it.
- niij 10y agoIt is a public NTP server. Throttling public defeats the point.
- late2part 10y agoNot really. If an NTP client gets 1/10th of the updates that it wants, it will still keep reasonably good time.
- nickysielicki 10y agoKeeping a table of clients and their last-updated time is probably more expensive than just sending them a response.
- duaneb 10y agoWow, I never realized operators couldn't push fixes to their routers without permission. The internet is indeed a tragedy of the commons: trivial to ruin, but a Sisyphean task to fix.
- _delirium 10y agoSome operators do, mostly ISPs that lease routers to customers and retain a way to push firmware updates to them (for example, Comcast does this). But router manufacturers typically don't touch the device once it's out of their hands.
- jlgaddis 10y agoNote that cable modems (all of them, not just from Comcast) download their configuration from the provider every time they boot up. Ironically (since it uses TFTP, for one), this is called "secure provisioning". They might give you a web interface where you can configure certain settings (e.g. integrated Wi-Fi) but the ISP ultimately has at least some control over any cable modem connected to it.
- njharman 10y agoMost admins would consider having network infrastructure's firmware change outside of their control a bug/misfeature. Not to mention most devices would require reboot to apply change. And to be able to remotely change the code running a HUGE security issue.
- duaneb 10y agoSure. It's also why the internet is super vulnerable to 0-days.
- kalleboo 10y agoThe vast majority of admins don't even know that they're admins. They bought or received a cheap Netgear router, plugged it in, and never touched it again, except to maybe turn it off and on again when the internet was slow/down. If you're an admin who cares about their infrastructure, you're not using a bargain-basement Netgear router, and if you are, you'll have gone through every single menu and seen the auto-update option.
- quickben 10y agoThe university commitment to still serve the public is admirable.
- BoorishBears 10y agoThe most frustrating part of things like this, and that Snapchat issue, is the largest abusers could probably swallow the cost of their own NTP server usage as a rounding error to their bottom line.
- wmccullough 10y agoIt must be "rounding error" as an example for everything week on HN
- Spooky23 10y agoHats off to everyone contributing to public services like this. My then company wanted to give back ny doing this many years ago and it was an eye opening experience. We had troubles almost immediately with utilization and script kiddies. The company ended up only doing it for a relatively short period and ended up making contributions to projects instead
- brian_cunnie 10y ago[author] Thanks, Spooky23. As you pointed out, contributing to the community takes more time than originally expected.
- zanchey 10y agoOur student-run computing club added a machine to the pool and melted the University's firewall. Oops.
- jlgaddis 10y agoHonestly, that's the University's fault then. Properly configured, it should've had very little noticeable effect on the firewall (i.e. "permit udp any host 10.11.12.13 eq 123") as there's no need to do any inspection or tracking state ... ... unless they saturated the available bandwidth but, really, that's a different issue (although also preventable!).
- sigio 10y agopart 2: https://blog.pivotal.io/labs/labs/ntp-server-costing-500year-part-2-characterizing-ntp-clients https://blog.pivotal.io/labs/labs/ntp-server-costing-500year...
- rincebrain 10y agoI wish he'd explained somewhere how they leapt to examining virtualized NTP clients, or what they ultimately did (since there's no part 3 that I can find).
- wattt 10y agoI too was expecting some sort of conclusion.
- brian_cunnie 10y ago[author] My bad. Now that the post has been picked up on HN, I'll try to write up a conclusion.
- spydum 10y agoVirtualization and time sync have had notorious problems. One ugly work around was frequent NTP polling and adjustments. NTP has a min and max poll interval, and it determines how frequently it should poll automatically based on how far it sees drift happening. If it drifts pretty fast, it will quickly gravitate to the minpoll value, which is exactly what they show in their first graph: tons of polling at the minfrequency for certain hypervisors.
- raverbashing 10y agoI wonder, should this be solved with local NTP servers that resolve from the name pool.ntp.org?
- rhizome 10y agoI've found it to be good practice to run a single/few stratum 2 node(s) to serve local resources. To whatever degree, it's usually more important that these resources be more in sync with each other than with a satellite, which is fostered by having as few nodes as possible trying their luck over the internet to bogged down public stratum 1 NTP sources, instead configuring them to use a single source of time from a box more in its vicinity.
- Faaak 10y agoIt was mainly due to the poorly coded snapshat program: https://news.ntppool.org/2016/12/load/ https://news.ntppool.org/2016/12/load/ EDIT: this post was indeed from 2014. My bad then. however the same issue started again two weeks ago (~17 dec 2016).
- mw6621 10y agoSeems like the post was written in 2014 while the Snapchat NTP issue was more recent.
- sigio 10y agothis post was from 2014, so I doubt it was this months snapchat issue
- minsight 10y agoIt was mainly due to Virtualbox querying every 64 seconds.
- brian_cunnie 10y agoHere is a visual representation of the affect of the snapchat broken-ness on my NTP server: https://cloud.githubusercontent.com/assets/1020675/21468123/ac026274-c9d2-11e6-8334-2f56e9c9d20f.png https://cloud.githubusercontent.com/assets/1020675/21468123/... Note that inbound traffic which was steady at ~4k packets/sec spikes as high as five times as much. Also note that the snapchat traffic followed a circadian rhythm (much higher traffic during the daytime).
- libeclipse 10y agoAn interesting theory in one of the comments. > I wonder if Puerto Rico has run out of its pool of IPv4 addresses. After Europe and Asia, just this month Latin America as well, have exhausted their IPv4 pools, many local ISPs have resorted to using NAT to deal with the scarcity of addresses (of course, after years procrastinating IPv6 and pretending that this day wouldn't come about). Given that the source is a Puerto Rican ISP, and one of the offending addresses from a small /21 network, it's possible that NAT is to blame. As ISP NAT increasingly becomes more prevalent, this is going to be rather touchy to deal with abuses. For is it an abuser or just several innocent users behind a NAT?
- tyingq 10y agoI'm not understanding why NAT would cause it. I could see something like a misconfigured forwarding DNS cache causing it. Where it only queries pool.ntp.org once, and continues returning the result in the same order (with pivotal's ip at the top of the list) to a large number of querying clients. Then, perhaps, if there are a bunch of natted clients behind one ip? NAT, on it's own, without some other contributing factor, shouldn't cause this.
- discordianfish 10y agoNAT wouldn't cause it but hide that in fact those are many client all having the same source IP. Of course, that wouldn't explain why they observed a general increase in traffic.
- rupellohn 10y agoCould this have been the result of an NTP amplification attack? https://www.us-cert.gov/ncas/alerts/TA13-088A https://www.us-cert.gov/ncas/alerts/TA13-088A
- moxious 10y agoArticle said no, because the traffic was symmetrical and not lopsided. If this had been part of an attack you'd expect to see far more outgoing bandwidth than incoming.
- ddorian43 10y ago... because it's hosted on the cloud and you have no amount of free bandwidth with your vps ?
- johansch 10y agoBecause you use AWS and they charge insane fees for outgoing bandwidth.
- z92 10y agoThis $500 would have been $60, on a Digital Ocean box. DO has 1 TB/month limit. Their usage was 300 GB/month.
- brian_cunnie 10y ago[author] Digital Ocean is a great deal! Thanks for pointing that out. The reason I use {aws,azure,google} to host my NTP servers is that my day job is developing a VM orchestrator (BOSH) for Cloud Foundry, and BOSH doesn't support Digital Ocean yet (AFAIK). But that's a personal choice, and an admittedly expensive one.
- twic 10y agoFunnily enough, last time I had NTP troubles, it was on a BOSH deployment. BOSH wants to use ntpdate to keep time (because real NTP is too normal?) and we'd overzealously configured the security groups around our NAT setup so the machines couldn't reach their NTP server. Whoops. It'd be nice if BOSH could have detected this problem and warned us about it, ideally during deployment. But it'd be even nicer if we didn't suck at configuring AWS. If you could fix either of those things, that would be great!
- diegorbaquero 10y agoWhat about amazon lightsail?
- ilaksh 10y agoIt also runs on AWS so even though they try to trick you into thinking you don't pay for bandwidth you do. Also small EC2 instances are throttled so their server would probably not function most of the time.
- tyingq 10y agoI wonder if they somehow mistakenly joined their server to the region-specific pr.pool.ntp.org group[1]. At the moment, that pool exists, but has no servers in it. So, if you were the only server in the pool, perhaps you would get a lot of Puerto Rican traffic? [1]http://www.pool.ntp.org/zone/pr http://www.pool.ntp.org/zone/pr
- matt_wulfeck 10y agoSomeone else commented in the article, but it's probably related to Puerto Rico ISPs using a NAT because of lack of IPv4 address space. That single ip is probably many many people.
- tyingq 10y agoThat one doesn't make sense to me. Most people don't have ntp configured to point to ntp.pool.org...they are mostly PC's pointed at time.windows.com. And, the pool is big enough anyway that it would spread the load from a relatively small island pretty well. NAT could be a small part of it, but there's a different primary cause.
- matt_wulfeck 10y agoIt doesn't necessarily need to be Windows making the calls. Cell phones use a NAT typically and there was recently an issue with Snapchat DDoSing NATs.
- tyingq 10y agoThat's an example of a "different primary cause". It's not NAT in that case, it's an app using a library with terrible defaults.
- lucb1e 10y agoI'm not sure if I've missed it, but is the question (from the title) ever answered? The discrepancy between expected traffic volume and actual traffic volume is huge and seemingly unexplained.
- brian_cunnie 10y ago[author] My bad — I never wrapped it up. Thanks to the HN interest, I'll try to write Part 3 over the winter break. The short version is this: it's gonna cost a couple of hundred dollars to run a 1Gbe NTP server in pool.ntp.org, but you can tweak the ntp.conf to save ~$100.
- fxlv 10y agoI'm surprised there are no comments about the fact that these guys decided to run NTPD on a VM.
- cmrx64 10y agoI had the same thought. Does the world really need another (presumably) stratum-3 server running in Amazon's cluster, when Amazon already runs a pool of stratum-2 servers? ([0-3].amazon.pool.ntp.org).
- late2part 10y agot1.micro to boot - with more than the usual nondeterministic statmuxing and arbitrary "fairness" policy.
- brian_cunnie 10y ago[author] Yeah, maybe that's why my AWS instance is the most jittery of my 4 timeservers (Google, Hetzner, Azure).
- late2part 10y agoYour website says: Pivotal bridges the Silicon Valley state of mind, modern approach and infrastructure with your organization’s core expertise and values. Who we are and what we do together can reshape the world Your article suggests otherwise.
- Jabdoa 10y agoSecond part is here: https://blog.pivotal.io/labs/labs/ntp-server-costing-500year-part-2-characterizing-ntp-clients https://blog.pivotal.io/labs/labs/ntp-server-costing-500year...
- ChuckMcM 10y agoI think this is great look at walking through the analysis. I too experienced a huge spike in NTP traffic in 2014 but it was because of people exploiting NTP for reflection attacks to DDOS other parties. The forced me to use a GPS module and a Beaglebone Black as an internal time server (which has been great)
- daveguy 10y agoI have a few questions about that if you have a minute: What GPS module did you go with and is it still available? Did you have problems getting signal inside (need to be by a window, run an antenna, etc)?
- ChuckMcM 10y agoI used the Adafruit "ultimate" GPS module (https://www.adafruit.com/product/746 https://www.adafruit.com/product/746) which has the 1pps output and can connect to an external antenna. Then I got this antenna (https://www.adafruit.com/product/960 https://www.adafruit.com/product/960) and this adapter (https://www.adafruit.com/product/851 https://www.adafruit.com/product/851). Soldered a header connector to Beaglebone protocape (https://www.adafruit.com/product/572 https://www.adafruit.com/product/572), wired it to the serial port and PPS to the GPIO pin (just like this: https://web.archive.org/web/20131209092059/http://the8thlayerof.net/2013/12/08/adafruit-ultimate-gps-cape-creating-custom-beaglebone-black-device-tree-overlay-file/ https://web.archive.org/web/20131209092059/http://the8thlaye...). I put the GPS antenna on my window sill, it has no problem at all staying locked. My plan had been to stick it outside the window but turned out not to be necessary.
- jlgaddis 10y agoI've also used the Adafruit module, although with a Raspberry Pi. FWIW, the PPS signal seemed to be a little off from another receiver though (Garmin GPS 18x LVC). I never tried to pinpoint the issue but I strongly suspected the Pi. No noticeable issues w/ the antenna inside on a window sill (should be fine as long as you can see four satellites). I haven't tried it myself but I've heard of several other good experiences w/ the BeagleBone Black. The Garmin seemed to work the best for me, although it is a little more expensive. I was strongly considering putting a few of them in $work's (private) facilities as a fun, nerdy project but I never got around to actually doing it. The Garmin with a BBB might very well be a great combination for that. One other thing: make sure you use a "real" serial (or parallel) port -- not a USB to serial adapter!
- Animats 10y agoThis is the Snapchat bug reported yesterday, right? Incidentally, how is AWS dealing with the leap second next week? Google is going to have their time servers start to run fast around 20 minutes in advance of the leap second, so they're back in sync at 00:00:60 UTC.
- jeffbarr 10y agoDetails on AWS at https://aws.amazon.com/blogs/aws/look-before-you-leap-december-31-2016-leap-second-on-aws/ https://aws.amazon.com/blogs/aws/look-before-you-leap-decemb...
- jlgaddis 10y agoNope, this is from 2014. If memory serves, Google's "smoothing" the second out over (I think) a 24-hour period. I don't recall the exact time period off the top of my head but it's much, much longer than 20 minutes.
- feld 10y agoWhy are people joining VMs to the NTP pool? These servers should be identified by address space and blacklisted.
- foota 10y agoWhy?
- deleted 10y ago[deleted]
- therein 10y agoBecause VMs themselves might not be able to keep track of time accurately (potentially inconsistent tickrate) the way that a bare-metal setup would be able to. That's why they should be mere consumers (as in sync their time to whatever the remote says rather than contribute to the pool).
- feld 10y agoCorrect, unless you have a very specific VM configuration where you are truly dedicating a CPU/core to a VM, it's not fit for being an NTP server.
- jelder 10y agoWhy would anyone run an authoritative time service on a virtual server in the first place? My experience is that system time suffers greatly from noisy neighbor.
- Steeeve 10y agoThis is a 2 year old article. Where's the follow up? What did they end up doing? EDIT: AHA! Part 2: https://blog.pivotal.io/labs/labs/ntp-server-costing-500year-part-2-characterizing-ntp-clients https://blog.pivotal.io/labs/labs/ntp-server-costing-500year...
- technion 10y agoI ran an NTP server on a Raspberry Pi for some time. The bottleneck I kept hitting was the 65535 NAT translation limit on my Cisco router, at which point, load was quite manageable on the Pi. It's extraordinary how much traffic one cheap device could service.
- deleted 10y ago[deleted]