4 ms·
In contrast to his 4-digits-are-fine notions, let me offer a counterexample: http://scissor.com/transient/destination_heatmap/ http://scissor.com/transient/dest
by wpietri 7y ago
In contrast to his 4-digits-are-fine notions, let me offer a counterexample: http://scissor.com/transient/destination_heatmap/ http://scissor.com/transient/destination_heatmap/
I am working on a Twitter bot called @sfships, which monitors the comings and goings of large ships in the San Francisco Bay: https://twitter.com/sfships https://twitter.com/sfships
As part of that, I generated the above map from AIS data. [1] It's basically where ships stop. If you zoom in on the San Francisco waterfront, you will see a grid of dots. That's because the AIS protocol stores lat/lon as minutes/10000 [2], throwing away more detailed information.
This is adequate for its initial intended purpose, which is putting ships on radaresque displays so that ships don't hit one another, etc. But it produces all sorts of artifacts and issues when one tries to use the data more broadly.
And in case anybody is interested in playing with this, I have written a python parser for AIS data with a bunch of Unixy command line tools, including one that just turns the weird 90s protocol into more modern JSON: https://github.com/wpietri/simpleais https://github.com/wpietri/simpleais
[1] https://en.wikipedia.org/wiki/Automatic_identification_system https://en.wikipedia.org/wiki/Automatic_identification_syste...
[2] https://gpsd.gitlab.io/gpsd/AIVDM.html#_types_1_2_and_3_position_report_class_a https://gpsd.gitlab.io/gpsd/AIVDM.html#_types_1_2_and_3_posi...
- cpncrunch 7y agoAnother example: I was looking up my property details yesterday, and wanted to get exact coords for my property line. Their web app lets you click on the map to get coords, but it only displays degrees/minutes/seconds (with no decimal point), so it wasn't much use.
- widforss 7y agoAre you certain that the data is accurate enough to support better precision? Where I live we have avcess to vectorized property maps, but the data source may be a digitized 200 y/o map.
- wpietri 7y agoMy mom used to do a lot of work with property maps for oil and gas companies and at least in the US they were very precise. They had to be, as you don't want to build an expensive well and then later have to move it 3 feet. Or, worse, realize you didn't buy all the mineral rights and then have to pay somebody.
- tony_cannistra 7y agoI used to have a job where I analyzed global ship tracking data (AIS and other less-open (i.e. more classified) data streams) for law enforcement and environmental conservation purposes, and learned one thing: man is AIS data a bitch. This is especially true when you're dealing with potential bad actors who can spoof their transmissions, combined with incredibly poor vessel metadata databases and strange satellite AIS coverage issues. If there are any data scientists out there looking for a real head-basher, start an AIS project.
- iooi 7y agoWhy do bad actors spoof transmissions? Smuggling?
- wpietri 7y agoI'm sure Tony knows more, but one I've heard about is fishing. It's not legal to just fish anything anywhere. Poachers have an incentive to avoid monitoring and enforcement.
- jandrewrogers 7y agoMany reasons, and a significant percentage of all AIS transmissions are dodgy. Smuggling is just one reason. Other common reasons include illegal fishing in protected waters, covert meetings (e.g. a couple yachts coming together in the middle of the ocean), spy ships by state actors trying to look like ordinary commercial traffic, and misrepresenting state of shipping assets for commercial leverage in contract negotiations.
- wpietri 7y agoI would love to read more about this. Any suggestions?
- jandrewrogers 7y agoThese are things you learn by doing in-depth analysis of AIS and related data sources. As far as I know, no one writes about it. Physical world data sources are full of interesting patterns and anomalies. Few people ever look because the data sources are challenging to work with and very different from Internet data, which is the only kind of data most people are used to.
- dhritzkiv 7y agoI wonder if introducing some randomized jitter would combat the regular/grid-like placement of the dots.
- duskwuff 7y ago> If you zoom in on the San Francisco waterfront, you will see a grid of dots. And yet, some of those dots are significantly further inland than rounding can account for. There's one faint point near 4th and King, for example, and a strong one on Spear between Howard and Mission. All signs point to the source data being imprecise enough that transmitting it at a higher precision wouldn't improve the results.