4 ms·

As someone that has "repurposed cartographic tools" to analyze spatiotemporal datasets, what specifically do you feel is missing from current tools?

jandrewrogers 10y ago
I spent many years repurposing cartographic tools myself. :-) Three big missing features off the top of my head: - Insufficiently correct and high precision computational geometry, which compounds with the iterative/recursive nature of many complex sensor analytics. Many people don't notice unless they ground truth their analytic process; I learned this the hard way. For many industries, 1% cumulative computational error is a catastrophic bug for analytics and the reality can be much worse in many common systems. - Lack of first class tessellation types and operators. Once your data scientists have them, they'll wonder how they lived without them. Such things are completely useless for cartography and therefore don't exist in those platforms. - High-performance computational geometry. This is particularly noticeable if you work with sensor coverages (like drone data). Your typical cartographic system has serious difficulty joining a few terabytes of complex polygons, but these are tiny data sets for many remote sensing sources. It literally takes weeks or months to run these types of queries. You can optimize this to be much faster but there was no pressing market need in cartography and cartographic systems aren't designed for scale-out generally.
- huevosabio 10y ago
  Interesting! I'll ponder over these; thanks for the insight!
- marmaduke 10y ago
  Aren't these addressed by CGAL? Or you mean, you'd want database primitives using CGAL?
  - jandrewrogers 10y ago
    Unfortunately CGAL is not suitable, and yes you do need to support it in the underlying representation and database. The litmus test for a suitable computational geometry engine is the ability to do arbitrary ellipsoid computational geometry at something like one part per trillion error, and the ability to compute intersections of very complex polygons on that surface quickly. On that last point, you can execute an effective denial of service attack on most GIS databases with a well-crafted set of polygons and polygon intersection queries. It doesn't even require malicious intent. (Yet another thing I learned the hard way.)
    - peter_silie 10y ago
      I am not sure if all those goals can be combined. For example, the intersection of n polygons could have exponentially many segments. To prevent denial of service, accuracy has to be sacrificed, but "one part per trillion error" sounds hard. As another example, representing the intersection between two line segments without error requires a data type that has three times as many bits as the x/y components of their endpoints, so rounding has to happen here, too. The only solution I can come up with that would not have those problems would be to subdivide a plane into grid cells of size one trillionth of whichever unit, rasterize all polygons into it do the Boolean operations at pixel-level, but that would require huge amounts of memory and processing power. And additionally it would be vulnerable to a DOS attack where the attacker sends many very large overlapping rectangles. Lastly, huge amounts of data would require many computers to be able to work on this problem in parallel, but if an attacker sends only overlapping polygons, parallelization would be very tricky if not impossible.
    - repsilat 10y ago
      Any chance you could give an example use-case or three? Sounds fascinating -- I use PostGIS and find it to be slower than I'd like day to day, but it sounds like your uses demand a lot more precision than I have call for.
- playing_colours 10y ago
  I am late in this conversation, I would like to dig more into the IoT and the needs for tooling for spatial analysis. Could we talk more about the problems you are facing? My email is on the profile page. Thank you!

jandrewrogers 10y ago

I spent many years repurposing cartographic tools myself. :-) Three big missing features off the top of my head: - Insufficiently correct and high precision computational geometry, which compounds with the iterative/recursive nature of many complex sensor analytics. Many people don't notice unless they ground truth their analytic process; I learned this the hard way. For many industries, 1% cumulative computational error is a catastrophic bug for analytics and the reality can be much worse in many common systems. - Lack of first class tessellation types and operators. Once your data scientists have them, they'll wonder how they lived without them. Such things are completely useless for cartography and therefore don't exist in those platforms. - High-performance computational geometry. This is particularly noticeable if you work with sensor coverages (like drone data). Your typical cartographic system has serious difficulty joining a few terabytes of complex polygons, but these are tiny data sets for many remote sensing sources. It literally takes weeks or months to run these types of queries. You can optimize this to be much faster but there was no pressing market need in cartography and cartographic systems aren't designed for scale-out generally.

huevosabio 10y ago
Interesting! I'll ponder over these; thanks for the insight!
marmaduke 10y ago
Aren't these addressed by CGAL? Or you mean, you'd want database primitives using CGAL?
- jandrewrogers 10y ago
  Unfortunately CGAL is not suitable, and yes you do need to support it in the underlying representation and database. The litmus test for a suitable computational geometry engine is the ability to do arbitrary ellipsoid computational geometry at something like one part per trillion error, and the ability to compute intersections of very complex polygons on that surface quickly. On that last point, you can execute an effective denial of service attack on most GIS databases with a well-crafted set of polygons and polygon intersection queries. It doesn't even require malicious intent. (Yet another thing I learned the hard way.)
  - peter_silie 10y ago
    I am not sure if all those goals can be combined. For example, the intersection of n polygons could have exponentially many segments. To prevent denial of service, accuracy has to be sacrificed, but "one part per trillion error" sounds hard. As another example, representing the intersection between two line segments without error requires a data type that has three times as many bits as the x/y components of their endpoints, so rounding has to happen here, too. The only solution I can come up with that would not have those problems would be to subdivide a plane into grid cells of size one trillionth of whichever unit, rasterize all polygons into it do the Boolean operations at pixel-level, but that would require huge amounts of memory and processing power. And additionally it would be vulnerable to a DOS attack where the attacker sends many very large overlapping rectangles. Lastly, huge amounts of data would require many computers to be able to work on this problem in parallel, but if an attacker sends only overlapping polygons, parallelization would be very tricky if not impossible.
  - repsilat 10y ago
    Any chance you could give an example use-case or three? Sounds fascinating -- I use PostGIS and find it to be slower than I'd like day to day, but it sounds like your uses demand a lot more precision than I have call for.
playing_colours 10y ago
I am late in this conversation, I would like to dig more into the IoT and the needs for tooling for spatial analysis. Could we talk more about the problems you are facing? My email is on the profile page. Thank you!