4 ms·

Depends on the kind of data. Check out cartopy, descartes, rasterio, geopandas.

by llao 8y ago

dagw 8y ago
Another general point is that once you're working with sufficiently large data sets you no longer have a GIS problem, but a Big Data/High Performance Computing problem and you need to start working with tools available in those domains together with the GIS tools. Like in the article. If you have a problem that takes 11 days to run in QGIS, then you shouldn't be using QGIS, but a tool that is designed for processing large amounts of data. 1-2 million points might be a lot from a GIS tools perspective, but is absolutely nothing from a big data/HPC perspective, so be sure to check what those guy are doing.
- kaybe 8y ago
  So far I've been able to avoid that by only looking at chunks of the datasets. Can you give me some keyworks for what to look for/point me in the generally right direction? (I dread the day my supervisor wants to look at all the data at once.)
  - dagw 8y ago
    First rule. PostGIS is your friend. The PostGIS people have done a lot of work with making processing large sets of GIS data fast and easy, so always start there. If for some reason that doesn't work... As a general rule always try to ask yourself, what am I actually trying to do, mathematically, and then ask how would someone go about solving that math problem, if you didn't tell them it was a GIS problem? Much of raster analysis, for example, is just a combination of matrix math and convolutions. So look up how the numerical analysis people do matrix math and convolution on huge matrices. In python, for example, you have tools like numexpr for fast elementwise transformation of matrices and via numpy/scipy you can call BLAS and LAPACK. If you're dealing with rasters that don't fit in memory take a look at solutions like Dask. Same principle applies when dealing with vector data. Many problems are 'just' graph theory, so find out which library that graph theory people use to solve that sort of problem on large graphs and use that instead. Or if a problem reduces to a line/polygon intersection problem, well that's just raycasting, and the games industry has spent a lot of effort in making that really really fast. Finally, learn to use the underlying libraries from the command line and scripts rather than via the GUI and how to divide that work across several processors/machines. GDAL + GNU parallel from the command line will transform 1000 rasters faster than QGIS could ever hope to do.

Depends on the kind of data. Check out cartopy, descartes, rasterio, geopandas.

by llao 8y ago

Depends on the kind of data. Check out cartopy, descartes, rasterio, geopandas.

dagw 8y ago

Another general point is that once you're working with sufficiently large data sets you no longer have a GIS problem, but a Big Data/High Performance Computing problem and you need to start working with tools available in those domains together with the GIS tools. Like in the article. If you have a problem that takes 11 days to run in QGIS, then you shouldn't be using QGIS, but a tool that is designed for processing large amounts of data. 1-2 million points might be a lot from a GIS tools perspective, but is absolutely nothing from a big data/HPC perspective, so be sure to check what those guy are doing.

kaybe 8y ago
So far I've been able to avoid that by only looking at chunks of the datasets. Can you give me some keyworks for what to look for/point me in the generally right direction? (I dread the day my supervisor wants to look at all the data at once.)
- dagw 8y ago
  First rule. PostGIS is your friend. The PostGIS people have done a lot of work with making processing large sets of GIS data fast and easy, so always start there. If for some reason that doesn't work... As a general rule always try to ask yourself, what am I actually trying to do, mathematically, and then ask how would someone go about solving that math problem, if you didn't tell them it was a GIS problem? Much of raster analysis, for example, is just a combination of matrix math and convolutions. So look up how the numerical analysis people do matrix math and convolution on huge matrices. In python, for example, you have tools like numexpr for fast elementwise transformation of matrices and via numpy/scipy you can call BLAS and LAPACK. If you're dealing with rasters that don't fit in memory take a look at solutions like Dask. Same principle applies when dealing with vector data. Many problems are 'just' graph theory, so find out which library that graph theory people use to solve that sort of problem on large graphs and use that instead. Or if a problem reduces to a line/polygon intersection problem, well that's just raycasting, and the games industry has spent a lot of effort in making that really really fast. Finally, learn to use the underlying libraries from the command line and scripts rather than via the GUI and how to divide that work across several processors/machines. GDAL + GNU parallel from the command line will transform 1000 rasters faster than QGIS could ever hope to do.