4 ms·
Next time they better talk to experts for their own sanities' sake...! > We set the interpolation process running in QGIS on a Mac Pro and, a mere 11 days late
by llao 8y ago
Next time they better talk to experts for their own sanities' sake...!
> We set the interpolation process running in QGIS on a Mac Pro and, a mere 11 days later, found ourselves the proud owners of a raster 250m2 grid layer
Probably done in minutes to hours with a properly setup PostGIS database and one query. "Data Journalists" seem to love it when things take long, thinking they are doing something super innovative and novel. Often it's just that they were using the wrong hammer...
> 1. Don’t use GeoTIFFs for geometric visualization
What? I am not sure what exactly the image is supposed to show but it seems like some rescaling/filtering. Not the fault of GeoTIFF.
Why not render the raster tiles locally? gdal2tiles or Tilemill can do that for you.
Why does Mapbox recommend LZW for compression? DEFLATE is usually much smaller and with the horizontal differencing predictor it should be even more smaller.
> 2. Your GeoJSON can probably be way smaller
Does Mapbox not support TopoJSON? That helps a lot for local data with quantisation.
> Because nothing in GIS is straightforward, attempting to round the values of 2.7m polygons using the field calculator in QGIS invariably resulted in the application crashing, even on a high-end Mac Pro. After several attempts using QGIS 2.16, 2.18 and 3.2,
Did they ask the community or even just provide a bug report? This should not happen from a QGIS fault.
- dagw 8y agoYea, as someone who does this sort of thing for a living much of that was quite painful to read. I mean hats off to them for getting a nice result, but they could have done it so much more efficiently by using the right tools.
- reaperducer 8y agoWhile I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on. Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions. As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightmare. Especially when you're going in for the first time. I think they did a pretty good job considering they have to be jacks of all trades.
- dagw 8y agoYea, I guess I was being a bit harsh. They delivered a nice looking result and at the end of day that's what counts. And honestly thinking back at the first several times I did something similar I can't really say I did a more efficient job.
- llao 8y agoMy OP was very negative, sorry. Absolutely! Solving what you are set out to do is a great result and I am super happy for them and their end result! But this state of documentation (as you call it, I would also/rather consider all the needed background knowledge and lingo to be a hurdle) is all the more reason to just document their goal, their data(s), their abilities and capabilities and ask an expert for 15 minutes of their time for input and pointers. That is so much more efficient!
- kaybe 8y agoI'd be interested in mapping solutions for python. What is your preferred workflow, HN?
- deleted 8y ago[deleted]
- llao 8y agoDepends on the kind of data. Check out cartopy, descartes, rasterio, geopandas.
- dagw 8y agoAnother general point is that once you're working with sufficiently large data sets you no longer have a GIS problem, but a Big Data/High Performance Computing problem and you need to start working with tools available in those domains together with the GIS tools. Like in the article. If you have a problem that takes 11 days to run in QGIS, then you shouldn't be using QGIS, but a tool that is designed for processing large amounts of data. 1-2 million points might be a lot from a GIS tools perspective, but is absolutely nothing from a big data/HPC perspective, so be sure to check what those guy are doing.