9 ms·
Tools I Recommend for Building Geospatial Web Applications
- eric_khun 7y ago- Which API would you recommend to get accurate transit information data? - Is there any article talking about and path optimization including time constraint? I've done something homemade (a mix of bird flight distance and time to destination) it looks "okay" but I'm sure can be better. I've just built a tool to calculate directions for long flights layover [1], but the Google Maps API Directions sometimes lack of those data (Tokyo/Singapore/Istanbul for example). Any recommendations appreciated! [1] https://travelhustlers.co/layover-in/ https://travelhustlers.co/layover-in/
- jimktrains2 7y agoI would see if your system has a gtfs feed. https://developers.google.com/transit/gtfs/ https://developers.google.com/transit/gtfs/ You could use pgrouting for routing. https://pgrouting.org/ https://pgrouting.org/
- rohansingh 7y agoGTFS and GTFS-RT definitely work, but in practice it seems each transit agency implements them a little differently. I'm actually a fan of the HERE Transit API. They've done the work of interfacing with a bunch of providers, and you can query their whole dataset with a REST API. Unfortunately it's proprietary and only free at small volumes. But I found it much easier to hook up than working directly with a transit agency. That said, their coverage can be a bit random. For example, I ended up having to do direct GTFS-RT anyway because HERE doesn't seem to have real-time data for NYC subways.
- mvexel 7y agoNot a transit routing API, but transit.land offers aggregated transit network data as a service. https://transit.land/ https://transit.land/
- tmostak 7y agoAlso worth checking out OmniSci, (https://www.omnisci.com https://www.omnisci.com), formerly known as MapD, which can query and visualize tens of billions of geospatial records in sub-second timeframes. See here for an interactive demo of 11.6 billion ship AIS records: https://www.omnisci.com/demos/ships https://www.omnisci.com/demos/ships. Note that the backend database is open source, but the rendering engine and web front end are part of the paid offering. [Disclaimer: I work there]
- john_minsk 7y agoHi. Checked out the demo. Super interesting! If I want to become a developer and sell the projects on your platform: How can I start? What is proximate pricing schema?
- tmostak 7y agoHi John, thanks! One option is our cloud (pricing is on the page). https://www.omnisci.com/cloud https://www.omnisci.com/cloud . You can also spin us up from the AWS, Azure, or GCP marketplaces. If you need an on-prem option, feel free to reach out: info@omnisci.com.
- john_minsk 7y agoAlso, I can see that you have ODBC interface for the DB. Any DEMO database I can play with?
- tmostak 7y agoIf you sign up for a cloud trial, we can provide you with an ODBC interface. Other options are using our JDBC or our Python/JS interfaces, which are in our open source: https://github.com/omnisci https://github.com/omnisci .
- leowoo91 7y agoI'm surprised Redis hasn't been mentioned for simple use cases. It has built-in distance and search functionality: https://redislabs.com/redis-best-practices/indexing-patterns/geospatial/ https://redislabs.com/redis-best-practices/indexing-patterns...
- Stevvo 7y agoIt's essential if you are pulling any significant amount of data, e.g. shapes for dividing a country into administrative districts will be a couple MB uncompressed pulling from a PostGIS database.
- thinkingemote 7y agoInteresting, looking over the docs I couldn't find anything beyond simple radius queries. It seems that all redis handles is points. How would you query from shapes with redis?
- Stevvo 7y agoJust using it as a response cache for GeoJSON.
- softwarelimits 7y agoThere is of course https://github.com/sshuair/awesome-gis https://github.com/sshuair/awesome-gis I would be interesting to read more about why a tool was used, also: what are the current limitations?
- jakobdabo 7y agoThanks for the recommendations. I need to create a polygon of specified width based on an arbitrary GPS track (which can cross itself multiple times). Can anybody recommend a JS library which is capable of doing this?
- Philnz 7y agoPerhaps check out TurfJS for client side spatial buffers and point in polygon functions. https://turfjs.org/ https://turfjs.org/ Also leaflet.js and openlayers.js for web mapping.
- randomgis 7y agoYou should be able to this JSTS I think. Create a line with the gps track, create a buffer with the specified distance, feed the resulting polygon to your mapping library. https://github.com/bjornharrtell/jsts/blob/master/README.md https://github.com/bjornharrtell/jsts/blob/master/README.md Sample use: http://bjornharrtell.github.io/jsts/ http://bjornharrtell.github.io/jsts/
- franky47 7y agoI've been using Mapbox for client and personal projects, and so far it's been great but I'd like to give Leaflet a try (for the sake of less dependence on commercial companies for personal non-profit projects), anybody here went down the same road, especially in the context of React applications ?
- jbeales 7y agoLeaflet doesn't provide tiles, but lets you get tiles from a variety of sources including OSM, & Mapbox, so you still need a tile provider, or need to render your own tiles.
- lbutler 7y agoIf you want to use vector maps you should also check out Mapbox GL JS. I use their tiles but can use other providers or generate and host your own, which I also do. I've been making spatial apps for probably 8 years now and moved from Google maps and Leaflet towards vector maps and I don't think I'd ever go back. https://github.com/mapbox/awesome-vector-tiles/ https://github.com/mapbox/awesome-vector-tiles/
- urschrei 7y agoParticularly now that it's relatively straightforward to bake and serve your own MVT tiles, so you're not tied into the Mapbox service ecosystem if you don't need any of its other features (and you're happy with the scale / performance of your own tile server(s))
- lbutler 7y agoIf your data set isn't dynamic you don't even need a tile server. You can generate the tiles with tippecanoe [1] and serve them straight out of S3. When things get too large you can keep them in the mbtile file and serve them out with an express app. I've done it in a severless environment which scales well but would get crazy expensive for any high use. [1] https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe
- spython 7y agoDo you know of any tutorials on how to generate and host own (raster) tiles?
- lbutler 7y agoI would suggest starting with something static if you're just starting out. Best way to get a feel for how it works without having to worry about setting up a spatial database. I used this article myself to learn: https://geovation.github.io/build-your-own-static-vector-tile-pipeline https://geovation.github.io/build-your-own-static-vector-til... There are a couple more articles at the bottom of the awesome vector tiles I linked to before but haven't read them myself. Reading through the list is good as well because it's a good overview of the whole ecosystem. Edit: This article also shows a simple way to dynamically serve vector tiles directly from PostGIS. I didn't realise you could do it without the middlewear, time to look into doing it this way! https://medium.com/nyc-planning-digital/using-the-new-mvt-function-in-postgis-75f8addc1d68 https://medium.com/nyc-planning-digital/using-the-new-mvt-fu...
- pierredewet 7y agoI’ve used PostGIS with a Flask (and some js) front end when I was working on my MSc. (Mapping deprivation and services by postcode) Worked amazingly well and was relatively easy to get used to once I read the O’Reilly PostGIS book to get used to the function calls. As with any data project; munging the data into a usable format was the biggest hassle. Edit: used google maps for the display before they changed their licence. Haven’t updated to OSM yet...
- thibautg 7y agoSame. I use Flask because it's one of the lightest frameworks I could find. GeoJSON can be generated on the fly by PostGIS with very good performance. So Flask is just used as middleware to submit the query from URL params to PostgreSQL. For the front-end I usually use Leaflet with Mapbox (raster) tiles. I should switch to vector tiles but "@2x" raster tiles look good enough on HiDPI/Retina displays. I'm also an heavy QGIS user. I've managed to convince IT to put it in our internal Software Center so users can install it in one click without needing to be admin. It's becoming quite successful internally ;-). Take this Esri! However we still pay hundreds of k€ to Esri and nothing to QGIS. I've personnally donated a few euros but it's a really small amount compared to what we pay to Esri.
- ArtRichards 7y agoIve used elasticsearch geoapi in the past. Cant speak to its current status, but it was very helpful! https://www.elastic.co/blog/geo-location-and-search https://www.elastic.co/blog/geo-location-and-search
- sergiomattei 7y agoGeoDjango is really awesome. I’ve used it for a few things geospatial and I had no big troubles with it.
- rohansingh 7y agoReally interesting to see how stable these recommendations are over time. I built a commercial GIS system at a startup almost ten years ago, and GeoServer, PostGIS and OpenLayers were the go-to tools back then too. I left that startup soon after, but I heard that the backends for the GIS product really didn't have to be upgraded or revamped for years— PostGIS and GeoServer were just running solidly. Really a testament to the developers of those tools.
- based2 7y agohttps://medium.com/@mbostock/command-line-cartography-part-1-897aa8f8ca2c https://medium.com/@mbostock/command-line-cartography-part-1... Alt. for java devs: http://geoserver.org/about/ http://geoserver.org/about/ https://geotools.org/about.html https://geotools.org/about.html https://sis.apache.org/ https://sis.apache.org/
- alexhutcheson 7y agoIf you outgrow PostGIS, you may want to look into GeoMesa, which allows you to set up spatio-temporal indexing on top of NoSQL column stores including HBase and Cassandra: https://github.com/locationtech/geomesa https://github.com/locationtech/geomesa
- ohlookabird 7y agoThanks for the pointer, haven't heard about it. Spatial-temporal indexing sounds interesting! Do you happen to have any estimates when things start to break down for you in PostGIS? My largest table of PostGIS 3D geometries (line strings) has ~800,000,000 entries and most bounding box queries are still very fast (<100ms).
- alexhutcheson 7y agoNo, I'm not a PostGIS expert unfortunately. It probably depends a lot on the balance between reads and writes/updates.
- appleflaxen 7y agoThese are awesome tools. Do any of them support non-earth coordinates? I have a project where I want to do the equivalent of mapping data onto an asteroid (arbitrary shape, including the possibilities of concavities). What tools would work best / support that workflow?
- jofer 7y agoThat has nothing to do with these types of frontend tools and everything to do with projection libraries. In your case, all of the full libraries like proj4 already support it. What you're describing is a custom ellipsoid and datum. Plug that in, and you're good to go. You'll need to define the shape of your asteroid's equipotential surface through a reference ellipsoid dimensions and a datum that defines deviations from that ellipsoid. Making those is definitely non-trivial, but once you've got the datum grids made, the math is the same as it is for Earth. Also note that the datum is _not_ the shape of the asteroid directly. It's the 3D shape of a surface that's "flat" gravitationally. In other words, it's the shape of sea level. It's something that depends on the density distribution of the body and it's really not easy to measure, though it is easy to predict if you know the density distribution a-priori. The shape of the asteroid's surface is topography. That's independent of the coordinate system.
- rumanator 7y ago> That has nothing to do with these types of frontend tools and everything to do with projection libraries. Mapping frameworks need to support map APIs to render stuff, and some APIs such as Open Geospatial Consortium's WMS (OGC WMS) requires clients to specify coordinate systems. https://www.opengeospatial.org/standards/wms https://www.opengeospatial.org/standards/wms
- jofer 7y agoI'm unclear how that's relevant in this case... Yes projections need to be specified... The datum and ellipsoid are part of the projection. No, you won't be able to pass your custom datum along to an external service and have it recognize it automatically. That doesn't change the fact that you can define it locally and have your local tools recognize it. proj4 et al will happily let you define custom ellipsoids and register a new datum and work with projections on that ellipsoid and datum. You can view that just fine in QGIS et al. Also, WMS is pretty niche and not too widely used partly because it supports things like this use case and is therefore quite complex. I wouldn't bring WMS into this. However, if you want to, the projection specifications it uses _also allow for custom ellipsoids_. You can happily request things in a projection with a non Earth ellipsoid -- it's just a matter of whether the WMS server supports it.
- Game_Ender 7y agoIf you are looking for something to handle large amounts of data I would take a look at deck.gl [0]. It useful when you have some rich data you want to layer on top of a high quality map, with built in Mapbox GL integration [1]. 0 - https://deck.gl https://deck.gl 1 - https://deck.gl/#/examples/core-layers/geojson-layer-polygons https://deck.gl/#/examples/core-layers/geojson-layer-polygon...
- jandrewrogers 7y agoIt isn't clear how this handles large amounts of data? The examples are thousands of polygons, which while useful isn't "large". Drone data models and similar are often trillions of polygons. How do you scale-out the data model processing? As a tangent, deck.gl (first time I've looked at it) describes their 64-bit floating point computation as having "unparalleled accuracy". It is not possible to correctly compute some double precision geospatial relationships using double precision computation (it requires quad precision or greater to produce a correct double precision result) and other software systems, including in open source, do this correctly.
- Game_Ender 7y agoEverything is relative and in the Web usually large is quite a bit smaller than native, so here I assume “large” really meant “bigger than 2D Canvas or SVG could handle”. This example has a few 100k things [0] so you could judge by how well that works. Not sure about the 64 bit statement, but I assume it means relative to normal 32 GL shader math. Which is definitely not “unparalleled”, but I assume better than the truncated doubles of native JavaScript or the standard shader math. 0 - https://deck.gl/#/examples/core-layers/geojson-layer-paths https://deck.gl/#/examples/core-layers/geojson-layer-paths
- lsb 7y ago1) I've successfully put >1M points into a deck.gl point layer with cute hover effects, and on a modern computer I can easily get 30fps on a map in the millions of pixels. 1a) Large-scale map viz in the browser is not only challenging for in rendering (see https://github.com/uber/deck.gl/blob/master/docs/developer-guide/performance.md https://github.com/uber/deck.gl/blob/master/docs/developer-g... for details), but it's also challenging for data transfer. If you have a table of 7 variously-sized columns with a million rows of data, you'll probably want to send 7 arrays to the browser, just with the way WebGL is going to unpack parameters to your shaders, so you'll want something like Google's FlatBuffers, and FlatBuffers with named columns and more datatypes is the Apache Arrow format (which has excellent Javascript support, and strives to avoid copying buffers, which is what you want in high-performance applications). 2. By default, many of the deck.gl layers don't use 64-bit floats, they use 32-bit floats for lat/lng degrees, which supports many use cases for resolution and speed. In a WGS84 projection, with a base/meridian of (0°,0°), a point in northeast Russia represented by two IEEE754 32-bit floats (say: 70°N, 179°E) will be within roughly a meter of its original value (as per https://upload.wikimedia.org/wikipedia/commons/3/3f/IEEE754.png https://upload.wikimedia.org/wikipedia/commons/3/3f/IEEE754.... and as per the earth circumference is ~40M meters, 1° is at most at the equator ~ 40M/360 meters ~ 100 km, and a rounding error of 10^-5 on a float that's 10^2 means you'll be off by around a meter). The deck.gl layers allow you to provide your zero point, and you get 100x the resolution at 1 compared to 100 in the exponent+fraction format for floats (so pre-render your Δlat, Δlng in 64-bit float computations with standard 80-bit precision somewhere besides WebGL). If you have more than a 100 km x 100 km square for which you need sub-centimeter precision, computed at render-time, 32-bit floats aren't for you. Which brings me to speed: WebGL doesn't have 64-bit floats, or 64-bit ints, natively. You'll have to fudge the math in software in a shader, instead of the native add instructions on the GPU, and that is far slower. If you do need higher precision with only 32-bit integers, you can cluster your points around many different base/meridian points instead of the single (0,0), and take advantage of increased float precision on smaller numbers. Or use 64-bit floating computations with the standard 80-bit precision in Javascript, outside of GPU shaders, and compute millions of floats in Javascript typed arrays to send into deck.gl shaders.
- mlevental 7y agois there a js framework that has most (some) of the functionality of qgis?
- drewda 7y agoFor geographic and geometric operations in JavaScript: https://turfjs.org/ https://turfjs.org/ http://bjornharrtell.github.io/jsts/ http://bjornharrtell.github.io/jsts/ For projections: http://proj4js.org/ http://proj4js.org/ https://github.com/d3/d3-geo https://github.com/d3/d3-geo
- askytb 7y agoCan anyone recommend any tools for routing? Ideally with a couple sentence pros/cons breakdown
- sampo 7y ago> Can anyone recommend any tools for routing? Google's Operations Research Tools has implementations for some vehicle routing algorithms. https://developers.google.com/optimization/routing https://developers.google.com/optimization/routing
- chvid 7y agoSuppose one wants a simple world map with one or two levels of zoom and some interactive points plotted over it. What would be the recommended approach?
- thecodemonkey 7y agoGreat article! One thing that wasn’t mentioned was Geocoding — Converting between addresses and coordinates. This is also a very common need for geospatial web applications. Here’s a couple of options: geocode.earth -> Worldwide coverage geocod.io -> US/Canada only, but supports spreadsheets, data appends and batch geocoding smartystreets.com -> Ideal for USPS address verification Disclaimer: I’m the co-founder of Geocodio
- ProjectBarks 7y agoFor anyone who is looking for good 3D geospatial tools I recommend Cesium JS. It works great for pretty much any amount of data. cesiumjs.com (In full disclosure I did work as an Intern for CesiumGS)
- EGreg 7y agoWhy not just use geohash and text ranges?
- shade 7y agoI have a question that's kind of adjacent to this. I've wanted to play with mapping and especially with geolocating data, but at least right now, I don't want to put much money into it. Are there any reasonably decent tools for doing bulk address to latitude/longitude geolocation, preferably something I could pull down and run in Docker containers on my desktop machine? I'm aware of the Data Science Toolkit, but it seems a bit out of date.
- jijik 7y agoWe started building a web geospatial platform 5 years ago. We migrated from GeoServer to Mapbox VT but still using Leaflet because it brings us a better interactivity on the map. During the time, we've build a self service GIS SaaS product. In contrast with a layers based GIS tools, it supports a complex Data models as you know from BI tools. If you are interested, try our free trial: https://www.clevermaps.io/free-trial https://www.clevermaps.io/free-trial and documentation: https://clevermaps.atlassian.net/wiki/spaces/CDD/overview https://clevermaps.atlassian.net/wiki/spaces/CDD/overview
- tobilg 7y agoIn case someone needs geofencing capabilities, have a look at http://www.tile38.com http://www.tile38.com