8 ms·
Geospatial data science with Julia
- juliohm 3y agoGeospatial Data Science with Julia presents a fresh approach to data science with geospatial data and the Julia programming language. It contains best practices for writing clean, readable and performant code in geoscientific applications involving sophisticated representations of the (sub)surface of the Earth such as unstructured meshes made of 2D and 3D geometries.
- deleted 3y ago[deleted]
- eigenket 3y agoAre you a bot? Why did you copy and paste the top paragraph of the linked page?
- forgotpwd16 3y agoSeems to be the author and copied it as some form of abstract. @juliohm no need to be doing that.
- jstrickshire 3y agoI have a passion project 4x4anarchy.com that operates with a Python-MariaDB system for querying map data by latitude and longitude, transforming it into GeoJSON for map display. The website deals with sizable tables, approximately 1 GB in size. I've made extensive optimizations, relying on well-structured indexes, caching mechanisms, and query optimization to enhance performance. Given these circumstances, how might the incorporation of Julia and some geospatial DB (PostGIS) contribute to further optimizing geospatial data retrieval and presentation, especially when dealing with large datasets and intricate geospatial operations?
- benzofuran 3y agoCool site! Any chance of a adding a simple KMZ export for offline use for a given area of interest?
- jstrickshire 3y agoYeah, I can do that. Will get to it tomorrow!
- benzofuran 3y agoAwesome - getting KMZs of 4x4 routes is way harder than it should be. All the Colorado data is there but extracting it is challenging.
- tony_cannistra 3y agoInteresting! I work on a very similar product. I don't know Julia well, but I definitely would suggest exploring whether PostGIS can help improve the speed of your DB queries. I'd also consider how you deliver your geospatial data to your clients -- I'm not sure GeoJSON is your best bet. Protobuf tiles might be better for your use-case (e.g. the Mapbox Vector Tiles spec).
- p4ul 3y agoI completely agree! It would be hard to overstate the power of PostGIS! For anyone working with GIS data, it's absolutely worth investigating what PostGIS provides and the ease of integration to your existing application!
- gabegm 3y agoIt would depend on where most of the processing is happening. PostGIS gives you the benefit of spatial indexes which are extremely performant. I've seen Python GeoSpatial applications taking hours to finish processing which only took a few minutes when shifted onto PostGIS. If you're also doing a lot of processing in Python, exploring other languages could also help. In the case of Julia you get a typed language that's also JIT compiled.
- alekseiprokopev 3y agoNice thing about Julia is that you randomly find cool projects like this.
- nraynaud 3y agoBe mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .
- juliohm 3y agoThis is not true. Please read the book.
- nraynaud 3y agoScanning the site see mostly points algorithms, the only mention of polygons is a textbook LibGEOS call, I see no network at all. And I see no smart manipulation of anything else than points, I see no subdivision of space, etc.
- juliohm 3y agoYou probably need to re-scan the book. Meshes.jl is the submodule of the project entirely written in Julia with geometric processing algorithms.
- nraynaud 3y agoI have worked with it. It was just stating, very little useful code in it. Going back to the source code, I see they added a bit more. A quick look around suggest that only one algorithm uses an indexing structure. Clipping seems limited between a convex polygon and a concave one.
- ZeroCool2u 3y ago
- beeburrt 3y agoIn the preface you list: - Generate high-performance code - Specialize on multiple arguments - Evaluate code interactively - Exploit parallel hardware > This list of requirements eliminates Python, R and other mainstream languages used for data science. Can you elaborate on why/how? Awesome work by the way
- wodenokoto 3y agoPython and R do not generate high performing code. At best they generate calls to high performing code.
- trostaft 3y ago> At best they generate calls to high performing code. It should be noted that this is usually sufficient. But particularly for earth scale problems it can often not be.
- th0ma5 3y agoJulia is designed to seem to win arguments as best I can tell... If you complain about the need to break abstractions and the lack of general purpose application you're accused of not understanding. When you say it slow they say you can inline assembler, and when you say that's dumb why have a high level language then, they then say well you don't have to it is fast as is and everyone else is slow, and it just devolves into circular arguments. Abstractions exist in layers for reasons.
- sainez 3y agoYou can obviously provide the same abstraction with different implementations that yield different performance characteristics. Julia provides the same level of flexibility (if not more) as Python without any of the design decisions which cause Python to be so slow. I fail to see how this is a contentious point.
- th0ma5 3y ago