6 ms·

Offtopic, but while I may have nerdy GISers gathering here I'll shoot my shot. Someone knows what is a common/standard way of displaying vector information (po

by BbzzbB 5y ago

Offtopic, but while I may have nerdy GISers gathering here I'll shoot my shot. Someone knows what is a common/standard way of displaying vector information (points in this case) without having that information scrape-able? Is that even possible? I have a project where I want to sell some georeferenced data and be able to show said data (without all attributes, but showing all points) to potential customers in a map, but IIRC I've never failed to scrape vector web map data so I'm not sure it's even possible. I could imprint them in the tile rasters but then they wouldn't be interactable. Thanks (if offtopic ain't permitted on HN just delete this, sorry) Edit: So many insightful answers, thank you so much for the pointers HN, love y'all... and sorry OP for piggybacking.

moritonal 5y ago
Google Maps has the same issue. Their trick is to pass a ton of the data as protobuf, then decrypt it in WASM and load it into WebGL to render and interact with. Whilst all a massive pain, you can still scrape it with raw obj dumps from the GPU. So it's always a & game.
- BbzzbB 5y ago
  Interesting solution, thanks for sharing.
3np 5y ago
I’d just choose some small subset of your dataset as a sample dataset, expose just that, and accept that some users will scrape and make use of that. If this really isn’t satisfactory, the best you can do is non-interactive video, I think. For your actual paying customers, you control it via contract rather than DRM. (Worked with a data business, not specifically GIS)
- BbzzbB 5y ago
  Thanks for your input! I'll be trying to implement hardly-scrapeable data as a learning opportunity (as I'm working towards coding + GIS career), but settling for a subset of data + demonstrative video would be a satisfactory compromise if the former turns out to be unachievable or overkill.
arthurcolle 5y ago
Haha one of the vesseltracking sites used .swf for the longest time to be able to not allow you to grab the points corresponding to the ships. You could probably prerender images but that's a bad solution. Good question, I'm sure someone else will have a better idea of the latest state-of-the-art for this.
- BbzzbB 5y ago
  Thanks for the input. >I'm sure someone else will have a better idea of the latest state-of-the-art for this. More than I could've hoped for!
hampelm 5y ago
Nope, not that I've seen -- if the browser gets the data, anyone can get the data. You can make it more difficult, and the defenses I've seen vary based on the type of data. Here are a few: - Limit the geography of the sample - Use raster tiles at far zooms and switch to vector at close zooms for interactivity. Combine this with a limit on the number of tiles an unauthenticated user can consume to make mass downloading more difficult. - Have data that changes frequently enough that a one-time scrape decreases in value pretty quickly - Only share the real value at scale with authenticated customers. The real value might be the geometries or the attributes or the combo pack. - Trust that most serious customers will prefer to pay to work with you rather than abscond with the data. Those that are willing to put work into scraping it probably won't pay anyways Really, the last point is the key. You want to have a data product where most consumers want to create a legitimate business relationship with you. My opinion is if your potential customers don't fit in this bucket, there likely are deeper concerns, and if most do, you'll be fine! Aka sweat getting the first customer rather than blocking the first scraper.
- hampelm 5y ago
  If you really want to go hog wild, you could use a system where tokens with a short expiry are used to authenticate requests even when users aren't logged in. You'd combine that with rate limits + IP-level bans for when active or expired tokens are overused. I would say that's total overkill for 99% of services though.
- BbzzbB 5y ago
  Thank you for the suggestions, especially that last one - I may be "paranoid" considering the target audience are professionals, I really should focus on the product rather than the fence (besides some basic defense so the points aren't in plain JSON).
Doctor_Fegg 5y ago
You could serve them in vector tiles, which are served protobuf-encoded. It's still fairly easily scrapeable (get the URL via the browser's Network tab, run through vt2geojson) but would probably deter the casual scraper.
- BbzzbB 5y ago
  Thanks I'll look into that. Deterring the casual scraper would be the goal basically, make them work enough that it's not worth the hassle in respect to the price for legitimate access as a motivated and technical person with a lot of time would always get to extract data which is shown client-side.
gregsadetsky 5y ago
You could bake the points into the tiles and then have a web service that answers requests on mouse clicks i.e. when a user clicks one of the points, you send the coordinates to the server and the server determines if there's a point (or multiple) nearby and sends back the related attributes. You'd also add some rate limiting on the server-side so that someone couldn't easily request from your server all attributes for point 0,0 then 0,1 then 0,2 etc. If someone was very determined to break this, they could make a screenshot and manually triangulate each point (depending how many there are -- if you're hiding 10 points, don't bother. If it's 1k or more, that'd be harder to do manually) or even use computer vision/pixel color thresholding to extract points (say, red pixels). Same thing for the attributes, they could always use different IP addresses to break any IP-based rate limit. In response to that, you could force users to authentify (and use recaptcha during signup) to minimize the IP-rotation problem.
- BbzzbB 5y ago
  Love this idea, great way to have a raster be interactive without keeping point coordinates client-side. I do assume that someone very motivated and knowledgeable will always find a way to scrape it, I just have to make it hard enough that it's not worth the hassle over just buying the (reasonably priced) product. Thank you
tyingq 5y ago
You could limit the window by encrypting the data stream with some time-limited "long enough for a demo" key. The can open a debugger of course.
- BbzzbB 5y ago
  Thanks for that suggestion!
londons_explore 5y ago
You could quantize the coordinates, making them still look great for viewing on a screen, but insufficiently accurate for whatever the scrapers want to use them for.
- BbzzbB 5y ago
  True, or in the same vein limiting free access to clustered data. Thanks!
Chyzwar 5y ago
You could try to "encrypt" data and use Mapbox expressions/frontend transform to decrypt. Point coordinates will be randomly shifted, you will send in a separate request as Wasm/J module to reposition features on the map. Wasm module could call to Mapbox expression to reposition points. This would make it very hard to scrape. If someone scrape vector data without reverse engendering decryption module, then will get incorrect data. You just make sure that wasm module obfuscated. Not sure if mapbox expression can perform change of coordinates. But there might be different ways to transform vector data on frontend. https://docs.mapbox.com/mapbox-gl-js/style-spec/expressions/ https://docs.mapbox.com/mapbox-gl-js/style-spec/expressions/
- BbzzbB 5y ago
  Thank you! I'll dig around Mapbox encrypting sounds promising.
  - deleted 5y ago
    [deleted]
  - Chyzwar 5y ago
    Only risk that if someone reverse engineer yours wasm/JS module, they will still get the data from that point of time. The above approach is good if data changes over time. I think this should stop most people.
mistrial9 5y ago
> displaying vector information render to PNG and serve that.. you can even get the clicks and respond as you see fit with details, with some work
tmostak 5y ago
You could try OmniSci, it’s a database, rendering engine, and interactive analytics frontend (or any combination of the above) and can easily query and render millions to tens of billions of points interactively while allowing for things like tooltips on the data. See omnisci.com/demos for some live examples.
devwastaken 5y ago
Realtime server side rendering that updates with something like jpeg based live streaming? For example vnc can be pretty snappy because it only sends what's changed, and doesn't use video codecs, mostly libturbojpeg. If the mapping is mostly plane colors then I'd imagine it would be responsive enough. Maybe could get even more creative if all the map colors fit into 8 bit png's, purely hypothetical.
- BbzzbB 5y ago
  Sounds a bit heavy without a tool designed for maps or rasters, but this is similar in nature to other tips I got here using GIS-tools (namely the GeoServer one). Thank you!
kingofpandora 5y ago
Use GeoServer to render the data and serve it as a WMS layer. This can be combined with whatever base layer you want underneath. Then to make it interactive, you have WFS service to handle the clicks. At the end of the day, the data is still scrapable, but it's not nearly as easy as just grabbing the x and y using browser tools.
- BbzzbB 5y ago
  Awesome I'll check it out, thanks for the specificity!

by BbzzbB 5y ago

moritonal 5y ago

Google Maps has the same issue. Their trick is to pass a ton of the data as protobuf, then decrypt it in WASM and load it into WebGL to render and interact with. Whilst all a massive pain, you can still scrape it with raw obj dumps from the GPU. So it's always a & game.

BbzzbB 5y ago
Interesting solution, thanks for sharing.

3np 5y ago

I’d just choose some small subset of your dataset as a sample dataset, expose just that, and accept that some users will scrape and make use of that. If this really isn’t satisfactory, the best you can do is non-interactive video, I think. For your actual paying customers, you control it via contract rather than DRM. (Worked with a data business, not specifically GIS)

BbzzbB 5y ago
Thanks for your input! I'll be trying to implement hardly-scrapeable data as a learning opportunity (as I'm working towards coding + GIS career), but settling for a subset of data + demonstrative video would be a satisfactory compromise if the former turns out to be unachievable or overkill.

arthurcolle 5y ago

Haha one of the vesseltracking sites used .swf for the longest time to be able to not allow you to grab the points corresponding to the ships. You could probably prerender images but that's a bad solution. Good question, I'm sure someone else will have a better idea of the latest state-of-the-art for this.

BbzzbB 5y ago
Thanks for the input. >I'm sure someone else will have a better idea of the latest state-of-the-art for this. More than I could've hoped for!

hampelm 5y ago

Nope, not that I've seen -- if the browser gets the data, anyone can get the data. You can make it more difficult, and the defenses I've seen vary based on the type of data. Here are a few: - Limit the geography of the sample - Use raster tiles at far zooms and switch to vector at close zooms for interactivity. Combine this with a limit on the number of tiles an unauthenticated user can consume to make mass downloading more difficult. - Have data that changes frequently enough that a one-time scrape decreases in value pretty quickly - Only share the real value at scale with authenticated customers. The real value might be the geometries or the attributes or the combo pack. - Trust that most serious customers will prefer to pay to work with you rather than abscond with the data. Those that are willing to put work into scraping it probably won't pay anyways Really, the last point is the key. You want to have a data product where most consumers want to create a legitimate business relationship with you. My opinion is if your potential customers don't fit in this bucket, there likely are deeper concerns, and if most do, you'll be fine! Aka sweat getting the first customer rather than blocking the first scraper.

hampelm 5y ago
If you really want to go hog wild, you could use a system where tokens with a short expiry are used to authenticate requests even when users aren't logged in. You'd combine that with rate limits + IP-level bans for when active or expired tokens are overused. I would say that's total overkill for 99% of services though.
BbzzbB 5y ago
Thank you for the suggestions, especially that last one - I may be "paranoid" considering the target audience are professionals, I really should focus on the product rather than the fence (besides some basic defense so the points aren't in plain JSON).

Doctor_Fegg 5y ago

You could serve them in vector tiles, which are served protobuf-encoded. It's still fairly easily scrapeable (get the URL via the browser's Network tab, run through vt2geojson) but would probably deter the casual scraper.

BbzzbB 5y ago
Thanks I'll look into that. Deterring the casual scraper would be the goal basically, make them work enough that it's not worth the hassle in respect to the price for legitimate access as a motivated and technical person with a lot of time would always get to extract data which is shown client-side.

gregsadetsky 5y ago

You could bake the points into the tiles and then have a web service that answers requests on mouse clicks i.e. when a user clicks one of the points, you send the coordinates to the server and the server determines if there's a point (or multiple) nearby and sends back the related attributes. You'd also add some rate limiting on the server-side so that someone couldn't easily request from your server all attributes for point 0,0 then 0,1 then 0,2 etc. If someone was very determined to break this, they could make a screenshot and manually triangulate each point (depending how many there are -- if you're hiding 10 points, don't bother. If it's 1k or more, that'd be harder to do manually) or even use computer vision/pixel color thresholding to extract points (say, red pixels). Same thing for the attributes, they could always use different IP addresses to break any IP-based rate limit. In response to that, you could force users to authentify (and use recaptcha during signup) to minimize the IP-rotation problem.

BbzzbB 5y ago
Love this idea, great way to have a raster be interactive without keeping point coordinates client-side. I do assume that someone very motivated and knowledgeable will always find a way to scrape it, I just have to make it hard enough that it's not worth the hassle over just buying the (reasonably priced) product. Thank you

tyingq 5y ago

You could limit the window by encrypting the data stream with some time-limited "long enough for a demo" key. The can open a debugger of course.

BbzzbB 5y ago
Thanks for that suggestion!

londons_explore 5y ago

You could quantize the coordinates, making them still look great for viewing on a screen, but insufficiently accurate for whatever the scrapers want to use them for.

BbzzbB 5y ago
True, or in the same vein limiting free access to clustered data. Thanks!

Chyzwar 5y ago

You could try to "encrypt" data and use Mapbox expressions/frontend transform to decrypt. Point coordinates will be randomly shifted, you will send in a separate request as Wasm/J module to reposition features on the map. Wasm module could call to Mapbox expression to reposition points. This would make it very hard to scrape. If someone scrape vector data without reverse engendering decryption module, then will get incorrect data. You just make sure that wasm module obfuscated. Not sure if mapbox expression can perform change of coordinates. But there might be different ways to transform vector data on frontend. https://docs.mapbox.com/mapbox-gl-js/style-spec/expressions/ https://docs.mapbox.com/mapbox-gl-js/style-spec/expressions/

BbzzbB 5y ago
Thank you! I'll dig around Mapbox encrypting sounds promising.
- deleted 5y ago
  [deleted]
- Chyzwar 5y ago
  Only risk that if someone reverse engineer yours wasm/JS module, they will still get the data from that point of time. The above approach is good if data changes over time. I think this should stop most people.

mistrial9 5y ago

> displaying vector information render to PNG and serve that.. you can even get the clicks and respond as you see fit with details, with some work

tmostak 5y ago

You could try OmniSci, it’s a database, rendering engine, and interactive analytics frontend (or any combination of the above) and can easily query and render millions to tens of billions of points interactively while allowing for things like tooltips on the data. See omnisci.com/demos for some live examples.

devwastaken 5y ago

Realtime server side rendering that updates with something like jpeg based live streaming? For example vnc can be pretty snappy because it only sends what's changed, and doesn't use video codecs, mostly libturbojpeg. If the mapping is mostly plane colors then I'd imagine it would be responsive enough. Maybe could get even more creative if all the map colors fit into 8 bit png's, purely hypothetical.

BbzzbB 5y ago
Sounds a bit heavy without a tool designed for maps or rasters, but this is similar in nature to other tips I got here using GIS-tools (namely the GeoServer one). Thank you!

kingofpandora 5y ago

Use GeoServer to render the data and serve it as a WMS layer. This can be combined with whatever base layer you want underneath. Then to make it interactive, you have WFS service to handle the clicks. At the end of the day, the data is still scrapable, but it's not nearly as easy as just grabbing the x and y using browser tools.

BbzzbB 5y ago
Awesome I'll check it out, thanks for the specificity!