8 ms·

Hi from the OpenStreetMap Foundation. Please don't add AI-detected features directly to the database. The algorithms have problems with false positives, and wi

by stereo 2y ago

Hi from the OpenStreetMap Foundation. Please don't add AI-detected features directly to the database. The algorithms have problems with false positives, and with mapping straight or rectangular objects as wobbly, as shown in the second-to-last screenshot. As a helper to detect missing features, this is a precious tool. But we still need human intervention to make sure the detected objects are drawn correctly. See also: https://wiki.openstreetmap.org/wiki/Import/Guidelines https://wiki.openstreetmap.org/wiki/Import/Guidelines and https://wiki.openstreetmap.org/wiki/Automated_Edits_code_of_conduct https://wiki.openstreetmap.org/wiki/Automated_Edits_code_of_...

daavoo 2y ago
> But we still need human intervention to make sure the detected objects are drawn correctly. Hi, I am the author. The demo app and any provided code example includes a step asking a human to verify the detected features. You can't upload them automatically unless you modify the source code. I reiterate the human verification across the docs, linked post, and any code samples. I haven't ever uploaded features automatically. In fact I manually edited and labeled hundreds of swimming pool samples myself before even training the first version. Happy to hear and implement any ideas on how to improve the process to prevent automated features to be uploaded. I know some people might say: just don't publish the tool, I think we can do better at embracing AI and having an open discussion.
- halosghost 2y ago
  I am afraid your “human in the loop” isn't [0]. You acknowledge the problem of AI slop up-front, but seem to have chosen to plow forward anyway. Please do consider your actions and their effects carefully [1]. You are in a position of potential influence; try not to squander it. All the best, -HG [0]: https://pluralistic.net/2024/10/30/a-neck-in-a-noose/ https://pluralistic.net/2024/10/30/a-neck-in-a-noose/ [1]: https://web.cs.ucdavis.edu/~rogaway/papers/radical.pdf https://web.cs.ucdavis.edu/~rogaway/papers/radical.pdf
  - Vinnl 2y ago
    "plow forward" and implying they didn't consider their actions and their effects seem ungenerous, given the list of precautions in the comment you're replying to. You can disagree on whether the measures are effective, of course, but they're clearly not thoughtless.
    - benatkin 2y ago
      It was the equivalent of vibe coding: > The polygons the algorithm had drawn were consistently of poor quality with stray nodes and nodes far outside the pool boundaries, and the imports hadn't been discussed with local communities. https://news.ycombinator.com/item?id=43448498 https://news.ycombinator.com/item?id=43448498
      - Vinnl 2y ago
        That too seems an ungenerous characterisation, and the GP could not have deduced that from the OP. I'm glad the author is constructively incorporating criticism and working to turn this into a useful tool that OSM users will benefit from, because they wouldn't have been the first to get overly defensive after their work was interpreted in the worst light possible.
  - necovek 2y ago
    I believe your critique would be more valuable if you've actually uncovered cases where the human in the loop missed these issues and bad or "wobbly" data ended up in the OSM database: did this happen? (the other comment from another person confirms it has) Otherwise, you are discounting their effort based on a prejudice — others might be unable to supervise an AI, but someone who's actually developed it might have a better chance of success.
    - yorwba 2y ago
      The screenshot in the article shows wobbly data. You may need to zoom in and look closely to notice.
  - 2y ago
- yorwba 2y ago
  > Happy to hear and implement any ideas on how to improve the process to prevent automated features to be uploaded. Idea: do not automatically create features that a human can simply approve, instead require them to draw the polygon themselves.
  - IshKebab 2y ago
    That kind of defeats the point surely?
    - echoangle 2y ago
      Well it can still detect features in the satellite image that are missing on the map. That would already be a large help, no?
      - banana_dick_3 2y ago
        [dead]
    - timewizard 2y ago
      The point of what? Slamming a bunch of contributions in? Or actually improving the product for end users?
      - IshKebab 2y ago
        Adding missing features. I'm pretty sure if users are searching for swimming pools they'd rather have a swimming pool with slightly bent sides than none at all.
  - daavoo 2y ago
    That is what I will implement before bringing back the demo https://news.ycombinator.com/item?id=43448649 https://news.ycombinator.com/item?id=43448649
  - K2h 2y ago
    long time editor of osm here. what you describe is what the rapid [1] editor from meta does where user is forced to manually select objects overlayed sat imagery. is limited to 50 objects before user must push. a great method i think [1] https://rapideditor.org/ https://rapideditor.org/
- banana_dick_2 2y ago
  [dead]
- boredpudding 2y ago
  I don't understand the 'human verification' aspect. Your docs show a simple image where the user can choose to keep a new object or not. [0] Afterwards it says: "The ones you chose to keep will be uploaded to OpenStreetMap using upload_osm.". This is uploading features automatically. The fact that it asks 'are you sure' is just silly. We all know if humans have to click yes 90% of the time, and no 10% of the time, they'll miss a lot of no's. The image also proofs that: - You don't see any polygons properly. You just see a an image of where the pool is. Already on the image I can see that if the polygons align to that image, it will be a total mess. - You don't see any polygons further away from the object. Both these points are in stereo's reply that the resulted data was a mess. Please consider pulling the project. This will generate a lot of data that volunteers will have to check and revert. [0] https://github.com/mozilla-ai/osm-ai-helper/blob/main/docs/step-by-step-guide.md#step-4-review-filter-and-upload-the-new-polygons-to-openstreetmap https://github.com/mozilla-ai/osm-ai-helper/blob/main/docs/s...
  - daavoo 2y ago
    > This will generate a lot of data that volunteers will have to check and revert. This is just not true. The data can be easily identified with the `created_by` tag. And I have been reviewing myself any data uploaded with the demo (with a clear different criteria on what is good enough)
    - banana_dick_2 2y ago
      [dead]
    - deleted 2y ago
      [deleted]
    - karlgkk 2y ago
      adding a "created_by" tag is not opt in. it's opt out. you are de facto choosing how OSM volunteers must approach their work, without their consent.
    - AyyEye 2y ago
      Can I suggest also adding a tag for ML originated features that's not specific to your demo/app? Maybe this can help put extra eyes on them and/or help prevent them from polluting the DB wholesale. Maybe client apps could have a toggle to allow/reject them.
- deleted 2y ago
  [deleted]
- cratermoon 2y ago
  [flagged]
- akimbostrawman 2y ago
  I know at this point it's almost one of Mozilla's mottos to be horrible at communication but how come nobody felt the need to maybe talk about stuff like this before publishing it?
daavoo 2y ago
> straight or rectangular objects as wobbly, as shown in the second-to-last screenshot. This is a because the polygon is drawn as a mask in order to overlay it on the image. The actual polygon being uploaded doesn't have the wobbly features. It is True there are cases were the predicted polygon is wobbly and I encourage people to discard them. However I didn't publish this demo until I got a first version of the model that reached some minimum quality. There is logic in the code to simplify the shape of the predicted polygon in order to avoid having too many nodes.
- banana_dick_1 2y ago
  [flagged]
- stereo 2y ago
  Hi! The Data Working Group had a look at the data, and decided to revert the two pool changesets. The polygons the algorithm had drawn were consistently of poor quality with stray nodes and nodes far outside the pool boundaries, and the imports hadn't been discussed with local communities.
  - boredpudding 2y ago
    Can you link the (now reverted) changesets? I can't seem to find them.
    - stereo 2y ago
      https://www.openstreetmap.org/changeset/163855992 https://www.openstreetmap.org/changeset/163855992 and https://www.openstreetmap.org/changeset/163863954 https://www.openstreetmap.org/changeset/163863954 are the ones I've reverted. There are more in daavoo's changeset history.
  - daavoo 2y ago
    Hi, thanks for the feedback. I have disabled the hosted demo for now, and will remove the uploading part from the code in favor of showing an URL that will open the editor at the location. If its of any help, you can find any contributed polygon with the tag `created_by=https://github.com/mozilla-ai/osm-ai-helper https://github.com/mozilla-ai/osm-ai-helper`. Feel free to remove all of them (or I can do it myself once I access a PC). I will be happy to continue the discussion on what is a good prediction or not. I have mapped a lot of swimming pools myself and edited and removed a lot of (presumably) human contributed polygons that looked worse (too my eyes) than the predictions I approved to be uploaded.
banana_dick_1 2y ago
[flagged]
banana_dick_3 2y ago
[dead]
banana_dick_5 2y ago
[dead]
AyyEye 2y ago
Replied to daavoo, can I suggest adding a tag for ML originated features? As other comments have stated, it is likely that these tools are already being used (potentially semi-automatically) and this could help prevent them from polluting the DB wholesale.
- daavoo 2y ago
  Hi there! Feel free to jump into https://community.openstreetmap.org/t/about-mapping-features-with-computer-vision/127790/4 https://community.openstreetmap.org/t/about-mapping-features... , I think is a good point to discuss.

by stereo 2y ago

daavoo 2y ago

> But we still need human intervention to make sure the detected objects are drawn correctly. Hi, I am the author. The demo app and any provided code example includes a step asking a human to verify the detected features. You can't upload them automatically unless you modify the source code. I reiterate the human verification across the docs, linked post, and any code samples. I haven't ever uploaded features automatically. In fact I manually edited and labeled hundreds of swimming pool samples myself before even training the first version. Happy to hear and implement any ideas on how to improve the process to prevent automated features to be uploaded. I know some people might say: just don't publish the tool, I think we can do better at embracing AI and having an open discussion.

halosghost 2y ago
I am afraid your “human in the loop” isn't [0]. You acknowledge the problem of AI slop up-front, but seem to have chosen to plow forward anyway. Please do consider your actions and their effects carefully [1]. You are in a position of potential influence; try not to squander it. All the best, -HG [0]: https://pluralistic.net/2024/10/30/a-neck-in-a-noose/ https://pluralistic.net/2024/10/30/a-neck-in-a-noose/ [1]: https://web.cs.ucdavis.edu/~rogaway/papers/radical.pdf https://web.cs.ucdavis.edu/~rogaway/papers/radical.pdf
- Vinnl 2y ago
  "plow forward" and implying they didn't consider their actions and their effects seem ungenerous, given the list of precautions in the comment you're replying to. You can disagree on whether the measures are effective, of course, but they're clearly not thoughtless.
  - benatkin 2y ago
    It was the equivalent of vibe coding: > The polygons the algorithm had drawn were consistently of poor quality with stray nodes and nodes far outside the pool boundaries, and the imports hadn't been discussed with local communities. https://news.ycombinator.com/item?id=43448498 https://news.ycombinator.com/item?id=43448498
    - Vinnl 2y ago
      That too seems an ungenerous characterisation, and the GP could not have deduced that from the OP. I'm glad the author is constructively incorporating criticism and working to turn this into a useful tool that OSM users will benefit from, because they wouldn't have been the first to get overly defensive after their work was interpreted in the worst light possible.
- necovek 2y ago
  I believe your critique would be more valuable if you've actually uncovered cases where the human in the loop missed these issues and bad or "wobbly" data ended up in the OSM database: did this happen? (the other comment from another person confirms it has) Otherwise, you are discounting their effort based on a prejudice — others might be unable to supervise an AI, but someone who's actually developed it might have a better chance of success.
  - yorwba 2y ago
    The screenshot in the article shows wobbly data. You may need to zoom in and look closely to notice.
- 2y ago
yorwba 2y ago
> Happy to hear and implement any ideas on how to improve the process to prevent automated features to be uploaded. Idea: do not automatically create features that a human can simply approve, instead require them to draw the polygon themselves.
- IshKebab 2y ago
  That kind of defeats the point surely?
  - echoangle 2y ago
    Well it can still detect features in the satellite image that are missing on the map. That would already be a large help, no?
    - banana_dick_3 2y ago
      [dead]
  - timewizard 2y ago
    The point of what? Slamming a bunch of contributions in? Or actually improving the product for end users?
    - IshKebab 2y ago
      Adding missing features. I'm pretty sure if users are searching for swimming pools they'd rather have a swimming pool with slightly bent sides than none at all.
- daavoo 2y ago
  That is what I will implement before bringing back the demo https://news.ycombinator.com/item?id=43448649 https://news.ycombinator.com/item?id=43448649
- K2h 2y ago
  long time editor of osm here. what you describe is what the rapid [1] editor from meta does where user is forced to manually select objects overlayed sat imagery. is limited to 50 objects before user must push. a great method i think [1] https://rapideditor.org/ https://rapideditor.org/
banana_dick_2 2y ago
[dead]
boredpudding 2y ago
I don't understand the 'human verification' aspect. Your docs show a simple image where the user can choose to keep a new object or not. [0] Afterwards it says: "The ones you chose to keep will be uploaded to OpenStreetMap using upload_osm.". This is uploading features automatically. The fact that it asks 'are you sure' is just silly. We all know if humans have to click yes 90% of the time, and no 10% of the time, they'll miss a lot of no's. The image also proofs that: - You don't see any polygons properly. You just see a an image of where the pool is. Already on the image I can see that if the polygons align to that image, it will be a total mess. - You don't see any polygons further away from the object. Both these points are in stereo's reply that the resulted data was a mess. Please consider pulling the project. This will generate a lot of data that volunteers will have to check and revert. [0] https://github.com/mozilla-ai/osm-ai-helper/blob/main/docs/step-by-step-guide.md#step-4-review-filter-and-upload-the-new-polygons-to-openstreetmap https://github.com/mozilla-ai/osm-ai-helper/blob/main/docs/s...
- daavoo 2y ago
  > This will generate a lot of data that volunteers will have to check and revert. This is just not true. The data can be easily identified with the `created_by` tag. And I have been reviewing myself any data uploaded with the demo (with a clear different criteria on what is good enough)
  - banana_dick_2 2y ago
    [dead]
  - deleted 2y ago
    [deleted]
  - karlgkk 2y ago
    adding a "created_by" tag is not opt in. it's opt out. you are de facto choosing how OSM volunteers must approach their work, without their consent.
  - AyyEye 2y ago
    Can I suggest also adding a tag for ML originated features that's not specific to your demo/app? Maybe this can help put extra eyes on them and/or help prevent them from polluting the DB wholesale. Maybe client apps could have a toggle to allow/reject them.
deleted 2y ago
[deleted]
cratermoon 2y ago
[flagged]
akimbostrawman 2y ago
I know at this point it's almost one of Mozilla's mottos to be horrible at communication but how come nobody felt the need to maybe talk about stuff like this before publishing it?

daavoo 2y ago

> straight or rectangular objects as wobbly, as shown in the second-to-last screenshot. This is a because the polygon is drawn as a mask in order to overlay it on the image. The actual polygon being uploaded doesn't have the wobbly features. It is True there are cases were the predicted polygon is wobbly and I encourage people to discard them. However I didn't publish this demo until I got a first version of the model that reached some minimum quality. There is logic in the code to simplify the shape of the predicted polygon in order to avoid having too many nodes.

banana_dick_1 2y ago
[flagged]
stereo 2y ago
Hi! The Data Working Group had a look at the data, and decided to revert the two pool changesets. The polygons the algorithm had drawn were consistently of poor quality with stray nodes and nodes far outside the pool boundaries, and the imports hadn't been discussed with local communities.
- boredpudding 2y ago
  Can you link the (now reverted) changesets? I can't seem to find them.
  - stereo 2y ago
    https://www.openstreetmap.org/changeset/163855992 https://www.openstreetmap.org/changeset/163855992 and https://www.openstreetmap.org/changeset/163863954 https://www.openstreetmap.org/changeset/163863954 are the ones I've reverted. There are more in daavoo's changeset history.
- daavoo 2y ago
  Hi, thanks for the feedback. I have disabled the hosted demo for now, and will remove the uploading part from the code in favor of showing an URL that will open the editor at the location. If its of any help, you can find any contributed polygon with the tag `created_by=https://github.com/mozilla-ai/osm-ai-helper https://github.com/mozilla-ai/osm-ai-helper`. Feel free to remove all of them (or I can do it myself once I access a PC). I will be happy to continue the discussion on what is a good prediction or not. I have mapped a lot of swimming pools myself and edited and removed a lot of (presumably) human contributed polygons that looked worse (too my eyes) than the predictions I approved to be uploaded.

banana_dick_1 2y ago

[flagged]

banana_dick_3 2y ago

[dead]

banana_dick_5 2y ago

[dead]

AyyEye 2y ago

Replied to daavoo, can I suggest adding a tag for ML originated features? As other comments have stated, it is likely that these tools are already being used (potentially semi-automatically) and this could help prevent them from polluting the DB wholesale.

daavoo 2y ago
Hi there! Feel free to jump into https://community.openstreetmap.org/t/about-mapping-features-with-computer-vision/127790/4 https://community.openstreetmap.org/t/about-mapping-features... , I think is a good point to discuss.