4 ms·

I'm Director of Engineering at Creative Commons and part of the team that is working on CC Search. We've been working on the product for over a year and we are

by kgodey 7y ago

I'm Director of Engineering at Creative Commons and part of the team that is working on CC Search. We've been working on the product for over a year and we are just out of beta today! One of CC's goals is to encourage the use and remixing of CC-licensed content, and we hope that CC Search will help make that content more discoverable. The current version is very much an MVP and only searches images, but we plan to add more content types in the future and index the ~1.4 billion works out there under a CC license. We would love any feedback you might have. Also, CC Search, the associated API, and the scripts we use to index data are all open-source and developed completely openly. Our sprints and roadmap are public and we welcome contributions from the community. Relevant links: CC Search code: https://github.com/creativecommons/cccatalog-frontend/ https://github.com/creativecommons/cccatalog-frontend/ CC Catalog API code: https://github.com/creativecommons/cccatalog-api/ https://github.com/creativecommons/cccatalog-api/ CC Catalog code: https://github.com/creativecommons/cccatalog/ https://github.com/creativecommons/cccatalog/ 2019 Vision: https://creativecommons.org/2019/03/19/cc-search/ https://creativecommons.org/2019/03/19/cc-search/ Roadmap: https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egzrRayyDdxlzxZOTCm_pc/ https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz... Active sprint: https://github.com/orgs/creativecommons/projects/7 https://github.com/orgs/creativecommons/projects/7 Backlog: https://github.com/orgs/creativecommons/projects/10 https://github.com/orgs/creativecommons/projects/10 How to contribute to CC projects: https://creativecommons.github.io/contributing-code/ https://creativecommons.github.io/contributing-code/

detaro 7y ago
If I put images in the internet, how can I tell your search about them/about the license?
- jdc 7y ago
  https://wiki.creativecommons.org/wiki/Marking_Works_Technical https://wiki.creativecommons.org/wiki/Marking_Works_Technica...
  - detaro 7y ago
    That should probably be linked somewhere prominently then (I was aware it exists, but unsure if it is still current, given that there have been little changes to that page in the last decade). And I guess I have to hope I end up in Common Crawl?
    - aldenpage 7y ago
      Hello, I work on this project! For the most part, we use Common Crawl to discover which websites carry the most CC content, and then integrate the platform through either their API, if available, or put together a bespoke scraper. If you put your content on one of these integrated platforms, eventually your work will appear in our collection. In my mind, the dream is to have the user embed an asset from our servers on their web page (like an updated version of these old CC license buttons [0]), read the referrer headers from the server logs, and then dispatch crawlers that read ccREL [1] data embedded on the page, which would allow us to instantly index content as soon as it is published. Performing broad web crawls searching the web for ccREL data is also possible but probably not what we're looking to do in the near-term. We have a ways to go before we are able to do this, since there's no easy way for end users to create and embed ccREL at the moment, and there are of course lots of other unanswered questions about how we would moderate incorrect attribution, how these tools might be abused, etc. [0] https://licensebuttons.net/l/by-nd/4.0/88x31.png https://licensebuttons.net/l/by-nd/4.0/88x31.png [1] https://www.w3.org/Submission/ccREL/ https://www.w3.org/Submission/ccREL/
- kgodey 7y ago
  We plan to implement an API for users to submit CC-licensed works that they'd like to add to our database later this year. Once we have that, you can call our API when you publish a new CC-licensed work and we'll automatically add it to our catalog. We have to work out some details like how we can verify that the content is actually CC-licensed. Encouraging use of ccREL embedding like aldenpage mentioned in another response is also in our long-term vision since that would require no additional actions from the creators.
  - antpls 7y ago
    You could have the same (only existing?) process than other content platforms like YouTube, Github, etc. In practice, most people are fair and post the content with the right license, but anyone can send a DMCA request. An annual transparency report is probably mandatory with that process, like these ones : https://github.blog/2019-01-23-2018-transparency-report/ https://github.blog/2019-01-23-2018-transparency-report/ https://transparency.facebook.com https://transparency.facebook.com
mendeza 7y ago
This is nice, I can see this being used for computer vision researchers and practitioners to train CNNs. Have you thought of implementing a reverse image search engine? That would be a great feature or project to work on as it can help users find images they are interested in as tags tend to be noisy.
- kgodey 7y ago
  Thanks! I’ve added this to the pipeline of ideas on our roadmap [0]. [0] https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egzrRayyDdxlzxZOTCm_pc/edit#heading=h.grk4z0h0w6ck https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...
Ao7bei3s 7y ago
Nice project! Thanks for working on it! It would be great if there were more search filters. Especially size ("larger than") and date ("within x-y", "last 1 day/week/month/year"). And when you have filters anyway, filtering by EXIF data (geoposition, and typical photography details like camera model / aperture / focal length) would be pretty cool.
- kgodey 7y ago
  Thanks for the feedback! I've added those filters to our pipeline of ideas for our roadmap [0]. [0] https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egzrRayyDdxlzxZOTCm_pc/edit#heading=h.grk4z0h0w6ck https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...
spectaclepiece 7y ago
Excellent project, this simplifies the process I currently have of searching many different sources and applying a cc filter (mostly Flickr and Google images). The two image filters I use most often there are image size (larger than) and orientation (portrait or landscape). If these would be included here it would be perfect.
- kgodey 7y ago
  Thank you for the feedback!
pbhjpbhj 7y ago
Do you mind me asking how many man-years the project took? Is there an article on/diary covering the dev decisions?
- kgodey 7y ago
  We've been working on it for a year and we have one data engineer, one backend engineer, and one frontend engineer. We recently set up a tech blog[0] and we have upcoming posts that will talk about the architecture and decisions we made. Here's a post about the original proof-of-concept (we are no longer using that version): https://hackernoon.com/cc-search-developer-notes-and-reflection-418271194682 https://hackernoon.com/cc-search-developer-notes-and-reflect... [0] https://creativecommons.github.io/blog/ https://creativecommons.github.io/blog/

by kgodey 7y ago

detaro 7y ago

If I put images in the internet, how can I tell your search about them/about the license?

jdc 7y ago
https://wiki.creativecommons.org/wiki/Marking_Works_Technical https://wiki.creativecommons.org/wiki/Marking_Works_Technica...
- detaro 7y ago
  That should probably be linked somewhere prominently then (I was aware it exists, but unsure if it is still current, given that there have been little changes to that page in the last decade). And I guess I have to hope I end up in Common Crawl?
  - aldenpage 7y ago
    Hello, I work on this project! For the most part, we use Common Crawl to discover which websites carry the most CC content, and then integrate the platform through either their API, if available, or put together a bespoke scraper. If you put your content on one of these integrated platforms, eventually your work will appear in our collection. In my mind, the dream is to have the user embed an asset from our servers on their web page (like an updated version of these old CC license buttons [0]), read the referrer headers from the server logs, and then dispatch crawlers that read ccREL [1] data embedded on the page, which would allow us to instantly index content as soon as it is published. Performing broad web crawls searching the web for ccREL data is also possible but probably not what we're looking to do in the near-term. We have a ways to go before we are able to do this, since there's no easy way for end users to create and embed ccREL at the moment, and there are of course lots of other unanswered questions about how we would moderate incorrect attribution, how these tools might be abused, etc. [0] https://licensebuttons.net/l/by-nd/4.0/88x31.png https://licensebuttons.net/l/by-nd/4.0/88x31.png [1] https://www.w3.org/Submission/ccREL/ https://www.w3.org/Submission/ccREL/
kgodey 7y ago
We plan to implement an API for users to submit CC-licensed works that they'd like to add to our database later this year. Once we have that, you can call our API when you publish a new CC-licensed work and we'll automatically add it to our catalog. We have to work out some details like how we can verify that the content is actually CC-licensed. Encouraging use of ccREL embedding like aldenpage mentioned in another response is also in our long-term vision since that would require no additional actions from the creators.
- antpls 7y ago
  You could have the same (only existing?) process than other content platforms like YouTube, Github, etc. In practice, most people are fair and post the content with the right license, but anyone can send a DMCA request. An annual transparency report is probably mandatory with that process, like these ones : https://github.blog/2019-01-23-2018-transparency-report/ https://github.blog/2019-01-23-2018-transparency-report/ https://transparency.facebook.com https://transparency.facebook.com

mendeza 7y ago

This is nice, I can see this being used for computer vision researchers and practitioners to train CNNs. Have you thought of implementing a reverse image search engine? That would be a great feature or project to work on as it can help users find images they are interested in as tags tend to be noisy.

kgodey 7y ago
Thanks! I’ve added this to the pipeline of ideas on our roadmap [0]. [0] https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egzrRayyDdxlzxZOTCm_pc/edit#heading=h.grk4z0h0w6ck https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...

Ao7bei3s 7y ago

Nice project! Thanks for working on it! It would be great if there were more search filters. Especially size ("larger than") and date ("within x-y", "last 1 day/week/month/year"). And when you have filters anyway, filtering by EXIF data (geoposition, and typical photography details like camera model / aperture / focal length) would be pretty cool.

kgodey 7y ago
Thanks for the feedback! I've added those filters to our pipeline of ideas for our roadmap [0]. [0] https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egzrRayyDdxlzxZOTCm_pc/edit#heading=h.grk4z0h0w6ck https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...

spectaclepiece 7y ago

Excellent project, this simplifies the process I currently have of searching many different sources and applying a cc filter (mostly Flickr and Google images). The two image filters I use most often there are image size (larger than) and orientation (portrait or landscape). If these would be included here it would be perfect.

kgodey 7y ago
Thank you for the feedback!

pbhjpbhj 7y ago

Do you mind me asking how many man-years the project took? Is there an article on/diary covering the dev decisions?

kgodey 7y ago
We've been working on it for a year and we have one data engineer, one backend engineer, and one frontend engineer. We recently set up a tech blog[0] and we have upcoming posts that will talk about the architecture and decisions we made. Here's a post about the original proof-of-concept (we are no longer using that version): https://hackernoon.com/cc-search-developer-notes-and-reflection-418271194682 https://hackernoon.com/cc-search-developer-notes-and-reflect... [0] https://creativecommons.github.io/blog/ https://creativecommons.github.io/blog/