10 ms·
Show HN: CC Search – search engine for 300M CC-licensed images
- Mirioron 7y agoI had no idea this even existed. That's really cool.
- dang 7y agoA related blog post is https://creativecommons.org/2019/04/30/cc-search-images/ https://creativecommons.org/2019/04/30/cc-search-images/. (Via https://news.ycombinator.com/item?id=19791801 https://news.ycombinator.com/item?id=19791801, which we merged into this thread.)
- kgodey 7y agoI'm Director of Engineering at Creative Commons and part of the team that is working on CC Search. We've been working on the product for over a year and we are just out of beta today! One of CC's goals is to encourage the use and remixing of CC-licensed content, and we hope that CC Search will help make that content more discoverable. The current version is very much an MVP and only searches images, but we plan to add more content types in the future and index the ~1.4 billion works out there under a CC license. We would love any feedback you might have. Also, CC Search, the associated API, and the scripts we use to index data are all open-source and developed completely openly. Our sprints and roadmap are public and we welcome contributions from the community. Relevant links: CC Search code: https://github.com/creativecommons/cccatalog-frontend/ https://github.com/creativecommons/cccatalog-frontend/ CC Catalog API code: https://github.com/creativecommons/cccatalog-api/ https://github.com/creativecommons/cccatalog-api/ CC Catalog code: https://github.com/creativecommons/cccatalog/ https://github.com/creativecommons/cccatalog/ 2019 Vision: https://creativecommons.org/2019/03/19/cc-search/ https://creativecommons.org/2019/03/19/cc-search/ Roadmap: https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egzrRayyDdxlzxZOTCm_pc/ https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz... Active sprint: https://github.com/orgs/creativecommons/projects/7 https://github.com/orgs/creativecommons/projects/7 Backlog: https://github.com/orgs/creativecommons/projects/10 https://github.com/orgs/creativecommons/projects/10 How to contribute to CC projects: https://creativecommons.github.io/contributing-code/ https://creativecommons.github.io/contributing-code/
- detaro 7y agoIf I put images in the internet, how can I tell your search about them/about the license?
- jdc 7y agohttps://wiki.creativecommons.org/wiki/Marking_Works_Technical https://wiki.creativecommons.org/wiki/Marking_Works_Technica...
- detaro 7y agoThat should probably be linked somewhere prominently then (I was aware it exists, but unsure if it is still current, given that there have been little changes to that page in the last decade). And I guess I have to hope I end up in Common Crawl?
- aldenpage 7y agoHello, I work on this project! For the most part, we use Common Crawl to discover which websites carry the most CC content, and then integrate the platform through either their API, if available, or put together a bespoke scraper. If you put your content on one of these integrated platforms, eventually your work will appear in our collection. In my mind, the dream is to have the user embed an asset from our servers on their web page (like an updated version of these old CC license buttons [0]), read the referrer headers from the server logs, and then dispatch crawlers that read ccREL [1] data embedded on the page, which would allow us to instantly index content as soon as it is published. Performing broad web crawls searching the web for ccREL data is also possible but probably not what we're looking to do in the near-term. We have a ways to go before we are able to do this, since there's no easy way for end users to create and embed ccREL at the moment, and there are of course lots of other unanswered questions about how we would moderate incorrect attribution, how these tools might be abused, etc. [0] https://licensebuttons.net/l/by-nd/4.0/88x31.png https://licensebuttons.net/l/by-nd/4.0/88x31.png [1] https://www.w3.org/Submission/ccREL/ https://www.w3.org/Submission/ccREL/
- 7y ago
- nategri 7y agoLooks like its currently experience the hug. Unless a search for 'cat' returning no results is expected behavior.
- O1111OOO 7y ago> Looks like its currently experience the hug. Same thing happened to me (no results) and that was my initial thought but, in my case, I had forgotten to check uMatrix. Some vital CDNs and subdomains were allowed but uMatrix was blocking api.creativecommons.engineering
- DOsinga 7y agoThis is cool and it is great that the software is all open source, but randomly clicking around it does seem that an awful lot of images all come from Flickr, with the images even being hosted by Flickr. It seems to me the interface could make the source more clear.
- kgodey 7y agoThanks for your feedback! We don't host any of the images, we link people directly to the source. Flickr does have orders of magnitude more images than other providers. We will add making the source more prominent to our list of issues to fix.
- mikece 7y agoNeat... though searching on "F-14" returned a lot of yoga images instead of the naval fighter (searching "F14" returned what I was looking for).
- tsumnia 7y agoAgreed, "ninja" came with a lot of dogs and cats (assumbly named Ninja). And "test" provides a weird assortment of not tests. I'd say filename keyword matching might be dangerous and some measure of image analysis should be made.
- pbhjpbhj 7y agoPresumably they're going to allow tag editing, maybe using a known-good-user concordance system, with meta grading (if your tags are always removed you lose trading rights).
- kgodey 7y agoThanks for the feedback! Currently the search query parsing is pretty naive and some search terms work far better than others. A big chunk of our work over the next few months is going to be focused on improving search relevance.
- propogandist 7y agohave you considered using something like the reCaptcha approach, where you have users classify/improve results?
- exabrial 7y agoDefinitely my favorite: https://search.creativecommons.org/photos/9988a06f-c7d1-466d-bd73-ad6817da075b https://search.creativecommons.org/photos/9988a06f-c7d1-466d... Thank you for this.
- s_y_n_t_a_x 7y agoI'd remove the unnecessary animations when loading the images.
- kgodey 7y agoThanks for the feedback! This is on our list to do soon, we've had other complaints.
- rixrax 7y agoThank You for doing this. I hope one day we will see a similar search for CC licensed music where you can search based on license type, and e.g. music style, beats per minute, length, audio format,... I was recently trying to find music for a side project and searching for CC licensed music appears to have gone from somewhat onerous to very in sites like Jamendo.com, free music archive, etc. This is off topic but if someone can point to a website with CC licensed music where it’s possible to search based on license type (e.g. BY-NC-SA) I would love to know.
- kgodey 7y agoThanks for your feedback! We plan to add audio to CC Search later this year (probably in Q4). We will have searching by license type built in from the beginning and those all sound like great filters to add.
- nvr219 7y agoTried searching for penis and was impressed by the variety of SFW results
- pbhjpbhj 7y agoThere doesn't appear to be a visible "sfw" flag in the search interface, but I did your search and searched "vulva" and got barely any photos of human anatomy. That seemed unusual somehow, is there an active nsfw filter in place?
- kgodey 7y agoWe don't have any filtering on our end but all the sites that we index data from have content policies. For example, we only index public images from Flickr which automatically excludes mature content.
- carapace 7y agoBlank page with JS disabled.
- kgodey 7y agoThanks for the feedback. We're going to be working on improving accessibility next month and will address this.
- scanny 7y agoOut of curiosity, are there any checks run on these images to see if they really are CC? This search result for 'satellites' is a screenshot of a google maps result, which includes the copyright information for the imagery within the CC image [see the bottom right attribution text in the imagery]. How can that really be in CC? https://search.creativecommons.org/photos/3f0eddb0-55b4-46e0-ba91-4be614e71520 https://search.creativecommons.org/photos/3f0eddb0-55b4-46e0... How could we be sure that the images themselves are genuinely licensed rather than someone copying it from another source and slapping on the licence?
- deleted 7y ago[deleted]
- kgodey 7y agoWe do our best to verify the license but cannot 100% guarantee that the image is CC-licensed. We have custom scripts that we use to ingest content from each content provider. For Flickr, we use their API which contains license information for each work. If you click through to the source image, you can see that the user who uploaded the image did license it under CC BY-SA 2.0. We don't yet have a way of dealing with content where the user incorrectly licensed it, I think we could add a "report image" function that would help us identify and remove this type of content. There's a disclaimer if you scroll down that says "Verify at the source: Flickr" with an explanation of why we can't guarantee the license, maybe we should make that more prominent.
- scanny 7y agoThanks for the answer, that is completely understandable. I guess the user just needs to do a bit of due diligence before running with it.
- adrianN 7y agoMaybe the Flickr user did buy a license from Google for that particular screenshot that allowed them to publish it as CC BY-SA. How would you verify this? What's the legal situation here? Can Google sue me when I use that image, or do they have to sue the Flickr user?
- not2b 7y agoI arbitrarily searched for "hula hoop" and most of the top images have people in them. People using this facility to find images to freely use would be advised to avoid images of recognizable people, because you don't know if they signed model releases or consent to being a part of your web site.
- kgodey 7y agoThanks for the feedback! We will figure out if there is a way to communicate this better.
- empressplay 7y agoThere are _a lot_ of CC licensed images coming up through Flickr search that are nowhere to be found in CC Search and since the vast majority of CC Search's images come from Flickr, you would be better off just searching on Flickr...
- kgodey 7y agoThanks for the feedback! We are working with Flickr to ensure that we are able to index all their CC-licensed images; currently there's an issue with their API that hides some images. It should be resolved by the end of the summer. We have images from a lot of other collections, such as the Met, Rijksmuseum, Behance, Thingiverse etc. Flickr has more images than any of them by a couple of orders of magnitude, though.
- endergen 7y agoWhere is it claimed there 300M photos? Would love a source before claiming that in my design slack channel
- kgodey 7y agoHere's the official blog post: https://creativecommons.org/2019/04/30/cc-search-images/ https://creativecommons.org/2019/04/30/cc-search-images/ I've also seen the database :).
- sbr464 7y agoIs there a search API available or would it be against the terms to access the site programmatically?
- kgodey 7y agoWe do have an API available, although we're not encouraging use of it until we build a few more features in. Docs: https://api.creativecommons.engineering/ https://api.creativecommons.engineering/ Terms of Service: https://api.creativecommons.engineering/terms_of_service.html https://api.creativecommons.engineering/terms_of_service.htm... Code: https://github.com/creativecommons/cccatalog-api/ https://github.com/creativecommons/cccatalog-api/
- sbr464 7y agoThanks for the info.
- Sommer 7y agoDefinitely appreciate the programmatic access, but in terms of straight search results it's very hard to improve on Google Image Search with the Usage Rights filter.. Info: https://support.google.com/websearch/answer/29508?p=ws_images_usagerights&hl=en https://support.google.com/websearch/answer/29508?p=ws_image...
- kgodey 7y agoThanks for the feedback! A big chunk of our upcoming work is going to be towards improving search relevance. We also plan to add content types other than images this year, starting with open textbooks and audio, so that will differentiate us.
- Sommer 7y agoThat would be fantastic. Also like to +1 the reverse image search support, finding CC images visually similar to a restricted license image would be really cool feature that Google does not seem to support.
- tomcam 7y agoRemember that for commercial use you still need a model release in the USA when faces are identifiable. It is totally separate from the licensing of the image.
- miguelmota 7y agoThis is awesome. I wonder how long til Google implements a similar filter..
- kkaranth 7y agoThey already have it. You can choose your required "usage rights" in the advanced search settings page(atleast for images)
- miguelmota 7y agoCool! Didn’t see it in mobile but did see the “Labeled for Reuse” filter option
- 200_OK 7y agoThis is great. Is this all CC0 or do some of the images have conditions? My wishlist would include also including videos and music/audio and an API to access it. I'm sure that's a big ask though.
- kgodey 7y agoAll content is not CC0, we index content shared under all CC licenses and public domain. You can filter by license or usage rights. Adding audio is planned for Q4. Video will likely be in 2020. We already have an API, although we're not encouraging use of it until we build a few more features in. Docs: https://api.creativecommons.engineering/ https://api.creativecommons.engineering/ Code: https://github.com/creativecommons/cccatalog-api/ https://github.com/creativecommons/cccatalog-api/
- z92 7y agoExcellent job. Thanks for all the hard work. A few usability issues. 1/ Doesn't work without javascript. and 2/ At screen resolution 800x600 the search field gets completely hidden. Shown bellow. https://imgur.com/a/bPkdOoT https://imgur.com/a/bPkdOoT
- kgodey 7y agoThanks for your feedback!
- lapnitnelav 7y agoI am very pleased with the quantity of cats any given search returns. You may proceed.
- reiinakano 7y agoI did a search for Flamingo and there were no flamingos...
- chmars 7y ago@kgodey: Your default HTML source code includes CC icons loaded from creativecommons.org. That is a problem: 1. It results in tracking and could violate the GDPR and other data privacy laws. 2. It is third-party content and might get blocked by ad / content blockers. I suggest to offer a default HTML source code without CC icons.
- kgodey 7y agoThanks for the feedback chmars! CC Search is hosted at search.creativecommons.org so assets loaded from creativecommons.org should not count as third-party content or tracking. Let me know if I'm missing something.
- Amorymeltzer 7y agoPretty neat! Another great resource is the 53 million or so files on Wikimedia Commons — https://commons.wikimedia.org/wiki/ https://commons.wikimedia.org/wiki/ — which don't appear to be searched by this. Commons has plenty of material that is PD, so I suppose that is technically out of scope, but it also restricts licenses to CC-BY or CC-BY-SA whereas some of these are NC.
- kgodey 7y agoThank you! Wikimedia Commons is our next big priority. We do also index PD content.
- cannedslime 7y agoIt is always nice to have search engines for free stuff. I have minor gripes with the engine. - Browser navigation doesn't work. Going back doesnt take you to your previous search query. - Search seems to be solely keyword based? And keywords are kind of hit and miss on many images.
- kgodey 7y agoThanks for the feedback, we'll add browser navigation to our list of things to fix. Our search is still pretty naive, a big chunk of our upcoming work will be focused on improving results.
- jpincheira 7y agoUnsplash has 0,95M, and CC Search 300M. Love this. No more flickr search for when I needed to find CC photos. <3 Thanks for much for your work @kgodey and to your team.
- kgodey 7y agoThank you! :)
- pbhjpbhj 7y agoIt's interesting to me how many images are lost when you filter by "right to modify". I wonder if "CC-SA" should be the default licence, ie should be what "CC" means. Similarly if the default CC license were "NC" then I imagine many more shared images would be excluded from commercial use. My suggestion is that people probably wouldn't mind modification of their CC images as a default. I usually use CC-BY-SA. Edit: looking afresh at the CC license material it appears my understanding of the licenses is weak (eg see look down-thread), that the default does allow modification. Which makes it weirder that people would go out there way to specify that their pretty poor quality images could only be used as-is add not modified.
- puranjay 7y agoI've always wondered: what does "modification" include? Is resizing modification? Cropping? If I slap some text on the raw image, would that be constituted as modification as well?
- aurelwu 7y agoNice, I have been mostly using bing as search engine for public domain // CC0 images and it worked reasonable well but a search engine specialized on this area is something which is very useful. What I immediately noticed is that when you search for something including "icon" it shows lots of images of religious icons which make sense in a way but it probably not was most people want if they search for "fireball icon" or "mail icon" or "car icon".
- kgodey 7y agoThanks for the feedback! A big chunk of our upcoming work will focus on improving search results.