11 ms·
Nice. They should probably add HTTP headers (X-Robots-Tag) to the list. Cloaking, too.
by rbinv 5y ago
Nice. They should probably add HTTP headers (X-Robots-Tag) to the list. Cloaking, too.
- walshemj 5y agoAnd ban the use of PDF's - which is another way this could be avoided. Oh and mandate clean links no funky javascipt links that search crawlers don't follow.
- bawolff 5y agoSearch engines can index pdfs.
- dvfjsdhgfv 5y agoProvided the content is not encrypted.
- dvfjsdhgfv 5y agoI stand corrected: encrypted PDFs are indexed unless a password is set. However, in the past they were unable to do so[0]: "Generally we can index textual content (written in any language) from PDF files that use various kinds of character encodings, provided they're not password protected or encrypted." [0] https://developers.google.com/search/blog/2011/09/pdfs-in-google-search-results https://developers.google.com/search/blog/2011/09/pdfs-in-go...
- bawolff 5y agoI think the assumption is that encrypted pdfs would not be used here because users still have to view these documents
- dvfjsdhgfv 5y agoNo, the PDF standard explicitly allows encryption without the need to set up user password; in this case, the owner password is set. The only change for the user is that the word "Secured" usually appears next to the filename in the reader.
- walshemj 5y agoYeh but they don't work as well and wont be as discoverable
- lmkg 5y agoPer the main project link, the pricing information required to be posted in JSON format with specific schema and file names.
- breischl 5y agoJSON is allowed, but not required. They just required open, non-proprietary, formats. They specifically gave YAML and XML as other examples.
- charrondev 5y agoIf you check the main read me proprietary formats are specifically not allowed, with notable examples of PDF, and excel