4 ms·

> 9. Has a robots.txt > You may have a lot of things you do not want the world to know about. (Eg user uploaded media). Use robots.txt to hide them from search

> 9. Has a robots.txt > You may have a lot of things you do not want the world to know about. (Eg user uploaded media). Use robots.txt to hide them from search engines. That's not what a robots.txt file is for. It doesn't hide files, if anything it makes files that you don't want the world to know about easier to find.

shabda 15y ago
True, what I want to say is. If you have a ton of user uploaded media (and such), you probably want to not want SE to index everything. (For example to save bandwidth). I should probably re-word that line, but I dont want to get too technical.
- ceejayoz 15y ago
  > If you have a ton of user uploaded media (and such), you probably want to not want SE to index everything. (For example to save bandwidth). Bandwidth is cheap. Ranking poorly in search engines is not. If you've got a ton of user uploaded media, you've potentially got a ton of content for Google to pick up on and rank you for.
  - cma 15y ago
    If you aren't able to control user spam effectively, Google will think you are the spammer. (maybe another item should be: use nofollow on user links)
  - jimmyjim 15y ago
    For something like an image hosting or pastebin site, isn't that possible a privacy concern? I'm curious, what is the standard practice for the storage of static media - where are the static files (text/images) stored on the server, and how are the appropriate config files that concern that media set up?
    - mtogo 15y ago
      If you don't want the world to see your files, actually stop them from being accessed instead of politely asking robots to not index them.
- mkr-hn 15y ago
  If the images are relevant then you're tossing earned SE traffic and extra value in the other ranking signals Google uses. If you have 1000 posts on gardening and an image for each with gardening in the filename, you're going to have an easier time ranking for gardening.

shabda 15y ago

True, what I want to say is. If you have a ton of user uploaded media (and such), you probably want to not want SE to index everything. (For example to save bandwidth). I should probably re-word that line, but I dont want to get too technical.

ceejayoz 15y ago
> If you have a ton of user uploaded media (and such), you probably want to not want SE to index everything. (For example to save bandwidth). Bandwidth is cheap. Ranking poorly in search engines is not. If you've got a ton of user uploaded media, you've potentially got a ton of content for Google to pick up on and rank you for.
- cma 15y ago
  If you aren't able to control user spam effectively, Google will think you are the spammer. (maybe another item should be: use nofollow on user links)
- jimmyjim 15y ago
  For something like an image hosting or pastebin site, isn't that possible a privacy concern? I'm curious, what is the standard practice for the storage of static media - where are the static files (text/images) stored on the server, and how are the appropriate config files that concern that media set up?
  - mtogo 15y ago
    If you don't want the world to see your files, actually stop them from being accessed instead of politely asking robots to not index them.
mkr-hn 15y ago
If the images are relevant then you're tossing earned SE traffic and extra value in the other ranking signals Google uses. If you have 1000 posts on gardening and an image for each with gardening in the filename, you're going to have an easier time ranking for gardening.