5 ms·

See also: https://www.bloomberg.com/robots.txt https://www.bloomberg.com/robots.txt

dazc 5y ago
> If you can read this then you should apply here. If you also know why we're making some basic mistakes here, especially apply here.
- ewidar 5y ago
  I don't know much about the content of robots.txt, can you share some mistakes they are making here?
  - dazc 5y ago
    It's not really a good idea to restrict crawlers at directory level since it doesn't prevent any of those pages being indexed. You can get weird behaviour such as googlebot trying to crawl a page but not being allowed to. If this happens a lot then they are going to penalise you for it because they don't want a bunch of pages in their index that they don't know the content of. If this number of pages is a significant percentage of your site then you are in real trouble. Much better to use x-robots noindex, nofollow for pages you don't want to be in the public domain. https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...
    - zinekeller 5y ago
      ... if you're specifically targeting Google. If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.
- deleted 5y ago
  [deleted]
agilob 5y ago
>Disallow: /polska Why is only poland blocked for all robots? Why is there no /deutchland?
- zinekeller 5y ago
  What's this: https://www.bloomberg.com/polska https://www.bloomberg.com/polska Seems that Bloomberg hosts something for the Polish, but I'm not sure what.
  - agilob 5y ago
    Financial reports about companies, not for their average readers
- undebuggable 5y ago
  Looks like aggregation of press releases coming from public companies listed on Warsaw's stock exchange.
cookiengineer 5y ago
That I, Robot (Asimov) reference though. [1] [1] https://en.wikipedia.org/wiki/I,_Robot https://en.wikipedia.org/wiki/I,_Robot

dazc 5y ago

> If you can read this then you should apply here. If you also know why we're making some basic mistakes here, especially apply here.

ewidar 5y ago
I don't know much about the content of robots.txt, can you share some mistakes they are making here?
- dazc 5y ago
  It's not really a good idea to restrict crawlers at directory level since it doesn't prevent any of those pages being indexed. You can get weird behaviour such as googlebot trying to crawl a page but not being allowed to. If this happens a lot then they are going to penalise you for it because they don't want a bunch of pages in their index that they don't know the content of. If this number of pages is a significant percentage of your site then you are in real trouble. Much better to use x-robots noindex, nofollow for pages you don't want to be in the public domain. https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...
  - zinekeller 5y ago
    ... if you're specifically targeting Google. If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.
deleted 5y ago
[deleted]

agilob 5y ago

>Disallow: /polska Why is only poland blocked for all robots? Why is there no /deutchland?

zinekeller 5y ago
What's this: https://www.bloomberg.com/polska https://www.bloomberg.com/polska Seems that Bloomberg hosts something for the Polish, but I'm not sure what.
- agilob 5y ago
  Financial reports about companies, not for their average readers
undebuggable 5y ago
Looks like aggregation of press releases coming from public companies listed on Warsaw's stock exchange.

cookiengineer 5y ago

That I, Robot (Asimov) reference though. [1] [1] https://en.wikipedia.org/wiki/I,_Robot https://en.wikipedia.org/wiki/I,_Robot