yeah, crawlers don't work like that or rather, ethical companies don't release crawlers that don't follow robot.txt instructions semalt can not fix this by off

yeah, crawlers don't work like that or rather, ethical companies don't release crawlers that don't follow robot.txt instructions semalt can not fix this by offering an 'opt-out' while ignoring robot.txt very stupid of them because they are shooting themselves in the foot by this very simple but MASSIVE breach of standards of conduct