Not obeying robots.txt is at most a civil matter, such as Craigslist vs. 3Taps (largely unsettled btw, because internet is public by design). My question still

Not obeying robots.txt is at most a civil matter, such as Craigslist vs. 3Taps (largely unsettled btw, because internet is public by design). My question still stands.