For well-bahaved bots, robots.txt. For ill-behaved ones, it depends on why you're trying to block them. Rate throttling, IP blocking, requiring login, or just

For well-bahaved bots, robots.txt. For ill-behaved ones, it depends on why you're trying to block them. Rate throttling, IP blocking, requiring login, or just gating all access to the site with HTTP Basic Auth can all work.