4 ms·

Show HN: robots.nxt – see, control, and convert bot traffic

3 points by timsylvester 2y ago

timsylvester 2y ago
Commercial websites live and die by audience analytics and conversion to revenue. Bots are the largest audience for most websites, but there aren't good tools for managing bots. robots.nxt provides a free bot traffic analytics dashboard to see how many bots you're getting, what bots they are, what they're costing you, and what content they're crawling. We provide rules for specific bots, bot categories, and specific content or content categories, and more. You can modify and combine these, such as rate limiting or access denial to selected areas, and add a pricing model any combination of rules or identities. For example, if ChatGPT tries to crawl your content pages, you can set robots.nxt so that your server will charge the bot $0.25 per page, but if an indexer crawls your pages there's no charge. And if a bad bot shows up, you can serve it a 404 or dead, link-less homepage so that the bot thinks there's just nothing there. If a bot pays for your content, you can include an AI training license. You can dynamically render robots.txt, explaining what they're allowed to do, while ensuring they can't do anything you don't want. An iron fist in a velvet glove. You can manage your server responses to, for example, filter any form submission element from any pages served to a bot. You don't have to challenge, because the form never gets served to a bot in the first place. We can see any attempts to request insecure files or site areas, so you can see if someone is trying an exploit. Humans will never be the largest source of web traffic ever again. The tools we've built with robots.nxt are an emerging standard for website and web app management in this new bot-dominated internet. We're starting an open beta today, with all our released capabilities free for the time being. Sign up and tell us what you think!
chfritz 2y ago
Funny, just yesterday I was complaining about OpenAI's gptbot's stupidity costing site operators too much: https://x.com/chfritz/status/1863689365740020012 https://x.com/chfritz/status/1863689365740020012
- timsylvester 2y ago
  Hey sorry I didn't get back to you sooner, I guess HN doesn't sent reply notices. Want to talk about it? Would be good to connect with someone else interested in this.

3 points by timsylvester 2y ago

Commercial websites live and die by audience analytics and conversion to revenue. Bots are the largest audience for most websites, but there aren't good tools for managing bots. robots.nxt provides a free bot traffic analytics dashboard to see how many bots you're getting, what bots they are, what they're costing you, and what content they're crawling. We provide rules for specific bots, bot categories, and specific content or content categories, and more. You can modify and combine these, such as rate limiting or access denial to selected areas, and add a pricing model any combination of rules or identities. For example, if ChatGPT tries to crawl your content pages, you can set robots.nxt so that your server will charge the bot $0.25 per page, but if an indexer crawls your pages there's no charge. And if a bad bot shows up, you can serve it a 404 or dead, link-less homepage so that the bot thinks there's just nothing there. If a bot pays for your content, you can include an AI training license. You can dynamically render robots.txt, explaining what they're allowed to do, while ensuring they can't do anything you don't want. An iron fist in a velvet glove. You can manage your server responses to, for example, filter any form submission element from any pages served to a bot. You don't have to challenge, because the form never gets served to a bot in the first place. We can see any attempts to request insecure files or site areas, so you can see if someone is trying an exploit. Humans will never be the largest source of web traffic ever again. The tools we've built with robots.nxt are an emerging standard for website and web app management in this new bot-dominated internet. We're starting an open beta today, with all our released capabilities free for the time being. Sign up and tell us what you think!

chfritz 2y ago

Funny, just yesterday I was complaining about OpenAI's gptbot's stupidity costing site operators too much: https://x.com/chfritz/status/1863689365740020012 https://x.com/chfritz/status/1863689365740020012

timsylvester 2y ago
Hey sorry I didn't get back to you sooner, I guess HN doesn't sent reply notices. Want to talk about it? Would be good to connect with someone else interested in this.