3 ms·

Why not adhere to the HTTP standard for once and return a 429 Too many requests or 403 Forbidden? 404 Not Found should only be used for “Not Found” errors. If a

by pgraf 3y ago

Why not adhere to the HTTP standard for once and return a 429 Too many requests or 403 Forbidden? 404 Not Found should only be used for “Not Found” errors. If adhered to this posts advice, you can really screw your search ranking if a major web crawler gets blocked and de-lists all of your pages because of 404

Alifatisk 3y ago
I think you can still adhere to the HTTP standards (Rails have built-in keywords for those aswell). This article just talks about blocking hosts that make too many requests to a 404 page (I guess because it's crawling?) and how to mitigate those using Rack Attack.
- pgraf 3y ago
  Could you please explain how sending a 404 to clients sending too many requests would adhere to the HTTP standards? I’m not against the blocking itself. For reference: https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.4 https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.4
  - Alifatisk 3y ago
    When I say you can still adhere to the HTTP standards, I mean that you just have to adjust the suggested settings in the config, so requests reaching over the throttle should give 429 instead of 404: Like the following: # config/routes.rb get "429", to: "welcome#429", code: 429 rescue_from ActiveRecord::RecordNotFound, with: :too_many_requests def too_many_requests redirect_to "/warning" end # config/initializers/rack-attack.rb class Rack::Attack ... blocklist("block 429") do |request| Allow2Ban.filter("too_many_requests-#{request.ip}", maxretry: 5, findtime: 3.minutes, bantime: 1.day) do request.path == '/429' end ... end # config/initializers/rack-attack.rb Rack::Attack.blocklisted_responder = lambda do |request| [ 429, {}, ["You are blocked. If you think are not a bot and you think it was due to a mistake, reach out to us at support@yourdomain.com"]] end The way I interpreted the article wasn't to redirect everything to a 404 page, but how to handle massive amount of requests to a 404 page!

by pgraf 3y ago

Alifatisk 3y ago

I think you can still adhere to the HTTP standards (Rails have built-in keywords for those aswell). This article just talks about blocking hosts that make too many requests to a 404 page (I guess because it's crawling?) and how to mitigate those using Rack Attack.

pgraf 3y ago
Could you please explain how sending a 404 to clients sending too many requests would adhere to the HTTP standards? I’m not against the blocking itself. For reference: https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.4 https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.4
- Alifatisk 3y ago
  When I say you can still adhere to the HTTP standards, I mean that you just have to adjust the suggested settings in the config, so requests reaching over the throttle should give 429 instead of 404: Like the following: # config/routes.rb get "429", to: "welcome#429", code: 429 rescue_from ActiveRecord::RecordNotFound, with: :too_many_requests def too_many_requests redirect_to "/warning" end # config/initializers/rack-attack.rb class Rack::Attack ... blocklist("block 429") do |request| Allow2Ban.filter("too_many_requests-#{request.ip}", maxretry: 5, findtime: 3.minutes, bantime: 1.day) do request.path == '/429' end ... end # config/initializers/rack-attack.rb Rack::Attack.blocklisted_responder = lambda do |request| [ 429, {}, ["You are blocked. If you think are not a bot and you think it was due to a mistake, reach out to us at support@yourdomain.com"]] end The way I interpreted the article wasn't to redirect everything to a 404 page, but how to handle massive amount of requests to a 404 page!