Booter – Bots & Crawlers Manager

Descrição

Booter – Bots & Crawlers Manager is a preventative measure (treatment in advance) and treatment of damages caused by crawlers and bots.
The plugin uses a number of existing technologies which are known by crawlers and bots and takes them one step forward – smartly and almost completely automatically.
To allow the plugin to function correctly, you must follow the instructions and manually enter some data (which must be done by a human being to avoid errors).

At the prevention level

  • Booter allows you to manage and create an advanced dynamic robots.txt file.
  • View a 404 error log to see the most common bad links.
  • Blocking bad bots that cause high server loads due to very frequent page crawls, or are used to search for security vulnerabilities.

At the treatment level

  • Booter allows you to limit the amount of requests from crawlers and bots, if or when they exceed the specified amount of requests per minute, it will be rejected for a specified period of time.
  • Rejecting links that we do not want in the fastest way, not by just blocking but by sending the appropriate HTTP status code to make search engines forget them.

Instructions for use in case of damage treatment

  1. Activate the plugin.
  2. Enable the 404 error log option.
  3. Set the access rate limit.
  4. Watch the 404 log, try to find common parts in the URLs that repeats most often.
  5. Enter the common parts to the “reject links” page, and ensure the rejection code is 410.
  6. Clear the 404 error log.
  7. Repeat the process once every few hours until the 404 error log remains blank.
  8. Check the status of your website’s index coverage every few days.

Ecrãs

  • Plugin General Settings
  • Robots.txt Management
  • Reject Links Settings

Instalação

  1. Upload booter-crawlers-manager folder to the /wp-content/plugins/ directory
  2. Activate the plugin through the ‘Plugins’ menu in WordPress
  3. The plugin will start rate limiting as soon as it is activated, however it is recommended to update the settings to suit your needs, under ‘Settings’ -> ‘Booter – Crawlers Manager’ menu

Avaliações

Ler todas as 3 avaliações

Contribuidores e programadores

“Booter – Bots & Crawlers Manager” é software de código aberto. As seguintes pessoas contribuíram para este plugin:

Contribuidores

Registo de alterações

1.4.3

  • Fix rate limit/block applied to cli requests

1.4.2

  • Fix error breaking rejected strings input
  • Updated tested WP version

1.4.1

  • Fix typos
  • Added option to block all users without a useragent
  • Added additional strings to default settings

1.4

  • New logo and banner
  • Added auto detection for sitemaps from All-in-one-SEO, Jetpack
  • Added a debug option to log which rule cause a block
  • Updated the lists of robots
  • Updated default settings
  • Updated error responses
  • UI improvements