ContentMonkBot

ContentMonkBot runs the free SEO and AEO site checks on contentmonk.io. It visits a site only when a check of that site is started, and it does not crawl the web on its own.

Operator

ContentMonk, contentmonk.io. Contact: ugi@contentmonk.io

User agent

Mozilla/5.0 (compatible; ContentMonkBot/1.0; +https://answers.contentmonk.io/bot)

robots.txt token: ContentMonkBot

What it does

The SEO check reads robots.txt, the sitemaps and up to 200 pages of the site without running JavaScript, and checks the links it found.

The AEO check looks at the one page that was entered. It reads the site's robots.txt, asks for the page once as ContentMonkBot without running JavaScript, and opens it once in Google Chrome, to compare what a crawler reads with what a visitor sees. Chrome sends its own user agent with the ContentMonkBot token added:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version>.0.0.0 Safari/537.36 ContentMonkBot/1.0 (+https://answers.contentmonk.io/bot)

The AEO check also reports what the site's robots.txt says to the crawlers of AI and search companies. It never sends a request in another company's user agent.

Blocking it

To stop ContentMonkBot, add this to robots.txt:

User-agent: ContentMonkBot
Disallow: /

Both checks read a site's robots.txt before they ask for a page, and follow the ContentMonkBot group (or the * group when there is none).

The SEO check follows it for every request and every redirect, and does not crawl a site whose robots.txt it cannot read.

The AEO check asks for the page only when that group allows it, and does not follow a redirect to a page the group disallows. When it does not ask for the page, it does not open it in Chrome either. When robots.txt cannot be read, it still looks at the one page that was entered.

How fast it goes

• One request at a time to each host, and at least 0.5 seconds between the start of two requests.
• A Crawl-delay is honoured up to 60 seconds. A longer one ends the check for that host.
• Retry-After is honoured. The first 429 pauses the host for at least 10 seconds, and a second 429 ends the check for that host.
• 3 failed requests in a row (server errors or no answer) end the check for that host.
• At most 200 pages and 600 requests per check.

Verifying it

ContentMonkBot signs its requests with Web Bot Auth (HTTP Message Signatures). Our keys: https://answers.contentmonk.io/.well-known/http-message-signatures-directory

The address this checker sends from: /bot/ips.json

When a site blocks or challenges ContentMonkBot, the report says so. We never solve CAPTCHAs or challenges, never move to other IP addresses, and never change our user agent or identity to get past a block.

Data use

What ContentMonkBot reads goes into the report for the person who started the check. It never logs in or fills in forms, and it keeps no cookies from one check to the next.