Seymour, our web crawler

Seymour is the web crawler run by See Me Please (SMP). It reads the public web pages of Australian government agencies, and of other organisations SMP is asked to check.

Seymour reads each page to:

• measure how inclusive the published language is
• check whether footnotes are accessible
• check the alt text (text descriptions) written for images

The name is a play on “see more”. Seymour reads more, so everyone sees more.

User-Agent

Every request Seymour makes sends this User-Agent:

SeymourBot/1.0 (+https://seemeplease.com/bot)

In robots.txt, Seymour answers to the name SeymourBot. Upper or lower case does not matter.

IP addresses

Seymour’s requests come only from the addresses below.

If a request uses Seymour’s name but comes from a different address, it is not from us. If our address changes, we update this list.

• It follows your robots.txt file, using the rules in the Robots Exclusion Protocol (RFC 9309) (opens in a new tab).
• If your robots.txt file is missing, Seymour treats your site as having no rules. If the file exists but cannot be read (for example, a server error or access denied), Seymour does not crawl the site.
• It re-reads robots.txt at least once every 24 hours, so changes reach Seymour within a day.
• It sends at most 1 request every 5 seconds to each website, and only 1 request at a time.
• It honours a Crawl-delay in robots.txt of up to 2 minutes. A longer delay is treated as 2 minutes.
• If your server is busy or returns an error, Seymour slows down. If it sends a Retry-After header, Seymour waits at least that long before trying again. If it asks for a wait longer than 1 hour, Seymour stops retrying and leaves your site alone until that time has passed.
• It reads web pages (HTML) only. It does not run JavaScript, and it does not analyse PDFs or other documents.
• It only visits websites on its approved list.

• It does not log in, fill in forms or submit anything. It only requests pages.
• It does not keep cookies.
• It does not try to collect personal information.
• It does not try to get past paywalls, logins or access controls.

Seymour keeps a copy of the public pages it reads so results can be checked. These copies are stored in the United States (Amazon Web Services, us-east-1).

Use robots.txt

To block Seymour from your whole site, add this to the robots.txt file at the root of your website:

User-agent: SeymourBot
Disallow: /

To block one part of your site only, name the folder:

User-agent: SeymourBot
Disallow: /private/

To slow Seymour down instead, set a Crawl-delay in seconds:

User-agent: SeymourBot
Crawl-delay: 30

Ask us by email

You can also ask us to exclude your site. Email hello@seemeplease.com with the website address you want excluded. We will add it to Seymour’s exclusion list and reply to confirm.

If Seymour is causing a problem for your site, or you have any questions, email hello@seemeplease.com. It helps if you include the date, time and IP address from your logs. You can also reach us through our contact page.