Seymour, our web crawler

Seymour is the web crawler run by See Me Please (SMP). It reads the public web pages of Australian government agencies, and of other organisations SMP is asked to check.

Seymour reads each page to:

• measure how inclusive the published language is
• check whether footnotes are accessible
• check the alt text (text descriptions) written for images

The name is a play on “see more”. Seymour reads more, so everyone sees more.

User-Agent

Every request Seymour makes sends this User-Agent:

SeymourBot/1.0 (+https://seemeplease.com/bot)

Seymour is moving to its own web address, seymour.seemeplease.com. After the move, its User-Agent will point there instead:

SeymourBot/1.0 (+https://seymour.seemeplease.com/bot)

Both are genuine. seemeplease.com and seymour.seemeplease.com are both run by See Me Please, and everything on this page applies to Seymour under either address.

In robots.txt, Seymour answers to the name SeymourBot. Upper or lower case does not matter.

IP addresses

Seymour’s requests come only from the addresses below.

If a request uses Seymour’s name but comes from a different address, it is not from us. If our address changes, we update this list.

The same list is published as JSON, in the format Google and OpenAI use for their crawlers, so you can load it into a firewall or bot rule: https://seemeplease.com/bot/seymour.json

Reverse DNS

Each address has a reverse DNS name, and that name points back to the same address. Live crawls use names ending in crawl.seemeplease.com. Testing runs use names ending in crawl.dev.seemeplease.dev.

To check a request, look up the address, then look up the name you get back:

dig -x 100.50.99.98 +short
dig use1.crawl.seemeplease.com A +short

Both answers must match: the name must end in one of the suffixes above, and it must point back to the address in your logs.

Signed requests

Seymour signs every request with an HTTP message signature, using the Web Bot Auth method that some web security services check. Each signed request carries a Signature-Agent header naming https://seemeplease.com. The public keys that verify the signatures are published here:

https://seemeplease.com/.well-known/http-message-signatures-directory

From header

Every request also sends a From header with an address you can write to about Seymour: seymour@seemeplease.com.

• It follows your robots.txt file, using the rules in the Robots Exclusion Protocol (RFC 9309) (opens in a new tab).
• If your robots.txt file is missing, Seymour treats your site as having no rules. If the file exists but cannot be read (for example, a server error or access denied), Seymour does not crawl the site.
• It re-reads robots.txt at least once every 24 hours, so changes reach Seymour within a day.
• It sends at most 1 request every 5 seconds to each website, and only 1 request at a time.
• It honours a Crawl-delay in robots.txt of up to 2 minutes. A longer delay is treated as 2 minutes.
• If your server is busy or returns an error, Seymour slows down. If it sends a Retry-After header, Seymour waits at least that long before trying again. If it asks for a wait longer than 1 hour, Seymour stops retrying and leaves your site alone until that time has passed.
• It reads web pages (HTML) only. It does not run JavaScript, and it does not analyse PDFs or other documents.
• It only visits websites on its approved list.

• It does not log in, fill in forms or submit anything. It only requests pages.
• It does not keep cookies.
• It does not try to collect personal information.
• It does not try to get past paywalls, logins or access controls.

Seymour keeps a copy of the public pages it reads so results can be checked. These copies are stored in the United States (Amazon Web Services, us-east-1).

Seymour does not train AI models. It reads pages only to produce the findings described at the top of this page.

Some sites state their preferences about AI in robots.txt, using Content Signals (for example Content-Signal: ai-train=no) or the IETF AI Preferences vocabulary. Seymour’s use already fits a “no AI training” preference, because it never trains. Seymour does not yet read these lines, so to stop Seymour visiting, use the robots.txt rules in the next section.

Use robots.txt

To block Seymour from your whole site, add this to the robots.txt file at the root of your website:

User-agent: SeymourBot
Disallow: /

To block one part of your site only, name the folder:

User-agent: SeymourBot
Disallow: /private/

To slow Seymour down instead, set a Crawl-delay in seconds:

User-agent: SeymourBot
Crawl-delay: 30

Ask us by email

You can also ask us to exclude your site. Email hello@seemeplease.com with the website address you want excluded. We will add it to Seymour’s exclusion list and reply to confirm.

If you think something Seymour found is wrong, or a page Seymour read includes your personal information, you can ask us to review it. Anyone can ask. You do not need to work for the organisation that runs the website.

Use our review request form. If you cannot use the form, email hello@seemeplease.com and tell us:

• the address of the page or report you are asking about
• what you think is wrong, or what you would like us to change
• how we can reply to you

A person at See Me Please reviews every request and tells you what we decided. We may correct a finding, or stop showing a finding or a quote in what we publish. We keep the copies of public pages that Seymour has read, so that our results can still be checked.

If you only want Seymour to stop visiting your site, see how to stop Seymour visiting your site.

If Seymour is causing a problem for your site, or you have any questions, email hello@seemeplease.com. It helps if you include the date, time and IP address from your logs. You can also reach us through our contact page.