Keenable crawlers
Last updated: September 2026
Keenable.ai, Inc. operates the crawlers described on this page. We build a search index that answers queries from AI agents and applications, and we deliver web content to third parties, including AI developers. Each of those jobs has its own robots.txt token, so you can allow one without allowing the other.
Questions, load complaints, a request to slow us down, or a report of something claiming to be us: abuse@keenable.ai. We reply within one business day.
Our crawlers
| User-agent token | What it does | If you disallow it |
|---|---|---|
Keenable-SearchBot | Crawls and indexes public pages for the Keenable search index, which answers queries from AI agents and applications. Answers cite the source with a link. | Your pages stop appearing in, and being cited by, results served through Keenable. |
Keenable-User | Fetches one specific page in real time because a person or their agent asked for it — for example, when someone pastes your link. Not a bulk crawler. | Link summaries and on-demand retrieval of your pages break for those users. |
Keenable-Extended | Not a crawler — it never sends a request. A robots.txt token that controls whether content we have already crawled may be included in datasets we deliver to third parties, including AI developers. | Your content is excluded from those deliveries. Indexing and citation are unaffected. |
The full user-agent strings we send:
Mozilla/5.0 (compatible; Keenable-SearchBot/1.0; +https://keenable.ai/bot) Mozilla/5.0 (compatible; Keenable-User/1.0; +https://keenable.ai/bot)
Only the major version appears in the token, so a rule you write today keeps matching after we ship updates.
Controlling us with robots.txt
We follow the Robots Exclusion Protocol (RFC 9309), and we read noindex and nofollow in both X-Robots-Tag and <meta name="robots">. A rule takes effect on our next fetch of your robots.txt.
Block everything
User-agent: Keenable-SearchBot User-agent: Keenable-User User-agent: Keenable-Extended Disallow: /
Stay searchable, but out of datasets we deliver to third parties
User-agent: Keenable-Extended Disallow: /
Allow search, block the rest
User-agent: Keenable-SearchBot Allow: / User-agent: Keenable-User User-agent: Keenable-Extended Disallow: /
Limit us to part of your site
A group naming one of our tokens wins over a wildcard group, per RFC 9309.
User-agent: Keenable-SearchBot Disallow: /private/
Verifying a request really came from us
Never trust the user-agent header alone. Any client can send Keenable-SearchBot, and treating the string as proof lets anyone bypass the rules you wrote. That applies to every crawler, not just ours.
We are publishing two ways to check us — a signed request per the IETF HTTP Message Signatures drafts, and a list of the addresses we crawl from. Until both are live this page will not pretend otherwise: if you need to confirm whether traffic in your logs is ours, mail abuse@keenable.ai with the hostname, timestamps and source addresses, and we will tell you.
How our crawlers behave
- Public pages only. We do not attempt to log in, submit forms, or bypass paywalls, and we do not evade a block placed on us.
- We back off when a host slows down or starts refusing: repeated failures, timeouts and rate-limit responses all widen the delay between our requests to that host.
- We accept gzip, and we identify ourselves on every request.
- A page we cannot fetch is not a page we retry indefinitely — a host that keeps refusing us is crawled less, not more.
Asking us to change something
Mail abuse@keenable.ai to report excessive load, ask for a lower crawl rate, request removal of specific content, or report a crawler misrepresenting itself as ours. Include your hostname and, where you have them, log lines with timestamps and source addresses.
Changes to this page
September 2026 — first publication, covering Keenable-SearchBot, Keenable-User and Keenable-Extended. A token we stop sending stays documented here, because rules naming it stay in your files.