StoqlabBot

StoqlabBot is the crawler operated by Stoqlab. It identifies itself with this user agent:

StoqlabBot/0.1 (+https://stoqlab.com/bot)

What it reads

• Files that sites publish for machines to read, at their standard locations: /robots.txt, /app-ads.txt and /sellers.json.
• Public app store listings, store sitemaps and public top charts.

It does not log in, submit forms, run scripts or follow links into the rest of a site.

How it behaves

• It follows robots.txt on every host and on every redirect.
• Each host has its own small request budget. Most sites receive at most a few requests per day.
• It honours Retry-After, and it backs off on its own after repeated errors.
• It keeps facts from these files, such as which entries a file lists and when they changed. It does not republish pages, images or descriptions.

Opting out

To stop StoqlabBot from reading a site, add this to the site's robots.txt:

User-agent: StoqlabBot
Disallow: /

The change is picked up the next time the bot reads robots.txt, usually within a day.

Contact

Questions about StoqlabBot, or a request about your site: use our contact form.