Crawler

BizzikBot

User-agent: BizzikBot/0.1 (+https://bizzik.com/bot)

Bizzik fetches a company's robots.txt, the sitemaps it declares, and a small set of HTML pages to compare public marketing signals: the homepage, a few sample URLs from the sitemap, and up to 14 key pages per site for the page audit (such as about, contact, pricing, and signup).

HTML fetches obey robots.txt Allow and Disallow rules, including wildcards. Declared sitemaps are read so URLs can be counted and classified. Pages matching Disallow are not fetched. Crawl-delay is applied to HTML requests, capped at 12 seconds.

A bot challenge such as a Cloudflare interstitial is recorded as unknown. Bizzik does not try to solve it.

App store figures come from the iTunes Lookup API, Apple's public customer-review RSS, the public App Store page, and the google-play-scraper library. Ad counts are not scraped.