Skip to content

klyrek-crawler

Status: built, not yet on PyPI (blocked on PyPI's new-project rate limit) — install from source

Breadth-first crawl that maps pages, forms, and JavaScript references within the declared scope.

from klyrek_crawler.crawler import Crawler

result = Crawler(client, max_pages=100, max_depth=3).crawl(target)

Real endpoint sources it produces: crawler (a visited page), crawler-script (a <script src> found on a page), crawler-form (a <form>'s action URL, with params set to the form's field names).

Against the sandbox, a real crawl of shop.sandbox.klyrek.com (with --include-host "*.sandbox.klyrek.com") produces 12 crawler, 4 crawler-form, and 1 crawler-script endpoint — see The Scan Pipeline for how these feed the rest of the packages.

API reference

klyrek_crawler.crawler

Breadth-first crawler that maps a target's pages, forms, and script references.

Crawler 

Crawler(client: KlyrekHTTPClient, max_pages: int = 100, max_depth: int = 3)

Breadth-first crawl of a target, discovering pages, forms, and script references.

Every fetch goes through the given KlyrekHTTPClient, so scope enforcement and rate limiting are inherited automatically. The crawler additionally checks scope before enqueuing a link, so it doesn't spend its page budget on hosts it isn't authorized to touch.

Source code in klyrek_crawler\crawler.py 
43
44
45
46
def __init__(self, client: KlyrekHTTPClient, max_pages: int = 100, max_depth: int = 3) -> None:
    self.client = client
    self.max_pages = max_pages
    self.max_depth = max_depth