klyrek-http

Status: live on PyPI — pip install klyrek-http

The scope-checked HTTP client every other package builds its requests through. Rate-limited per host, and it re-checks AuthorizationScope on every redirect hop, not just the first request.

from klyrek_http.client import KlyrekHTTPClient
from klyrek_core.config import KlyrekConfig

with KlyrekHTTPClient(scope, config=KlyrekConfig(rate_limit_per_host=5.0)) as client:
    response = client.get("https://shop.sandbox.klyrek.com/")

Every request — including ones your own code makes through this client, not just the crawler's — is checked against scope on the initial URL and on every redirect target via an httpx event hook.

API reference

klyrek_http.client

A scope-checked, rate-limited HTTP client shared across the Klyrek ecosystem.

KlyrekHTTPClient

KlyrekHTTPClient(scope: AuthorizationScope, config: KlyrekConfig | None = None, transport: BaseTransport | None = None)

Every Klyrek package that talks to a target should issue requests through this client.

It enforces the scan's :class:AuthorizationScope and per-host rate limit on every request, so no individual package has to remember to do so itself.

Source code in klyrek_http\client.py

def __init__(
    self,
    scope: AuthorizationScope,
    config: KlyrekConfig | None = None,
    transport: httpx.BaseTransport | None = None,
) -> None:
    self.scope = scope
    self.config = config or KlyrekConfig()
    min_interval = (
        1.0 / self.config.rate_limit_per_host if self.config.rate_limit_per_host > 0 else 0.0
    )
    self.rate_limiter = RateLimiter(min_interval=min_interval)
    self._client = httpx.Client(
        headers={"User-Agent": self.config.user_agent},
        timeout=self.config.timeout_seconds,
        verify=self.config.verify_tls,
        follow_redirects=True,
        transport=transport,
        event_hooks={"request": [self._enforce_scope_and_rate_limit]},
    )