Access to Premium Web Content by a Paying Web Crawler

This reference implementation demonstrates how a web crawler can access protected or premium website content by presenting valid kya-pay tokens with every request.

Instead of blocking all automation, the website allows access to only crawlers that provide valid identity and payment credentials.

Artifacts

Demo Video

Watch how verified payment tokens allow bots to access protected publisher content.

Live Demo

Run the crawler yourself and observe payment-gated access in action.

GitHub Repo

Explore the full crawler and bot manager source code.

This demo showcases the versatility of the KYAPay protocol and the near universal applicability of the protocol in enabling agentic commerce.

How It Works

The web crawler generates kya-pay tokens via the Skyfire API.
The crawler includes the token in each request to the protected website.
The bot blocker/WAF (Web Application Firewall):
1. Verifies the token signature
2. Verifies token claims
If valid, access is granted.
If the token is missing or invalid, then the bot blocker/WAF returns error codes according to this.

What This Demo Includes

This demo includes source code for:

Web crawler: Submits kya-pay tokens when accessing protected content
Bot blocker: Verifies tokens before granting access