Travel · x402

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and ho…

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit),…

by crawl-preflight.postnov01.workers.dev · discovered, not yet claimed by its owner · updated 2026-08-31

Mirrored from Coinbase's public x402 Bazaar. The description is the provider's own text and has not been reviewed here; the payment requirements are copied as published.

What it does

Tags: bazaar

Use with an agent

Discover. Agents find this listing with GET https://bazaar.saylorinnovations.com/discovery/search?query=…, the MCP tool search_resources, or /discovery/resources.
Inspect. Fetch /resources/8361.json for the price and payment options.
Pay. Call GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain. The provider answers 402 with its payment requirements. An x402 client signs one of the options below and retries with PAYMENT-SIGNATURE.
Execute. The provider returns 200 with the data. Agent Bazaar is not in the request path and never sees your payment.

For humans

You don't need an account or an API key; you need a wallet holding a small amount of USDC on Base, Polygon, Arbitrum One, and an x402-capable client (see the examples below). Call the endpoint unpaid first to see exactly what it asks for.

Pricing & payment

Price: $0.02 / request
Protocol: x402 v2
Auth: No account or API key. Pay per request.
Networks: Base, Polygon, Arbitrum One
Uptime: 1 of the last 1 checks answered
Calls (30d): 2 · 1 unique payers (third-party data from the public x402 Bazaar)

x402 payment requirements

Scheme	Network	Asset	Amount	Pay to	Timeout
exact	Base eip155:8453	USDC	$0.02	0x02d15E7f25d29743c27021d32a7B280d760B2bC2	60s
exact	Polygon eip155:137	USDC	$0.02	0x02d15E7f25d29743c27021d32a7B280d760B2bC2	60s
exact	Arbitrum One eip155:42161	USDC	$0.02	0x02d15E7f25d29743c27021d32a7B280d760B2bC2	60s

accepts[] (raw JSON)

[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "payTo": "0x02d15E7f25d29743c27021d32a7B280d760B2bC2",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "20000",
    "amountUsd": 0.02,
    "maxTimeoutSeconds": 60
  },
  {
    "scheme": "exact",
    "network": "eip155:137",
    "payTo": "0x02d15E7f25d29743c27021d32a7B280d760B2bC2",
    "asset": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359",
    "amount": "20000",
    "amountUsd": 0.02,
    "maxTimeoutSeconds": 60
  },
  {
    "scheme": "exact",
    "network": "eip155:42161",
    "payTo": "0x02d15E7f25d29743c27021d32a7B280d760B2bC2",
    "asset": "0xaf88d065e77c8cC2239327C5EDb3A432268e5831",
    "amount": "20000",
    "amountUsd": 0.02,
    "maxTimeoutSeconds": 60
  }
]

Schemas

Output schema (JSON Schema)

{
  "bazaar": {
    "info": {
      "input": {
        "method": "GET",
        "pathParams": {
          "domain": "sitepoint.com"
        },
        "queryParams": {},
        "type": "http"
      },
      "output": {
        "example": {
          "crawl_surface": {
            "is_estimate": true,
            "pages": 28400,
            "sitemaps_found": 2
          },
          "domain": "example.com",
          "infrastructure": {
            "cdn": [
              "Cloudflare"
            ],
            "paid_access": []
          },
          "reason": "No AI crawler restrictions declared.",
          "recommendation": "fetch",
          "robots": {
            "ai_crawlers_allowed": 16,
            "ai_crawlers_blocked": [],
            "blocks_all_agents": false,
            "present": true
          }
        },
        "type": "json"
      }
    },
    "routeTemplate": "/crawl-check/:domain",
    "schema": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "properties": {
        "input": {
          "additionalProperties": false,
          "properties": {
            "method": {
              "enum": [
                "GET"
              ],
              "type": "string"
            },
            "pathParams": {
              "properties": {
                "domain": {
                  "description": "Bare domain to check, without scheme or path",
                  "type": "string"
                }
              },
              "required": [
                "domain"
              ],
              "type": "object"
            },
            "queryParams": {
              "properties": {},
              "type": "object"
            },
            "type": {
              "const": "http",
              "type": "string"
            }
          },
          "required": [
            "type",
            "method"
          ],
          "type": "object"
        },
        "output": {
          "properties": {
            "example": {
              "type": "object"
            },
            "type": {
              "type": "string"
            }
          },
          "required": [
            "type"
          ],
          "type": "object"
        }
      },
      "required": [
        "input"
      ],
      "type": "object"
    }
  }
}

Example shape — generated from the schema, not a captured live response

null

Examples

curl

curl "https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain"
# -> 402 Payment Required, accepts[] lists how to pay
# retry with a PAYMENT-SIGNATURE (or PAYMENT header) once paid

JavaScript

const res = await fetch("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain");
if (res.status === 402) {
  const { accepts } = await res.json();
  // pay one of accepts[] via an x402 client, then retry with the payment header
}

Python

import httpx
res = httpx.get("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain")
if res.status_code == 402:
    accepts = res.json()["accepts"]
    # pay one of accepts[] via an x402 client, then retry with the payment header

Details

Provider: crawl-preflight.postnov01.workers.dev
Endpoint: https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
Machine-readable: /resources/8361.json