AI Inference · x402 · early usage

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GP…

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

by api.vextorium.com · discovered, not yet claimed by its owner · updated 2026-10-06

30d calls2

30d payers2

MethodPOST

Outputjson

Mirrored from Coinbase's public x402 Bazaar. The description is the provider's own text and has not been reviewed here; the payment requirements are copied as published.

What it does

Tags: robots · crawler · ai-bots · scraping · compliance · bazaar

Call contract

Method: POST
Input: json
Output: json
Last paid call: 2026-10-06

Input example from Bazaar metadata

{
  "agent": "GPTBot",
  "route": "/section/world",
  "url": "https://www.nytimes.com"
}

Output example

{
  "ai_bots": {
    "Amazonbot": "permitido",
    "Applebot-Extended": "bloqueado",
    "Bytespider": "bloqueado",
    "CCBot": "bloqueado",
    "ChatGPT-User": "bloqueado",
    "Claude-SearchBot": "bloqueado",
    "Claude-User": "bloqueado",
    "ClaudeBot": "bloqueado",
    "Diffbot": "bloqueado",
    "DuckAssistBot": "bloqueado",
    "GPTBot": "bloqueado",
    "Google-Extended": "bloqueado",
    "MistralAI-User": "permitido",
    "OAI-SearchBot": "bloqueado",
    "Perplexity-User": "bloqueado",
    "PerplexityBot": "bloqueado",
    "YouBot": "bloqueado",
    "anthropic-ai": "bloqueado",
    "cohere-ai": "bloqueado",
    "meta-externalagent": "bloqueado"
  },
  "blocked_ai_bots": 18,
  "groups": 63,
  "query": {
    "agent": "GPTBot",
    "allowed": false,
    "applied_group": [
      "gptbot"
    ],
    "route": "/section/world"
  },
  "robots_txt": true,
  "site": "https://www.nytimes.com",
  "sitemaps": [
    "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz"
  ]
}

Use with an agent

Discover. Agents find this listing with GET https://bazaar.saylorinnovations.com/discovery/search?query=…, the MCP tool search_resources, or /discovery/resources.
Inspect. Fetch /resources/37268.json for the price and payment options.
Pay. Call GET https://api.vextorium.com/v1/robots-txt. The provider answers 402 with its payment requirements. An x402 client signs one of the options below and retries with PAYMENT-SIGNATURE.
Execute. The provider returns 200 with the data. Agent Bazaar is not in the request path and never sees your payment.

For humans

You don't need an account or an API key; you need a wallet holding a small amount of USDC on Base, Solana, Polygon, Optimism, eip155:5042, and an x402-capable client (see the examples below). Call the endpoint unpaid first to see exactly what it asks for.

Pricing & payment

Price: $0.003 / request
Protocol: x402 v2
Auth: No account or API key. Pay per request.
Networks: Base, Solana, Polygon, Optimism, eip155:5042
Demand: 2 calls · 2 unique payers in 30d (public x402 Bazaar data)

x402 payment requirements

Scheme	Network	Asset	Amount	Pay to	Timeout
exact	Base eip155:8453	USDC	$0.003	0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf	300s
exact	Solana solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp	USDC	$0.003	HcjfmZLgWRZzwj4deEiS7pewc4KBYSw39mW5nTyt3pz2	300s
exact	Polygon eip155:137	USDC	$0.003	0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf	300s
exact	Optimism eip155:10	0x0b2C639c533813f4Aa9D7837CAf62653d097Ff85	3000	0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf	604900s
exact	eip155:5042 eip155:5042	0x3600000000000000000000000000000000000000	3000	0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf	604900s

accepts[] (raw JSON)

[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "3000",
    "amountUsd": 0.003,
    "maxTimeoutSeconds": 300
  },
  {
    "scheme": "exact",
    "network": "solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
    "payTo": "HcjfmZLgWRZzwj4deEiS7pewc4KBYSw39mW5nTyt3pz2",
    "asset": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
    "amount": "3000",
    "amountUsd": 0.003,
    "maxTimeoutSeconds": 300
  },
  {
    "scheme": "exact",
    "network": "eip155:137",
    "payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf",
    "asset": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359",
    "amount": "3000",
    "amountUsd": 0.003,
    "maxTimeoutSeconds": 300
  },
  {
    "scheme": "exact",
    "network": "eip155:10",
    "payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf",
    "asset": "0x0b2C639c533813f4Aa9D7837CAf62653d097Ff85",
    "amount": "3000",
    "maxTimeoutSeconds": 604900
  },
  {
    "scheme": "exact",
    "network": "eip155:5042",
    "payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf",
    "asset": "0x3600000000000000000000000000000000000000",
    "amount": "3000",
    "maxTimeoutSeconds": 604900
  }
]

Schemas

Output schema (JSON Schema)

{
  "properties": {
    "example": {
      "properties": {
        "ai_bots": {
          "properties": {
            "Amazonbot": {
              "type": [
                "string",
                "null"
              ]
            },
            "Applebot-Extended": {
              "type": [
                "string",
                "null"
              ]
            },
            "Bytespider": {
              "type": [
                "string",
                "null"
              ]
            },
            "CCBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "ChatGPT-User": {
              "type": [
                "string",
                "null"
              ]
            },
            "Claude-SearchBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "Claude-User": {
              "type": [
                "string",
                "null"
              ]
            },
            "ClaudeBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "Diffbot": {
              "type": [
                "string",
                "null"
              ]
            },
            "DuckAssistBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "GPTBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "Google-Extended": {
              "type": [
                "string",
                "null"
              ]
            },
            "MistralAI-User": {
              "type": [
                "string",
                "null"
              ]
            },
            "OAI-SearchBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "Perplexity-User": {
              "type": [
                "string",
                "null"
              ]
            },
            "PerplexityBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "YouBot": {
              "type": [
                "string",
                "null"
              ]
            },
            "anthropic-ai": {
              "type": [
                "string",
                "null"
              ]
            },
            "cohere-ai": {
              "type": [
                "string",
                "null"
              ]
            },
            "meta-externalagent": {
              "type": [
                "string",
                "null"
              ]
            }
          },
          "type": [
            "object",
            "null"
          ]
        },
        "blocked_ai_bots": {
          "type": [
            "number",
            "null"
          ]
        },
        "groups": {
          "type": [
            "number",
            "null"
          ]
        },
        "query": {
          "properties": {
            "agent": {
              "type": [
                "string",
                "null"
              ]
            },
            "allowed": {
              "type": [
                "boolean",
                "null"
              ]
            },
            "applied_group": {
              "items": {
                "type": [
                  "null",
                  "string"
                ]
              },
              "type": [
                "array",
                "null"
              ]
            },
            "route": {
              "type": [
                "string",
                "null"
              ]
            }
          },
          "type": [
            "object",
            "null"
          ]
        },
        "robots_txt": {
          "type": [
            "boolean",
            "null"
          ]
        },
        "site": {
          "type": [
            "string",
            "null"
          ]
        },
        "sitemaps": {
          "items": {
            "type": [
              "null",
              "string"
            ]
          },
          "type": [
            "array",
            "null"
          ]
        }
      },
      "type": [
        "object",
        "null"
      ]
    },
    "type": {
      "type": "string"
    }
  },
  "required": [
    "type"
  ],
  "type": "object"
}

Example shape — generated from the schema, not a captured live response

{
  "example": {
    "ai_bots": {
      "Amazonbot": "string",
      "Applebot-Extended": "string",
      "Bytespider": "string",
      "CCBot": "string",
      "ChatGPT-User": "string",
      "Claude-SearchBot": "string",
      "Claude-User": "string",
      "ClaudeBot": "string",
      "Diffbot": "string",
      "DuckAssistBot": "string",
      "GPTBot": "string",
      "Google-Extended": "string",
      "MistralAI-User": "string",
      "OAI-SearchBot": "string",
      "Perplexity-User": "string",
      "PerplexityBot": "string",
      "YouBot": "string",
      "anthropic-ai": "string",
      "cohere-ai": "string",
      "meta-externalagent": "string"
    },
    "blocked_ai_bots": 0,
    "groups": 0,
    "query": {
      "agent": "string",
      "allowed": true,
      "applied_group": [
        "value"
      ],
      "route": "string"
    },
    "robots_txt": true,
    "site": "string",
    "sitemaps": [
      "value"
    ]
  },
  "type": "string"
}

Examples

curl

curl "https://api.vextorium.com/v1/robots-txt"
# -> 402 Payment Required, accepts[] lists how to pay
# retry with a PAYMENT-SIGNATURE (or PAYMENT header) once paid

JavaScript

const res = await fetch("https://api.vextorium.com/v1/robots-txt");
if (res.status === 402) {
  const { accepts } = await res.json();
  // pay one of accepts[] via an x402 client, then retry with the payment header
}

Python

import httpx
res = httpx.get("https://api.vextorium.com/v1/robots-txt")
if res.status_code == 402:
    accepts = res.json()["accepts"]
    # pay one of accepts[] via an x402 client, then retry with the payment header

Details

Provider: api.vextorium.com
Endpoint: https://api.vextorium.com/v1/robots-txt
Machine-readable: /resources/37268.json