Travel · x402 · active demand

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-pa…

by minifetch.com · discovered, not yet claimed by its owner · updated 2026-10-13

30d calls45

30d payers14

MethodGET

Outputjson

Mirrored from Coinbase's public x402 Bazaar. The description is the provider's own text and has not been reviewed here; the payment requirements are copied as published.

What it does

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as given — does not follow redirects. Supports GET & POST

Tags: seo · robots-txt · preflight · crawlability · indexability · bazaar

Call contract

Method: GET
Input: query parameters
Output: json
Last paid call: 2026-10-13

Input example from Bazaar metadata

{
  "url": "https://github.com"
}

Output example

{
  "results": [
    {
      "data": {
        "allowed": true,
        "crawlDelay": 1,
        "message": "Access allowed by robots.txt",
        "minifetchCache": {
          "cachedAt": "2026-08-16T18:22:10.000Z",
          "expiresAt": "2026-08-17T18:22:10.000Z",
          "hit": false
        },
        "proxy": {
          "proxyType": "minifetch",
          "userAgent": "minifetch/1.0 (+https://minifetch.com/site-owner-faq)"
        },
        "url": "https://github.com"
      }
    }
  ],
  "success": true
}

Use with an agent

Discover. Agents find this listing with GET https://bazaar.saylorinnovations.com/discovery/search?query=…, the MCP tool search_resources, or /discovery/resources.
Inspect. Fetch /resources/26443.json for the price and payment options.
Pay. Call GET https://minifetch.com/api/v1/x402/preflight/url-check. The provider answers 402 with its payment requirements. An x402 client signs one of the options below and retries with PAYMENT-SIGNATURE.
Execute. The provider returns 200 with the data. Agent Bazaar is not in the request path and never sees your payment.

For humans

You don't need an account or an API key; you need a wallet holding a small amount of USDC on Base, Solana, and an x402-capable client (see the examples below). Call the endpoint unpaid first to see exactly what it asks for.

Pricing & payment

Price: $0.001 / request
Protocol: x402 v2
Auth: No account or API key. Pay per request.
Networks: Base, Solana
Demand: 45 calls · 14 unique payers in 30d (public x402 Bazaar data)

x402 payment requirements

Scheme	Network	Asset	Amount	Pay to	Timeout
exact	Base eip155:8453	USDC	$0.001	0xffA9155A0e4B3d8d42351Da226cC1feEe449686f	300s
exact	Solana solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp	USDC	$0.001	46vMcwuC4sK11sB3gkLhyA7J7GEwfkhn5rFyDtihBwqe	300s

accepts[] (raw JSON)

[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "payTo": "0xffA9155A0e4B3d8d42351Da226cC1feEe449686f",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "1000",
    "amountUsd": 0.001,
    "maxTimeoutSeconds": 300
  },
  {
    "scheme": "exact",
    "network": "solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
    "payTo": "46vMcwuC4sK11sB3gkLhyA7J7GEwfkhn5rFyDtihBwqe",
    "asset": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
    "amount": "1000",
    "amountUsd": 0.001,
    "maxTimeoutSeconds": 300
  }
]

Schemas

Output schema (JSON Schema)

{
  "properties": {
    "example": {
      "properties": {
        "results": {
          "items": {
            "properties": {
              "data": {
                "properties": {
                  "allowed": {
                    "description": "Whether the minifetch user agent may fetch this URL per robots.txt",
                    "type": "boolean"
                  },
                  "crawlDelay": {
                    "description": "Crawl-delay (seconds) for the minifetch UA; omitted when blocked or the domain is invalid",
                    "type": "number"
                  },
                  "message": {
                    "description": "e.g. 'Access allowed by robots.txt', 'URL is blocked by robots.txt', or 'robots.txt not found, defaulting to allowed'",
                    "type": "string"
                  },
                  "minifetchCache": {
                    "description": "robots.txt cache state (~24h TTL; pass fresh=true to bypass)",
                    "properties": {
                      "cachedAt": {
                        "type": "string"
                      },
                      "expiresAt": {
                        "type": "string"
                      },
                      "hit": {
                        "type": "boolean"
                      }
                    },
                    "type": "object"
                  },
                  "proxy": {
                    "description": "Fetch provenance — always minifetch UA here",
                    "properties": {
                      "proxyType": {
                        "enum": [
                          "minifetch"
                        ],
                        "type": "string"
                      },
                      "userAgent": {
                        "type": "string"
                      }
                    },
                    "type": "object"
                  },
                  "url": {
                    "description": "The URL checked, exactly as given — redirects are not followed",
                    "type": "string"
                  }
                },
                "required": [
                  "url",
                  "allowed",
                  "message",
                  "proxy"
                ],
                "type": "object"
              }
            },
            "type": "object"
          },
          "type": "array"
        },
        "success": {
          "type": "boolean"
        },
        "tip": {
          "description": "Present only when allowed=false: how to retry via the external proxy routes, and how site owners can unblock the minifetch user agent",
          "properties": {
            "message": {
              "type": "string"
            },
            "url": {
              "type": "string"
            }
          },
          "type": "object"
        }
      },
      "type": "object"
    },
    "type": {
      "type": "string"
    }
  },
  "required": [
    "type"
  ],
  "type": "object"
}

Example shape — generated from the schema, not a captured live response

{
  "example": {
    "results": [
      {
        "data": {
          "allowed": true,
          "crawlDelay": 0,
          "message": "string",
          "minifetchCache": {
            "cachedAt": "string",
            "expiresAt": "string",
            "hit": true
          },
          "proxy": {
            "proxyType": "minifetch",
            "userAgent": "string"
          },
          "url": "string"
        }
      }
    ],
    "success": true,
    "tip": {
      "message": "string",
      "url": "string"
    }
  },
  "type": "string"
}

Examples

curl

curl "https://minifetch.com/api/v1/x402/preflight/url-check"
# -> 402 Payment Required, accepts[] lists how to pay
# retry with a PAYMENT-SIGNATURE (or PAYMENT header) once paid

JavaScript

const res = await fetch("https://minifetch.com/api/v1/x402/preflight/url-check");
if (res.status === 402) {
  const { accepts } = await res.json();
  // pay one of accepts[] via an x402 client, then retry with the payment header
}

Python

import httpx
res = httpx.get("https://minifetch.com/api/v1/x402/preflight/url-check")
if res.status_code == 402:
    accepts = res.json()["accepts"]
    # pay one of accepts[] via an x402 client, then retry with the payment header

Details

Provider: minifetch.com
Endpoint: https://minifetch.com/api/v1/x402/preflight/url-check
Machine-readable: /resources/26443.json