web-search-and-content

Fetch any public URL and extract structured metadata — Open Graph tags, Twitter Card tags, canonical URL, title, description, favicon, article author/date, JSON-LD structured data, and content type — so agents can preview, classify, and enrich links without building their own scraper.

Fetch and parse any XML sitemap or sitemap index file — returns URLs with their priority, change frequency, and last modified date, follows sitemap indexes one level deep (first 3 child sitemaps), and auto-discovers the sitemap from robots.txt or common paths when given a bare domain — so agents can enumerate site content for crawling, indexing, and SEO analysis.

Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.

Fetch and parse any RSS 2.0 or Atom feed URL and return structured articles with title, link, description, author, publish date, and categories — so agents can monitor content sources, build news pipelines, and process any feed without building their own XML parser.

Fetch any public URL and extract structured metadata — Open Graph tags, Twitter Card tags, canonical URL, title, description, favicon, article author/date, JSON-LD structured data, and content type — so agents can preview, classify, and enrich links without building their own scraper.

Renders a webpage exactly as a real browser would, including all its scripts and dynamic content, then takes a picture of it. You can capture just the visible area or the whole scrollable page, choose the screen size, wait for slow-loading images to finish, and hide cookie banners or popups before the picture is taken.

Discover RTP robots. Filter by capability, chain, price, status.

Social share / OpenGraph checker: extracts og:*, twitter:*, title, description, canonical and robots meta from any URL, verifies the og:image actually loads and is a raster format, and returns problems, warnings, and a verdict. For publishing and SEO agents shipping pages that get shared.

Real-time web search for AI agents — search the web and get ranked SERP-style results as clean JSON: title, URL, snippet, relevance score, optional LLM-generated answer. Find up-to-date information online — latest news, current events, finance. Date, domain and exact-phrase filters.

Read any web page or article: fetch up to 5 URLs in one batch call and extract the main content as clean, LLM-ready markdown or plain text. Convert webpages to markdown, scrape page text, pull article content for RAG ingestion and AI agent context.

Detect the language of any text input using stopword-set matching and Unicode script analysis — returns the detected language, ISO code, a high/medium/low confidence level, and top alternative languages so agents can route multilingual content, trigger translation workflows, and classify text without a paid NLP API.

Text classification API — zero-shot text classifier / categorization: caller supplies 2–20 labels and Claude Haiku returns the best-matching category with confidence and per-label scores. Route, tag, triage, and detect intent/topic with your own taxonomy in one call.

Detect the language of any text input using character frequency analysis and n-gram pattern matching — returns the detected language, ISO code, confidence score, and top alternative languages so agents can route multilingual content, trigger translation workflows, and classify text without a paid NLP API.

Text summarizer / summarization API — condense text, Markdown, or a URL into a TL;DR plus key bullet points using Claude Haiku. Accepts raw text or a URL (fetched and extracted), enforces an input cap, and returns a clean structured summary so agents can compress documents and articles in one call.

Fetch a URL and extract clean main-content text with title, description, word count, and char count; boilerplate (nav/footer/sidebar/ads) is stripped.

Extract structured elements: heading hierarchy (h1-h6), lists, tables as 2D arrays, and images with alt text, plus element counts.

Fast real-time news scan across 44+ curated domains (configs/search_domain_profiles.json), every item source-attributed. Settlement proof: ProofOfSettledOutcome (kind 30120, data/proof_settled_outcomes.jsonl). The quick-scan tier below research.topic-deep-researcher's slower, citation-grade report.

Neural web search for agents (Exa upstream): pass a query, get the most relevant live web pages with title, URL, published date, author, relevance score and a clean text snippet as structured JSON. The find-then-read pair for /web-extract — RAG retrieval, research, fact-finding, monitoring. Keyless, pay-per-call, cached by query. A no-match query returns an empty result set as a valid answer.

Render a URL in a real headless browser (JavaScript executed) and return its article content with the clutter stripped — same cleaning + output formats as /api/url/clean, but for client-rendered / SPA pages whose content only appears after JS runs (where a raw HTTP fetch returns an empty shell). `format`: markdown (default), text, both (JSON envelope), html (self-contained reader page, raw text/html), or pdf (typeset reading doc, raw application/pdf). Optional `waitUntil` (load|domcontentloaded|networkidle0|networkidle2, default networkidle2) and `timeoutMs` (1000-15000, default 12000) control how long to let JS settle. Use /api/url/clean instead for server-rendered pages — it is faster and cheaper; only reach for render when JS rendering is required.

AI web search optimized for agents. Returns ranked results with the relevant extracted content of each page (not just a link + blurb), plus a relevance score. topic=news for recent reporting. Distinct from search.web (raw SERP) — this returns clean, LLM-ready page content per result.

Classify text content into categories with confidence scores and readability metrics

Extract structured text, tables, and metadata from any PDF URL. Returns page-by-page text content, detected tables as JSON arrays, and document metadata (title, author, page count). No PDF library or OCR setup needed — pay per extraction. AI agent API for document intelligence and data extraction

Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.

Search ArXiv for academic papers and preprints in AI, machine learning, CS, and mathematics. AI agent API for scientific research. Accepts USDC payments on Base and Solana