web-search-and-content

Enumerate a site's URLs from robots.txt + sitemap.xml (sitemap-index aware) — map a domain before crawling it

Enumerate a site's URLs from robots.txt + sitemap.xml (sitemap-index aware) — map a domain before crawling it

Hacker News full-archive search: stories, comments, Show HN and Ask HN with points, comment counts and URLs

Search arXiv preprints by query, filtered by field (title/abstract/author/all) and category. Returns title, authors (up to 6), abstract (first 600 chars), arXiv ID, PDF link, publish date, and subject categories. arXiv is the canonical source for AI/ML, CS, physics, math, and quantitative biology preprints — typically months ahead of peer-reviewed journals. Useful for AI research agents, literature scouts, competitive technique tracking, and grant-writing support. No API key required.

extract emails, extract urls, extract phone numbers, extract crypto addresses, extract ethereum addresses, regex extraction, text entity extractor. Extract emails, URLs, phone numbers, hashtags, mentions, IP addresses, ethereum and bitcoin addresses from any text. Returns categorized arrays. For AI agents parsing emails, scraping documents, content moderation.

extract meta tags from URL, HTML meta tags extractor, extract OG tags, extract Open Graph tags, extract Twitter Card meta tags, link preview meta tags. Parse any public webpage and extract title, description, keywords, canonical, favicon, language, OpenGraph (og:*), Twitter Card. For AI agents, link previews, SEO audits, content discovery.

Prepares a document for question-answering and RAG pipelines. Chunks the input text at paragraph/sentence boundaries, assigns deterministic chunk IDs, estimates token counts, and extracts document metadata (word count, type, headings). Returns ready-to-embed chunks with overlap support. No LLM or external API — pure text processing. Use mid-task when you've fetched a document and need it split before querying a vector store.

GlianaAI - pay-per-call AI: LLM chat (OpenAI-compatible), image, video, music, speech - 80+ models, no signup, USDC. This resource: recraftv4-pro (Text-to-Image). https://ai.glianalabs.com

Convert text to natural-sounding audio via ElevenLabs (returns base64 MP3). Pay per call with USDC (x402) or credits - archtools.dev

YouTube channel scraper — subscribers, description, links + the email, Instagram and contact details extracted from the channel About section (the lead-gen wedge). Real-time, no login. SEO: YouTube scraper API, youtube channel, youtube email scraper, creator data, influencer lead generation.

Retrieve recent news and developments for a company or topic. AI-powered aggregation from multiple real-time sources.

Extracts structured breadcrumb navigation from a URL. Returns domain, ordered path segments with human-readable labels, query parameters as key-value pairs, and a formatted breadcrumb trail string. Identifies numeric IDs vs. named path segments. Pure URL parsing — zero external calls. Useful for agents that process sitemaps, navigation menus, or need to understand page hierarchy.

Extract structured company intelligence from any public website. Returns company name, description, logo, emails, phones, address, founded date, social links (Twitter/LinkedIn/GitHub/etc.), and raw OpenGraph + schema.org/Organization data. Pure HTML extraction — no external APIs. $0.003 hedge against orbisapi web-scrape-company at $0.005.

Hacker News story and comment search via Algolia. Returns titles, scores, comment counts, authors, and URLs for posts matching the query. Filter by type (story/comment) and date range (day/week/month/year/all). Sorted by relevance by default; use sort=date for newest-first. Useful for tech news, community sentiment, or discovering discussion threads about a topic.

Transcribe audio from any publicly accessible URL using OpenAI Whisper. Supports mp3, mp4, m4a, wav, webm, ogg, flac, and wma up to 24 MB. Returns the full transcript text, detected language, and estimated duration in seconds. Optionally accepts an ISO 639-1 language hint to improve accuracy. Useful for processing voice memos, meeting recordings, podcast snippets, interview clips, and audio attached to social media. Undercuts orbisapi.com audio-transcription-api by 24%.

Academic paper search across 250M+ works via OpenAlex (free, no key). Returns top papers with title, authors, year, DOI, citation count, open-access status, and primary research topic. Covers all disciplines: AI/ML, medicine, physics, economics, law, biology, and more. Supports relevance, citation-count, and recency sorting; open-access filtering; and year-range constraints. Use for literature review, prior-art search, citation building, or finding the seminal papers in any field.

AI-powered URL content analysis. Fetches a URL, extracts the readable article text, and returns structured intelligence: 2–3 sentence summary, key points, named entities with types, sentiment score, topic tags, content type classification, and credibility signals (has author/date/sources). Use for content intelligence pipelines, research synthesis, or automated brief generation. $0.012/call.

Extracts structured content from any public URL: page title, meta description, H1-H3 headings, all links (with text and internal/external flag), and a 500-character text preview. Useful for research agents following link chains from on-chain data, auditing page structure, or seeding downstream text-generation calls.

Extracts all hyperlinks from any public webpage. Returns href URLs normalized to absolute URLs with visible link text. Filters out javascript:, mailto:, data: schemes. Optionally restrict to same-domain links, deduplicate, or include #anchor links. Useful for crawlers, sitemap builders, link graph analysis, and content audits.

Extracts all hyperlinks from a webpage. Fetches the target URL, resolves relative links to absolute URLs, and classifies each as internal (same domain) or external. Filter by all/external/internal, cap results with limit. Returns page title, total link count before filtering, and a structured array of {href, text, is_external, domain}. Priced at $0.004 — 20% below orbisapi web-scrape-links ($0.005/call). Upstream: direct HTTP fetch, no API key.

Fetches any public URL and returns the full readable article text as clean Markdown, stripped of navigation, ads, and boilerplate. Returns title, published date (if available), and the complete body ready for LLM summarization, analysis, or RAG ingestion. A $0.004 alternative to exa.ai/contents ($0.007) and web-read ($0.016) for the same full-text extraction primitive.

Extracts and parses JSON from mixed-content text. Handles LLM output with JSON embedded in prose, code fences (```json), trailing commas, single-quoted strings, JS-style comments, and bare object keys (JSON5-style). Returns the parsed data, a cleaned JSON string, extraction method used, and any repair applied. Pure text processing — zero external API calls.

What does the open web say about this query? Ranked title/url/snippet results for agents, served through an automatic multi-engine failover chain so a single call still answers when any one backend is blocked, rate-limited or timing out from a datacenter IP.

Web search for agents: a query returns ranked title/url/snippet results (no API key, no signup)