logo
DE

Webpage to Markdown

Turn web pages into clean Markdown for AI and RAG: one row per URL with the main content, title, author, publish date and the page's links split into same-site and external; also works on pages that need JavaScript; pages that cannot be read say why and are not billed. Not affiliated with the websites read

Entwickler
Preise
0,0005 $ / Page read (main content returned)
Zuletzt aktualisiert
2026-11-13

Übersicht

Give it a list of web pages and get back one row per page: the main content as clean Markdown, plus the title, site name, author, publish date and the page's links, already split into same-site and external links.

The first step of almost every AI workflow that touches the web — summarizing an article, answering questions from a docs site, filling a RAG knowledge base — is turning a page into clean text. Raw pages are mostly navigation, ads, footers and cookie banners, which waste context and confuse the model. This app keeps only the main content and converts it to Markdown with headings, lists, tables, links and images intact, so people and models can both read it.

Every URL gets a row with a clear status: read, opened but empty, asked for a human check, refused, disallowed by the site, gone, or not a web page. You never send 10 URLs and get 7 rows back without knowing what happened to the other 3, and pages that could not be read are not charged.

Data notes

  • This app reads public pages at the addresses you give and converts their format; the content belongs to the original websites, so follow their copyright and terms of use. This is not an official product of any of these websites and is not affiliated with them. The fee covers fetching, rendering and conversion, not the content itself.
  • Only the pages you list are read; links on them are not followed, so the number of pages and the cost are known before a run. To go deeper, pick the next pages from "Same-site links" and read them in another run.
  • Pages whose content only appears after JavaScript runs are read too.
  • Sites are treated politely: their robots.txt is respected, and logins, paywalls, CAPTCHAs and interactive checks are never bypassed. Such pages come back with a status instead of content.
  • "Author" and "Published" are only filled when the page marks them explicitly; they are never guessed from the text. Dates are given as year-month-day.
  • Not covered: content behind a login, PDF and other files, and sites that block automated visitors outright (many social networks and large marketplaces). Main content longer than 100,000 characters is cut, and the status details say so.

What the results look like

Each row is one URL.

URLStatusTitlePublishedCharacters
https://www.anthropic.com/news/claude-opus-4-5okIntroducing Claude Opus 4.52025-11-2410342
https://www.allbirds.com/products/mens-tree-runnersokMen's Tree Runner - Jet Black (White Sole)4000
https://stackoverflow.com/questions/231767verification0

Each row also has the Markdown content, site name, author, language, description, same-site links, external links, images, status details and the time it was read.

Use cases

  • Feed an AI: hand "Content (Markdown)" to a model for summaries, Q&A or translation, without menus and ads in the way.
  • Build a knowledge base: store "Content (Markdown)" with "Title", "URL" and "Published" for each page, so answers can cite where they came from.
  • Read a docs site or blog section layer by layer: start with the entry page, choose the next URLs from "Same-site links" by link text or path (for example only addresses under /docs/), and read them in the next run. You decide how deep to go and how many pages each round reads.
  • Clean a URL list: filter on "Status" to find pages that are gone, refuse access or have no content, and fix or drop them; "External links" shows which outside sources each page cites.

Anwendungsbereich & Grenzen

Up to 100 URLs per run; only the given pages are read, links on them are not followed. robots.txt is respected; logins, CAPTCHAs and interactive checks are never bypassed. Main content is capped at 100,000 characters per page. Billed per page whose main content was read; pages that could not be read also get a row but are not billed. The fee covers fetching, rendering and conversion, not the content itself

  • Geeignet für
  • You have a list of page URLs and need their main content as clean Markdown for an LLM to read, summarize or answer from, or to load into a RAG knowledge base
  • You need the title, author and publish date of articles or blog posts together with their text
  • You want to read a documentation site or a blog section layer by layer: read the entry page, pick the next URLs from `internal_links` (skipping ones already read), and call again with them; you decide what each round reads, so its cost is known in advance
  • Nicht verwenden für
  • You have no URLs and want to find pages by keyword: use a web search app first, then read the URLs it returns here
  • The content is behind a login or a paywall, or the site shows an interactive CAPTCHA: these come back as blocked or verification
  • You need PDF, Word or other files: only web pages are read (they come back as not_html)
  • Social networks and marketplaces that refuse automated access (LinkedIn, Facebook, Reddit, Amazon and similar) usually come back as blocked

Fehlerbehandlung

Vom Entwickler deklariertes Fehler- und Wiederholungsverhalten. Wir empfehlen, es bei der Integration in Ihren System-Prompt aufzunehmen.

  1. 1Every URL gets exactly one row. `status` says what happened: ok, empty_content, verification, blocked, robots_disallowed, not_found, not_html, timeout, dns_error, error; only ok is billed (`billed_pages` = 1)
  2. 2verification, blocked, timeout and error can be retried later
  3. 3robots_disallowed means the site's robots.txt asks crawlers not to fetch that page; it is not read
  4. 4empty_content means the page opened but has no readable main content (a list, login or app page); its title and links are still returned

Eingabe

Parameter, die zum Aufruf dieser App erforderlich sind, generiert aus dem input.schema in der manifest.json.

  • URLsurlsErforderlicharray<string>
    Web pages to read, 1–100. Each URL gets one row. Only these pages are read; links on them are not followed. A URL given twice is read once.
    Anzahlbis zu 100 EinträgeBeispiel["https://www.anthropic.com/news/claude-opus-4-5"]

Ausgabe

Feldstruktur eines einzelnen Datensatzes, generiert aus dem output.schema in der manifest.json.

  • URLinput_urlstring
    The URL as given in the input, to match rows to inputs
    Beispielhttps://www.anthropic.com/news/claude-opus-4-5
  • Final URLfinal_urlstring
    The address actually read; differs from the input when the page redirected
    Beispielhttps://www.anthropic.com/news/claude-opus-4-5
  • Statusstatusstring
    ok = main content read; empty_content = the page opened but has no readable main content; verification = the site asks for a human check; blocked = the site refused access; robots_disallowed = robots.txt does not allow this page; not_found = the page does not exist; not_html = not a web page (e.g. PDF); timeout = no response in time; dns_error = the domain does not exist; error = could not be opened. Only ok is billed
    Beispielok
  • Status detailsstatus_textstring
    Plain-English explanation of the status, with the reason when the page was not read; also notes when the content was cut at 100,000 characters
    BeispielMain content read

Beispieldaten

Aus output.schema erzeugt. Felder ohne Beispielwert verwenden strukturelle Platzhalter.

output.example
{
  "input_url": "https://www.anthropic.com/news/claude-opus-4-5",
  "final_url": "https://www.anthropic.com/news/claude-opus-4-5",
  "status": "ok",
  "status_text": "Main content read",
  "billed_pages": 1,
  "title": "Introducing Claude Opus 4.5",
  "site_name": "Anthropic",
  "author": "Anthropic",
  "published_at": "2025-11-24",
  "language": "en",
  "description": "Introducing Claude Opus 4.5",
  "markdown": "Our newest model, Claude Opus 4.5, is available today...",
  "char_count": 10785,
  "internal_links": "<internal_links>",
  "external_links": "<external_links>",
  "images": "<images>",
  "checked_at": "2026-11-12T10:00:00+00:00"
}

Integration

Diese App kann über MCP, API, SDK oder Dateiexport integriert werden — alle Kanäle teilen sich dieselben Fähigkeiten und Preise. Jede Anfrage authentifiziert sich über den Authorization: Bearer-Header mit einem API Key (langlebig, in der Data Hub Konsole erstellt); MCP-Clients können sich zusätzlich per OAuth ohne Key anmelden. Weitere Optionen wie CLI und Skill sind in Vorbereitung. Diese App läuft asynchron und kann zwischendurch abgebrochen werden: Nach dem Abbruch stoppt der Lauf. Bereits gelieferte Datensätze bleiben erhalten und werden abgerechnet.

Über das MCP (Model Context Protocol) können Sie diese App direkt aus KI-Clients wie Claude und Cursor aufrufen. Wählen Sie Ihren Client und den Authentifizierungsmodus und kopieren Sie dann die Konfiguration unten.

Client-Konfiguration

Ersetzen Sie den Wert nach Bearer durch Ihren langlebigen API Key. Funktioniert in jedem Client, in CI und in Headless-Umgebungen.

mcpServers config
{
  "mcpServers": {
    "octoparse_datahub": {
      "type": "http",
      "url": "https://mcp-v2.octoparse.com?pin=MagicData/webpage-to-markdown",
      "headers": { "Authorization": "Bearer <YOUR_API_KEY>" }
    }
  }
}

Lassen Sie die KI die Einrichtung übernehmen

Sie möchten Konfigurationen nicht von Hand bearbeiten? Kopieren Sie den Installations-Prompt und fügen Sie ihn in einen beliebigen KI-Client ein — er schließt die Einrichtung auf seine eigene Weise ab. (Standardmäßig führt er Sie durch die Anmeldung und fordert Sie nie auf, Ihren API Key im Chat zu senden.)

detail.access.mcp.composeHint

Preise

Page read (main content returned)

Gemessen durch Aufsummieren des Ausgabefelds billed_pages. Fehlgeschlagene Läufe werden nicht berechnet.

0,0005 $/ Page read (main content returned)

Mehrere Abrechnungsereignisse werden unabhängig voneinander kumuliert. Details siehe jeweilige Position. Fehlgeschlagene Läufe werden nicht berechnet.

Credits ansehen

Jetzt ausprobieren

Füllen Sie die Parameter aus und starten Sie — die Ergebnisse stammen aus einem echten Aufruf.

Beispiel
Parameter werden vor dem Absenden gegen input.schema validiert
0,0005 $ / Page read (main content returned)