On-Page API Fresh

The On-Page API is a customizable website crawling engine that extracts technical SEO data and evaluates page performance against SEO benchmarks. It can crawl entire websites or individual pages, check for hundreds of on-page issues, measure Core Web Vitals, and return raw HTML.

What It Covers

• Full-site technical SEO audits
• Page-level health checks (titles, descriptions, canonical tags, indexability)
• Internal and external link analysis
• Redirect chain detection
• Duplicate title and content detection
• Resource inventory (images, scripts, stylesheets)
• Core Web Vitals measurement (requires enable_browser_rendering)
• Keyword density analysis
• Raw HTML capture

How Task-Based Crawling Works

On-Page API uses the Standard task model:

1. POST a crawl task to /v3/on_page/task_post with your domain or URL and desired parameters
2. The crawler fetches pages from your target asynchronously
3. Poll the Summary endpoint to check crawl_progress — you can start pulling results before crawling is fully complete
4. GET results from any sub-endpoint using the task ID

You can also supply pingback_url to receive a notification when the task completes.

Task POST Parameters

Sent to POST https://api.dataforseo.com/v3/on_page/task_post

Parameter	Type	Description
target	string	Domain or URL to crawl. Required.
max_crawl_pages	integer	Maximum number of pages to crawl. Required.
load_resources	boolean	Load images, scripts, stylesheets. Extra cost.
enable_javascript	boolean	Execute JavaScript on crawled pages. Extra cost.
enable_browser_rendering	boolean	Measure Core Web Vitals. Extra cost.
calculate_keyword_density	boolean	Calculate keyword density per page. Extra cost.
store_raw_html	boolean	Store raw HTML for retrieval. Extra cost.
custom_js	string	Custom JavaScript code to execute during crawl.
checks_threshold	object	Override default threshold values for on-page checks.
pingback_url	string	Webhook URL to notify on task completion.

All Endpoints (24 total)

Task-Based Crawl Endpoints (20 endpoints)

Endpoint	Method	Description
/v3/on_page/task_post	POST	Start a site crawl — the primary entry point
/v3/on_page/tasks_ready	GET	List of completed crawl tasks
/v3/on_page/task_get/{id}	GET	Retrieve crawl task metadata
/v3/on_page/summary/{task_id}	GET	Aggregate issue counts and crawl status
/v3/on_page/pages/{task_id}	GET	Page-level audit results for all crawled pages
/v3/on_page/pages_by_resource/{task_id}	POST	Pages that reference a specific resource
/v3/on_page/resources/{task_id}	GET	All resources found: images, scripts, stylesheets
/v3/on_page/links/{task_id}	GET	Internal and external link inventory
/v3/on_page/duplicate_content/{task_id}	GET	Pages with substantially similar content
/v3/on_page/duplicate_tags/{task_id}	GET	Pages with duplicate title or meta description
/v3/on_page/errors/{task_id}	GET	Crawl errors: 4xx, 5xx, broken links
/v3/on_page/redirect_chains/{task_id}	GET	Multi-hop redirect paths
/v3/on_page/non_indexable/{task_id}	GET	Pages blocked from indexing (noindex, disallow, etc.)
/v3/on_page/waterfall/{task_id}	GET	Page load waterfall — resource timing per page
/v3/on_page/keyword_density/{task_id}	GET	Term frequency per page (requires calculate_keyword_density)
/v3/on_page/raw_html/{task_id}	GET	Raw HTML of a crawled page (requires store_raw_html)
/v3/on_page/content_parsing/{task_id}	GET	Parsed structured content from crawled pages
/v3/on_page/force_stop/{task_id}	POST	Stop a running crawl task
/v3/on_page/task_delete/{task_id}	DELETE	Delete a completed task and free storage
/v3/on_page/microformats/{task_id}	GET	Microformat data found on pages

Live Endpoints (No Task Required — 4 endpoints)

Endpoint	Method	Description	Cost Est.
/v3/on_page/instant_pages	POST	Immediate on-page scan for a single URL	~$0.004
/v3/on_page/page_screenshot/live	POST	Screenshot of a single page	~$0.01
/v3/on_page/lighthouse/live/json	POST	Full Lighthouse audit — CWV, performance, accessibility, SEO	~$0.02
/v3/on_page/lighthouse/live/html	POST	Lighthouse audit as HTML report	~$0.02

Key Metrics

Page-Level Checks

Each page in the Pages endpoint result includes a checks object with pass/fail indicators for:

• Title tag presence, length, and uniqueness
• Meta description presence and length
• Canonical tag correctness
• H1 tag presence
• Broken internal and external links
• Image alt attributes
• HTTPS usage
• Mobile friendliness
• Page size and load time
• Noindex / nofollow directives

Crawl Progress

The Summary endpoint includes:

Field	Description
crawl_progress	"in_progress" or "finished"
crawl_status.pages_crawled	Number of pages crawled so far
crawl_status.pages_in_queue	Pages queued but not yet crawled
domain_info.checks	Site-wide issue counts

Rate Limits

• Task Post: up to 2000 API calls per minute, max 100 tasks per POST
• All other endpoints: send one task per POST call to avoid system overload
• Max 30 simultaneous live requests

Code Examples

Set a Crawl Task

login="your_login"
password="your_password"
cred="$(printf ${login}:${password} | base64)"

curl --location --request POST "https://api.dataforseo.com/v3/on_page/task_post" \
  --header "Authorization: Basic ${cred}" \
  --header "Content-Type: application/json" \
  --data-raw '[
    {
      "target": "example.com",
      "max_crawl_pages": 100,
      "enable_javascript": false,
      "load_resources": false
    }
  ]'

Python Workflow

from client import RestClient
import time

client = RestClient("your_login", "your_password")

# Step 1: Set the crawl task
post_data = dict()
post_data[0] = dict(
    target="example.com",
    max_crawl_pages=100,
    load_resources=False,
    enable_javascript=False
)

response = client.post("/v3/on_page/task_post", post_data)

if response["status_code"] == 20000:
    task_id = response["tasks"][0]["id"]
    print(f"Task created: {task_id}")
else:
    print(f"Error: {response['status_message']}")
    exit()

# Step 2: Poll for completion
while True:
    summary = client.get(f"/v3/on_page/summary/{task_id}")
    progress = summary["tasks"][0]["result"][0]["crawl_progress"]
    print(f"Progress: {progress}")
    if progress == "finished":
        break
    time.sleep(10)

# Step 3: Retrieve page results
pages = client.get(f"/v3/on_page/pages/{task_id}")
for page in pages["tasks"][0]["result"]:
    print(page["url"], page["checks"]["title"])

Pricing

Additional cost applies for:

• load_resources — charges per resource loaded
• enable_javascript — charges per JS execution
• enable_browser_rendering — charges per CWV measurement
• calculate_keyword_density — charges per page
• store_raw_html — charges for HTML storage

Standard page crawls are charged per page crawled.