404 Watch API

404 Watch API

Link checker API with advanced features and reporting capabilities.

Response example. If you want to see more check the documentation.

Free Plan $0.00 Monthly Subscribe
100 Requests / Monthly
Free for Lifetime
No Credit Card Required
MOST POPULAR
Starter Plan $9.99 Monthly Subscribe
3,000 Requests / Monthly
Standard Support
Pro Plan $39.99 Monthly Subscribe
15,000 Requests / Monthly
Standard Support
Custom Plan Volume Monthly Contact Us
Any requests volume you need

Broken links are not only bad from the user's view, but also damages your site's search engine visibility, as Google discourages broken links and downgrades your SEO reputation accordingly. You should avoid links to broken content and also avoid having pages on your site that are not working.

404 Watch API is one of the most powerful link checkers in the market with advanced features as below.

  • Optionally can respect nofollow attributes on hrefs
  • Can check external links
  • Optionally can discard query parameters
  • Optionally can discard hash parameters
  • Can check images/js and css files for broken links
  • Can whitelist and exclude domains from checking
  • Can trigger a callback URL, when the link checking is done, or you may optionally poll the results for results.

Whitelisting domains

You may have multiple domains that is served from a single site, and need to treat them as internal resources. So you may add the multiple domains to thw whitelisted domains while creating the link checker job and you're done.

Excluding domains and URLs from checking

You may wish to exclude some specific domains from link checking for any reason. You just need to add the domain names to the excluded_domains_list or excluded_urls_list list variables while creating the link checker job

Callback when finished

Link checking is a time consuming job. As you may have multiple hundreds of URLs (and asssets) on your site. So we have chosen an asyncronous approach. You create a link checking job using the POST /job endpoint and get an ID in response. You may wish to poll the GET /job/{id} endpoint for watching out the ongoing link checking process or get the results for finished ones.

Optinally you may also provide a callback URL when creating the link checker job via POST /job endpoint.Doing so, you don't need to poll the GET /job endpoint for results, as the API will call provided callback URL (via HTTP POST) automatically when the process has finished.

Optinally you may also provide a callback_security variable when creating the link checker job. This variable will be placed in the HTTP header using the X-Callback-Secret header. You may check this header for authentication purposes.

Sample Request for creating a new link checker job

Below is a sample request that is done to the POST /job endpoint. It contains many of the config parameters for optimizing the link checking process.

curl --location --request POST 'https://api.apilayer.com/404_watch/job' \
--header 'Content-Type: application/json' \
--header 'apikey: YOUR API KEY' \
--data-raw '{
  "url": "https://p1.rs",
  "levels": 2,
  "fetch_external": false,
  "check_images": false,
  "check_css": true,
  "callback": "https://mydomain.com/callback",
  "callback_secret": "supersecret_key",
  "check_js": true,
  "whitelisted_domains_list": [
      "apilayer.com"
  ]
}'

When called you'll get a response such as below:

{
    "id": "c3f7b23e-a239-4af4-b9ec-698a3a6d0a21"
}

You can use this id for querying the results via GET /job/{id} endpoint.

curl --location --request GET 'https://api.apilayer.com/404_watch/job/c3f7b23e-a239-4af4-b9ec-698a3a6d0a21' \
--header 'apikey: YOUR API KEY'

The response contains comprehensive details about the ongoing process and the results. See below:

{
    "id": "c3f7b23e-a239-4af4-b9ec-698a3a6d0a21",
    "created_at": 1609681539,
    "status": "finished",
    "url": "https://apilayer.com",
    "progress": {
        "discovered": 117,
        "checked": 117,
        "percentage": 100.0
    },
    "status_codes": {
        "503": 4,
        "200": 110
    },
    "content_types": {
        "image/svg+xml": 10,
        "image/png": 25,
        "text/css": 6,
        "text/html": 60,
        "image/jpeg": 5,
        "application/javascript": 8,
        "application/x-javascript": 1
    },
    "options": {
        "callback_secret": null,
        "check_css": true,
        "max_levels": 3,
        "check_js": true,
        "max_links": 1000,
        "excluded_domains_list": [],
        "fetch_nofollow": false,
        "excluded_urls_list": [],
        "fetch_external": true,
        "whitelisted_domains_list": "assets.apilayer.com",
        "omit_query_params": false,
        "callback": null,
        "omit_hash_params": true,
        "check_images": true
    }
}

Date variables above (created_at) are timestamps.

Getting the details for each link that is checked

If you wish to get all the links that is discovered and been checked using the GET /job/{id}/links endpoint. See the following example.

curl --location --request GET 'https://api.apilayer.com/404_watch/job/c3f7b23e-a239-4af4-b9ec-698a3a6d0a21/links' \
--header 'apikey: YOUR KEY'

The response contains all the links as well as the details for content types and http status codes. You may filter and use it the way you desire.

{
    "job_id": "d0de484e-c18f-4ee8-b84e-4ba63907e283",
    "status": "finished",
    "created_at": 1609681539,
    "links": [
        {
            "url": "https://apilayer.com",
            "content_type": "text/html",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609681563
        },
        {
            "url": "https://assets.apilayer.com/apis/image_similarity.png",
            "content_type": "image/png",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609681578
        },
        {
            "url": "https://apilayer.com/marketplace/description/textgears-api",
            "content_type": "text/html",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609681592
        },
        {
            "url": "https://apilayer.com/marketplace/category/text-processing-apis",
            "content_type": "text/html",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746185
        },
        {
            "url": "https://js.hs-scripts.com/7564526.js",
            "content_type": "application/javascript",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746223
        },
        {
            "url": "https://apilayer.com/marketplace/tag/spelling",
            "content_type": "text/html",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746233
        },
        {
            "url": "https://apilayer.com/marketplace/tag/text-tools",
            "content_type": "text/html",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746256
        },
        {
            "url": "https://textgears.com/assets/img/logos/apple/120.png",
            "content_type": "image/png",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746270
        },
        {
            "url": "https://apilayer.com/assets/css/documentation.css?6",
            "content_type": "text/css",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746374
        },
        {
            "url": "https://apilayer.com/assets/js/marketplace/marketplace.js?52",
            "content_type": "application/javascript",
            "is_timeout": false,
            "http_status": 200,
            "fetched_at": 1609746577
        }
    ],
    "query": {
        "limit": 10,
        "offset": 0,
        "page": 0,
        "total_count": 117
    }
}