Bulk Google index check in Python: a safe script structure

Checking a handful of URLs by hand is fine. Checking a few thousand backlinks every month needs a script. This page shows a small Python structure for a bulk Google index check that stays the same whichever data source you plug in, so moving away from the Custom Search JSON API means changing one function, not rewriting the tool.

The five steps

1. Read the URL list from a file, one URL per line.
2. Clean it: trim spaces, drop empty lines and non-URLs, remove duplicates.
3. Check in batches, with a pause between batches and retries with growing waits on errors.
4. Save the result with a date, so you can compare runs over time.
5. Recheck only the URLs that failed or came back not indexed, after a sensible interval.

The script

import csv, time, datetime

def read_urls(path):
    with open(path, encoding="utf-8") as f:
        urls = [line.strip() for line in f]
    seen, clean = set(), []
    for u in urls:
        if u and u.startswith("http") and u not in seen:
            seen.add(u)
            clean.append(u)
    return clean

def check_batch(urls):
    """Return {url: status} for one batch.
    Plug in your data source here: the URL Inspection API for your own
    property, or the API of the index checker you use for other sites."""
    raise NotImplementedError

def check_all(urls, batch_size=50, pause=1.0, retries=3):
    results = {}
    for i in range(0, len(urls), batch_size):
        batch = urls[i:i + batch_size]
        for attempt in range(1, retries + 1):
            try:
                results.update(check_batch(batch))
                break
            except NotImplementedError:
                raise
            except Exception as err:
                wait = pause * 2 ** attempt
                print("batch", i, "attempt", attempt, "failed:", err, "- retry in", wait, "s")
                time.sleep(wait)
        else:
            for u in batch:
                results[u] = "error"
        time.sleep(pause)
    return results

def save(results, path):
    today = datetime.date.today().isoformat()
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["url", "status", "checked"])
        for url, status in results.items():
            w.writerow([url, status, today])

if __name__ == "__main__":
    urls = read_urls("urls.txt")
    results = check_all(urls)
    save(results, "index_status.csv")
    failed = [u for u, s in results.items() if s == "error"]
    print(len(urls), "checked,", len(failed), "to recheck later")

The structure is deliberately plain: standard library only, one CSV in, one CSV out. A failed batch is retried with exponential backoff and, if it still fails, its URLs are marked error so the next run picks them up.

Option A: your own property via the URL Inspection API

If the URLs belong to a Search Console property you control, check_batch can call the URL Inspection API for each URL. The method is urlInspection.index.inspect, and the response includes an index status section with a verdict and a coverage state:

from googleapiclient.discovery import build

def inspect(service, site_url, url):
    body = {"inspectionUrl": url, "siteUrl": site_url}
    res = service.urlInspection().index().inspect(body=body).execute()
    status = res["inspectionResult"]["indexStatusResult"]
    return status.get("verdict"), status.get("coverageState")

# service = build("searchconsole", "v1", credentials=creds)

Keep the pace inside Google's quotas — 600 inspections per minute and 2,000 per day for each property — and remember that every URL must be under the property in siteUrl. The URL Inspection API limits page covers how to plan that budget.

Option B: backlinks and other sites via an index checker API

For URLs on domains you do not own, connect check_batch to the API of an index checking service. Use the endpoints, authentication and batch sizes from that provider's documentation rather than guessing them, and keep the API key in an environment variable, never in the script or the repository.

A shorter, ready-to-copy version of this pattern is published as a gist: gist.github.com/uskotoivo81-netizen/b631373863d5381c843316ea7730ab41. Before you switch a production job from the Custom Search JSON API, run through this checklist to make sure nothing else in the pipeline still calls the old API.

Running it safely

• Start with a sample of 20–50 URLs and compare the output with a manual check.
• Log every batch, including errors, so a partial run can be resumed.
• Store the status reason, not just yes or no, whenever the data source provides one.
• Recheck on a schedule rather than in a loop. New links often need days or weeks before their status settles.
• Only check URLs you have a reason to check — your pages and your backlinks — and respect each service's terms.

The overview of ways to check index status explains which data source fits which kind of URL list.

Guides on this site

• Check Google index status without the Custom Search API — the overview and how to choose a method.
• URL Inspection API limits — 2,000 inspections per property per day and the scope rule.
• Vertex AI Search for index checks — what Google's suggested alternative is built for.
• The site: operator as an index check — what it shows and what it misses.
• Bulk index check in Python — a safe script structure with batches and retries.

FAQ

Can I just scrape Google results instead of using an API?

Automated scraping of Google results is not a supported way to get this data, and it breaks often. Use an API you are allowed to use.

How often should I rerun a bulk index check?

For new backlinks, a first check after one to two weeks and a recheck after about a month is a practical rhythm. Daily checks rarely add information.

Sources

• https://developers.google.com/webmaster-tools/v1/urlInspection.index/inspect
• https://developers.google.com/webmaster-tools/limits
• https://developers.google.com/custom-search/v1/overview