There are various factors you might need to locate all of the URLs on a website, but your exact goal will determine what you’re searching for. For instance, you might want to:

Establish just about every indexed URL to research concerns like cannibalization or index bloat
Gather present and historic URLs Google has seen, specifically for internet site migrations
Locate all 404 URLs to Recuperate from post-migration faults
In Just about every situation, just one tool received’t Present you with every little thing you'll need. Unfortunately, Google Look for Console isn’t exhaustive, plus a “web site:example.com” look for is restricted and difficult to extract data from.

Within this post, I’ll wander you thru some equipment to construct your URL record and in advance of deduplicating the information using a spreadsheet or Jupyter Notebook, depending on your internet site’s size.

Aged sitemaps and crawl exports
In case you’re trying to find URLs that disappeared in the Are living web-site not too long ago, there’s a chance another person with your team could possibly have saved a sitemap file or even a crawl export before the improvements had been built. In case you haven’t currently, look for these information; they might typically offer what you require. But, should you’re reading this, you most likely didn't get so Fortunate.

Archive.org
Archive.org
Archive.org is a useful Instrument for Search engine optimization responsibilities, funded by donations. In the event you try to find a domain and choose the “URLs” choice, you may access as many as 10,000 shown URLs.

However, There are many limitations:

URL limit: It is possible to only retrieve up to web designer kuala lumpur 10,000 URLs, which happens to be insufficient for more substantial sites.
Top quality: Lots of URLs could be malformed or reference useful resource data files (e.g., photos or scripts).
No export solution: There isn’t a built-in strategy to export the record.
To bypass the lack of an export button, make use of a browser scraping plugin like Dataminer.io. Having said that, these limitations suggest Archive.org may well not deliver a complete Resolution for larger sized sites. Also, Archive.org doesn’t show irrespective of whether Google indexed a URL—but if Archive.org identified it, there’s a superb prospect Google did, too.

Moz Pro
Even though you may perhaps normally utilize a connection index to find external web sites linking for you, these instruments also explore URLs on your web site in the process.


How you can use it:
Export your inbound backlinks in Moz Pro to acquire a swift and straightforward listing of target URLs from your website. Should you’re coping with a large Internet site, consider using the Moz API to export facts past what’s workable in Excel or Google Sheets.

It’s crucial that you note that Moz Pro doesn’t confirm if URLs are indexed or learned by Google. Nonetheless, since most internet sites apply the exact same robots.txt guidelines to Moz’s bots because they do to Google’s, this technique typically is effective effectively to be a proxy for Googlebot’s discoverability.

Google Search Console
Google Research Console features numerous precious resources for developing your list of URLs.

One-way links reviews:


Just like Moz Professional, the One-way links segment supplies exportable lists of concentrate on URLs. Sadly, these exports are capped at 1,000 URLs Every. You'll be able to use filters for distinct webpages, but due to the fact filters don’t implement on the export, you would possibly should trust in browser scraping instruments—limited to five hundred filtered URLs at a time. Not best.

Functionality → Search Results:


This export gives you an index of webpages getting look for impressions. When the export is restricted, You need to use Google Search Console API for larger sized datasets. You will also find absolutely free Google Sheets plugins that simplify pulling far more in depth facts.

Indexing → Web pages report:


This area gives exports filtered by problem kind, although they are also limited in scope.

Google Analytics
Google Analytics
The Engagement → Internet pages and Screens default report in GA4 is a superb source for amassing URLs, that has a generous limit of 100,000 URLs.


Better yet, it is possible to use filters to create distinctive URL lists, proficiently surpassing the 100k limit. For example, if you would like export only site URLs, stick to these steps:

Action 1: Increase a segment into the report

Action two: Click on “Make a new section.”


Move 3: Determine the segment that has a narrower URL pattern, which include URLs that contains /web site/


Notice: URLs located in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide useful insights.

Server log files
Server or CDN log documents are perhaps the last word Software at your disposal. These logs capture an exhaustive listing of each URL path queried by end users, Googlebot, or other bots during the recorded time period.

Criteria:

Info dimensions: Log files could be large, a lot of internet sites only retain the last two weeks of information.
Complexity: Examining log data files may be demanding, but many tools can be found to simplify the method.
Blend, and superior luck
After you’ve gathered URLs from all of these resources, it’s time to mix them. If your internet site is sufficiently small, use Excel or, for larger datasets, tools like Google Sheets or Jupyter Notebook. Make certain all URLs are persistently formatted, then deduplicate the listing.

And voilà—you now have an extensive listing of latest, outdated, and archived URLs. Superior luck!