There are various factors you might need to locate all of the URLs on a website, but your exact goal will determine what you’re searching for. For instance, you might want to: Establish just about every indexed URL to research concerns like cannibalization or index bloat