An SEO spider is a web crawler. When you think about the World Wide Web, the Internet itself is often called a web, and a spider is something that traverses a web. Essentially, Google functions similarly to a web crawler that navigates through different links, indexing content, and delivering search results to users.
A web crawler emulates Google. Google itself is a web crawler, and tools like Screaming Frog SEO Spider also crawl the web in a manner similar to Google, providing us with information to assist Googlebot—the small spider that navigates the web—with all the necessary data. This is powerful for identifying technical SEO issues and ensuring your website has a solid foundation for all your SEO initiatives. Additionally, it offers many tips and tricks to maximise your organic traffic and boost your SEO rankings.
You can crawl up to 500 URLs with the free version and £199 Per Year for a licensed version.
Working Of Screaming Frog Spider SEO Audit Tool
Before starting a crawl, go to Configuration → Crawl Config → Crawl, where you’ll find many options. If you want to crawl linked sitemaps, select the ‘Crawl Linked XML Sitemaps‘ option. You can even specify which sitemaps you want to crawl.
Next, go to Extraction. It’s usually best to leave the settings as they are, but if you want structured data, you can add it according to your needs.
For Rendering, by default, it is ‘Text Only.’ If you have a JavaScript website, you can change this to ‘JavaScript.’
In User Agent, you can change from Screaming Frog SEO Spider to Googlebot, so it appears as if Googlebot is visiting the URL, allowing you to see exactly what Googlebot sees. If you are diagnosing Bing traffic, perhaps due to a traffic drop on Bing, you’d want to change it to Bingbot. Similarly, for Yahoo or DuckDuckGo, you can adjust it to your requirements.
In API Access, you can connect your Google Search Console, Google Analytics, PageSpeed Insights, Ahrefs, Majestic, and Moz with Screaming Frog. This allows the tool to crawl and pull all the data for every one of your URLs.
Segments allow us to separate an e-commerce website by product categories, product pages, and blog posts, which is very useful. You can add any segments you need to categorise the URLs.
When you first launch it, you’ll see a large search box at the top labelled ‘Enter URL to Spider‘ where you input your website’s URL and then click start button. Thats going to start the crawl and going through this website and just crawling all the different URLs.
Once the crawl is complete, you can see all the pages on your website. You can also view all the SEO data, such as page titles, meta descriptions, H1, and H2 tags for each page.
Go to Overview → Response Codes → Internal & External → Internal → Internal Client Error (4xx). If you click on that, you can see if your website has any broken links with various status codes, like 404 or 430. You’ll need to investigate what those status codes mean and fix the issues, as you can’t have Google crawling links on your website that lead to 404 pages.
If we go to “Internal Server Error (5xx),” we can see your 5xx pages, which you’ll need to fix.
In the “Internal Redirection (3xx)” section, we can see your 3xx pages, which are ones that are redirected correctly.
Scrolling down the “Overview” tab, you’ll see the URL, Page titles, Meta descriptions, and H1 tags. These are all important factors to consider for SEO.
One of the most important elements is the Page Title. If you go to the “Page Titles” tab, it will show you all of your page titles by URL. If you go into the “Issues” tabs, it will instantly display the URLs with page titles over 60 characters, below 30 characters, missing page titles, duplicate page titles, and pages where the Title is the same as the H1 tag. It even prioritises the issues based on their importance, so you can easily identify what’s critical and what’s not.
You can Export any of these tables directly to Google Sheets, CSV, or Excel. This allows you to thoroughly review the data and optimise the elements that need improvement, so you can fix the identified issues.
