Crawl Errors

Crawl errors are unsuccessful attempts by search engine spiders to crawl a website. Google's Search Console divides crawl errors into two categories: Site Errors and URL Errors. Site errors affect an entire website. They include problems with DNS, server connectivity, and the robots.txt file.

URL errors affect specific URLs. They include Soft 404 errors, 404 errors, Access Denied, and Not Followed. Significant amounts of crawl errors point to poor website health, and that can negatively affect the user experience as well as search engine rank, crawl frequency, and crawl depth.
Learn more about crawl errors
Common URL errors

How do I fix soft 404 errors?

 

A soft 404 error occurs when the page a user request is invalid. However, instead of the server returning a 404 or 410 error message, it returns a 200 OK status code which means the server ignores the error and lets the search engines know the page is valid.

This means the search engines will continue to crawl the site and display it in the search results.

The first step in fixing a soft 404 error is finding it. This can be done with the Google Search Console Coverage report. The soft 404 error message will appear under the 'Type' heading.

Before explaining how to fix a soft 404, it's important to first verify if it is not a Search Console error. To do that, go to the Coverage report and find the error message. Click on the error message and you will be shown a list of affected pages. When you hover your mouse over one of the URLs you will see an "Open in a new tab” option. If you click it and you are directed to a live page you know that it is a false error. To ensure that it shows up in Google, click the Validate Fix button. This will force Google to recrawl the URL.

If the soft 404 error is correct, there are a number of ways to fix it.

• Configure the server to show a proper error code (404 or 410) If the page is really unavailable or invalid, it is important to configure your website to return the correct 404 or 410 error code. It should then be resubmitted for indexing using the Request Indexing tool. The easiest way to configure the page to show a 404 error is to delete the page. When Google tries to crawl the page it will show up as a 404.
• Improve the page If, on the other hand, the page is valid but Google considers it a soft 404, the best practice is to improve the content and submit it for reindexing by clicking the Validate Fix button.
• 301 redirect the page Another option is to redirect the affected page using a 301 redirect.
• Noindex A soft 404 error can be solved by simply no-indexing the page. This means it will no longer be in Google's index.

search console