When you’re working to improve your site’s online visibility, executing on keyword research, improving performance in search results, you have to take stock of all the technical seo and onsite SEO elements of your website to get a complete picture of the project. This means that you need to see a full list of the pages of your site, and understand all of the various elements that can impact your SEO outcomes. That’s why you’ll need a site crawling tool.
What is An SEO Crawler
An SEO Crawler is a suite of software run locally or online that systematically accesses each page and feature of your website, consolidating the data into a data sheet for extensive SEO analysis.
These crawling seo tools will typically mimic Google, Yahoo & other search engine “spider” bot behavior, to help you uncover technical issue setup issues that could be preventing your site from being properly “crawled”, understood, indexed and ranked.
List Of Currently Available SEO Crawlers Used By SEOs
| Name: | Beam Us Up | |
| User Interface: | Desktop Application | |
| Platforms Supported: | Windows, Macintosh – Uses Java – Be sure to update to latest version. | |
| Cost: |
Free | |
| Crawl Scope/Size: |
As many pages as you want (It uses local memory so it just takes time to crawl big sites.) Of course, you may regret trying to open an excel file with 500,000 entries so crawl responsibly | |
| Resource Article: |
A Simple DIY Audit |
| Name: | Screaming Frog | |
| User Interface: | Desktop Application | |
| Platforms Supported: | Windows, Macintosh, Ubunto | |
| Cost: |
£149.00 licence per year | |
| Crawl Scope/Size: |
You can crawl up to 500 urls w/o a license. With a license there is no limit outside of the processing of the file size. Be wary about 500k urls. | |
| Resource Article: |
Screaming Frog Guide |
Raven Site Auditor and Site Performance :
• Company/Parent Toolset: Raven Tools
• User Interface: Web Application, Browser Add-on
• Cost: Paid $99 and $249 a month (Part of overall tools set), Free 14 day trial
SEOMoz Crawl diagnostic:
• Company/Parent Toolset:
• User Interface: Web Application
• Cost: $99 a month (part of full SEOmoz subscription), Free 30 day trial
Craawler
Nifty tool does a full site crawl picking out site errors, spelling errors and giving tweets, likes and +1’s page by page.
• Company/Parent Toolset:
• User Interface: Website
• Cost: Free
• Crawl Depth: Site Wide • Unique Function: Checks spelling and provides tweets and +1s on a page by page view
A1 WebSite Analyzer –
• Company/Parent Toolset: Microsys
• User Interface: Desktop Download
• Platforms Supported: PC, Macintosh
• Cost: $69 30 day trial
• User Interface: Website
• Cost: Free
• Crawl Depth: Sitewide
• Unique Function: Provides graph breakdowns of page objects and header responses.
• Response from tool creator:
A1 Website Analyzer is a Windows and Mac website and link analysis tool.
After a site crawl, it can filter and show you all pages (including broken links
to internal and external URLs), file sizes, response codes, duplicate content,
titles, descriptions, H1, H2 canonical, internal anchor text to pages, link
line numbers, link follow/nofollow states, navigation click length, all top
keywords and a ton of other data. It will also calculate internal link juice
“score” of all pages. (The tool has been in constant development since 2006.)If configured so, it can also validate HTML/CSS, spell check and
perform custom search for text/patterns throughout entire website
and integrate with various online tools.There is no fixed limit of URLs (or links for that matter), and the
practical limit is well above 100,000 unique URLs – infact some have
reported success crawling many-million-page websites, but it depends
on factors such as the computer, website and program configuration.For those with complex websites, the tool features an extensive
set of website crawl “limit to” and “exclude” filtering options.
It is also possible to add multiple “start crawl from” paths.The program is shareware with a fully functional 30 days free
trial, but if you need a licensed review copy with no strings attached
before testing/deciding, I will be happy to send one of course.The license is not subscription based and not per-month or per-year.
For those who buy, it is a one-time price at $69 for version 7.x
including a free 8.x upgrade if released within one year of purchase
with no built-in restraints on how long you can keep using the software.For a full run-through and guide (including video) of all the features and uses, see:
http://www.microsystools.com/products/website-analyzer/help/site-analysis-seo-audit/P.S:
It is worth noting that A1 Website Analyzer also has five sibling
tools – one of them being A1 Sitemap Generator, which can create
XML sitemaps, video sitemaps, image sitemaps, HTML/CSS sitemap etc.
But here’s a link to a resource article going over most of the features
http://www.microsystools.com/products/website-analyzer/help/site-analysis-seo-audit/
w3c Link Checker –
• Company/Parent Toolset: Mozilla Foundation
• User Interface: Website
• Cost: Free
• User Interface: Website and Web Application
• Cost: Free scan and $89.99, $189.99, $289.99
• Crawl Depth: Site wide
• Unique Function: Malware identification and cleanup • Recommended Resource Article:
• Platforms Supported: Windows 95/98/2000/NT/ME/XP/Vista/7
• User Interface: Desktop Download
• Cost:
• Crawl Depth: Sitewide • Unique Function:
• Resource Article: