An important part of evaluating a site is when (or if) a site was crawled (visited) by a search engine spider.

For example, if you are thinking of buying a site or taking a company on as a client, it’s important to know if the site has any chance of getting free, natural traffic from search engines. If the site has never been indexed or is so rarely visited that new content would take months to be indexed, then you may want to reconsider

Determine Last Spider Visit

To determine the most recent spider visit for a site, simply type “site:yoursitename.com” into the search box of the search engine you are testing. Most search engines, including Google, Yahoo, and MSN, support this command. The search engine will display a list of all of the pages that it is aware of on that site. Each listed page will usually have a link that says “Cached” next to it. Click on “Cached”. There will be an information bar at the top of your browser that will say when the page was cached (unfortunately, Yahoo doesn’t display these dates).

Sometimes you will see the date each page was cached listed on the search engine results page, making your job much easier. The pages with dates displayed are usually the most recently crawled pages and were visited in the last 1-3 days.

By looking at the cache dates of each page, you will be able to see when the most recent spider visit was and the extent of the indexing it did. I usually check my home page, blog, and other high-page-rank pages first—they get indexed far more frequently than the rest of my site.

Check Crawling Reach and Frequency

One of your site evaluation criteria should be an evaluation of when a site was last visited by a search engine spider. Using the site command and cached dates, you can get an idea of how well and how frequently a site is crawled by the search engine spiders.