Robots.txt and Google

[FiberGuy]

I've got a poser for 'ya:


I'm blocking all dynamic content to my site via my robots.txt file.  This has been verified through the use of Google robots.txt analyzer (through Google's sitemaps).

Now try this search:  http://www.google.com/search?num=100&hl=en&safe=off&q=%22Cyrus+Rowlett+Smith%22+1941

In the SERP, go down 10-15 spots and you'll find our page (do a text search for dewittco if you want).

Now, note the two links.  The top link is the dynamic page and under that is the static page (the page that I WANT indexed).

Can someone explain this to me?

(((Note that the top link has the "www" subdomain, while the lower one doesn't.  Not sure what that means, either.)))

[jezebel]

Now, note the two links.  The top link is the dynamic page and under that is the static page (the page that I WANT indexed).

Can someone explain this to me?

(((Note that the top link has the "www" subdomain, while the lower one doesn't.

Maybe you should block the lower ranking static pages instead.

[FiberGuy]

It's been considered to eliminate the static pages altogether, but Google (et al) seem to crawl - and more importantly, index - static pages much, much faster.

I'm just a bit baffled as to why G is indexing blocked pages. 

[regic]

They're indexing the dynamic page but not caching it.

Try something like this:

Code:

User-agent: Googlebot
Disallow: /storefront/xcart/product.php?productid=*

or

Code:

User-agent: Googlebot
Disallow: /storefront/xcart/product.php*

[bargainbloodhound]

You could also try it without the asterisks

Code:

User-agent: Googlebot
Disallow: /storefront/xcart/product.php?productid=

You can try adding the query string  (the ?xxxid= part)to the other pages you want blocked in your robots.txt file.  Googlebot is probably seeing product.php and product.php?productid=xxx as 2 separate pages which is why your block isn't working.

[FiberGuy]

OK... thanks Regic & BBH - I'll be trying that... actually, I don't see the harm in adding all three suggestions, do you?

Aside from this, here's another question (I briefly discussed this with the boss earlier):

What if we do away will all of our product's static pages and allow crawling of those pages - any idea how big of a hit we'll take in the SERP's? 

I don't dare do this so close to the Holidays, but I'm thinking of doing it early next year. 

Please opine!

[bargainbloodhound]

What if we do away will all of our product's static pages and allow crawling of those pages - any idea how big of a hit we'll take in the SERP's?

The static pages aren't that old.  I think you'd recover quickly from any temporary drop.

I don't know why X-cart doesn't just use mod_rewrite instead of using static pages (which were a popular feature of shopping cart and directory scripts in the 90's before Apache added mod_rewrite).

[jezebel]

static pages (which were a popular feature of shopping cart and directory scripts in the 90's

Search engines have gotten much better at indexing dynamic content over the years.  Name me a site that doesn't use dynamic content today.