Google Forum

Double Your Adwords Profits in 7 minutes!


$7 Secrets
People are making thousands of dollars using the techniques and scripts included with this popular report. And it costs only $7! [Read more...]

AdWords Secrets
Free 5-day course that can help you make money using Google AdWords.

Articles for 25 Cents Each
Get 400 exact keyword optimized articles, delivered to your inbox, every month. A new and inexpensive way to build keyword rich web sites that can make money, month after month.

Trade Links with 5,000+ Sites
This is not an automated link system. The Add URL Directory is a directory that lists over 5,000 "add url" pages that have forms that you can use to add your site and exchange links. The directory is divided into categories, helping you find sites in your topic area.

Backlink Analyzer
Automatically analyze the anchor text of all of your backlinks. Analyze your competitors' backlinks to see how you compare in the anchor text department.

SEO Web Site Templates
Web site templates that have been designed specifically for search engine optimization (SEO).

My sponsered child, Hama from Niger, Africa
A portion of the proceeds from this site help sponsor Hama from Niger. Learn more about Child Sponsorship.

Robots.tx, nofollow, noindex


Author Message
GoogleGuy Says

PostPosted: March 11, 2004 6:11 PM 

Importance: Medium

GoogleGuy discusses some of the methods for preventing Google from indexing pages.

GoogleGuy Says: [Link to quote]

"I've noticed GoogleGuy hasn't been around here as much ..."

I keep getting drafted to help on lots of projects lately, so I've had less free time to post, but I'm still around.

jimbeetle had it: if a page is forbidden by robots.txt, we won't crawl it, but if someone else links to that page, we can return that link without ever crawling the page. Back in ancient times, for example, nytimes.com had a robots.txt that wouldn't let Googlebot fetch any pages. So we wouldn't fetch any pages from them. But if the user did the query New York Times, we would return a link to www.nytimes.com without ever crawling their pages, because we had a reasonably high confidence that the url was relevant, even though we didn't actually fetch it.

We do obey noindex/nofollow, but only if we're able to fetch the page in the first place in order to see those tags. :) Bear in mind that noindex will be respected, but if page A links to page B with nofollow *but* page C links to page B, we may still find page B by following the link from page C.

If you want to be safe, I'd recommend using a password via .htaccess.

Subscribe to this discussion: Email

Join the conversation:

Remember personal info?

Check to Subscribe to this Comment:
(email field must be filled in)

Subscribe Without Commenting