webSIGHTdesigns - Web Design, Web Development, Web Hosting

How to Prevent Search Engines from Indexing an Alternative URL

Posted on Wednesday, December 4th, 2013 at 2:37 pm
by webSIGHTdesigns

Services like CloudFlare offer a direct.domain.com URL you can use to bypass their content delivery network, but it can lead to duplicate results in Google searches. Use a dynamic robots.txt and Apache rewrites to resolve the issue.

If you have another URL you use to access your website for some reason, it could potentially lead to duplicate search results. Enter a search for site:yourdomain.com into Google to view your site's results. For instance, if you use the content delivery network CloudFlare then their "direct.yourdomain.com" URL which is used to bypass the caching to FTP or SSH into the box directly, may cause duplicate results. To request that web crawlers not index this other domain, create a robots.php file in your web root:

<?php
    header('Content-type: text/plain');
    if ($_SERVER['HTTP_HOST'] == 'direct.yourdomain.com') {
        echo "User-agent: *\n";
        echo "Disallow: /\n";
    } else {
        include('robots.txt');
    }
?> 

Next, in your web root's .htaccess file add a rewrite rule to serve the php file instead whenever the robots.txt file is requested from the alternative domain name:

RewriteEngine on
RewriteCond %{HTTP_HOST} ^direct.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots.php [NC,L]

The above Apache rewrite rule will show the robots.php instead whenever the host name begins with "direct" followed by a period. It may take some time before Google indexes your site again, but eventually the duplicate results disappear, if the search engine web crawlers honor robots.txt file's request to Disallow the entire web root.

Please Sign In

Please sign in to post a comment.

Web Development

View details »

Web Hosting

View details »

Our Portfolio

View portfolio »

WebSight Designs webSIGHTdesigns preferred email webSIGHTdesigns United States United States