CiviCRM Community Forums (archive)

Hi

I have had some members of my community complain about finding their personal information in Google because of our CiviCRM site.  I don't much like my personal information exposed in such a way either.

To address this, would you consider including in the online documentation installation instructions a note about robots.txt.

Here is a sample from a Drupal install.

# Paths (clean URLs)
Disallow: /user/
Disallow: /civicrm/
# Paths (no clean URLs)
Disallow: /?q=user/
Disallow: /?q=civicrm/


# $Id: robots.txt,v 1.7.2.1 2007/03/23 18:57:07 drumm Exp $
#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /database/
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /sites/
Disallow: /themes/
Disallow: /scripts/
Disallow: /updates/
Disallow: /profiles/
# Files
Disallow: /xmlrpc.php
Disallow: /cron.php
Disallow: /update.php
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /CHANGELOG.txt
Disallow: /MAINTAINERS.txt
Disallow: /LICENSE.txt
Disallow: /UPGRADE.txt
# Paths (clean URLs)
Disallow: /admin/
Disallow: /aggregator/
Disallow: /comment/reply/
Disallow: /contact/
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/
Disallow: /civicrm/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=aggregator/
Disallow: /?q=comment/reply/
Disallow: /?q=contact/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/
Disallow: /?q=civicrm/



Hi,

I am a newbie and trying to learn how to limit my users content to all search engines.

Unfortunately, it was not clear to me that enabling the profile was going to expose all my user's
personal information to Google's crawl.  I have changed the profile information back but the
search engine's have published all my user's information...

I have been told by Attracta and seen other posts online that the robot.txt is not the place to limit
the google crawl, but that this information needs to be added to the page in drupal and civicrm.

Here is the text that I have been told to add:
<meta name="robots" content="noindex, nofollow, noarchive"/>

Problem is I don't know how to add this to my CiviCRM pages, which is where this
content had been published from.

Can someone please point me to an exact page that will walk me through this process.

My users are irrate with me and I am ticked off that there wasn't a big warning screen
that comes up to explain that Google is going to expose all my user's personal information
to the world...

Donald and everyone else,

I fixed the profile.

It is no longer revealing the information.

But the google crawler holds some of this content since they
"peel off" some of the content from this previously exposed
profile.

The results of the crawl is now sitting out on google's search engine.

I am trying to clear google's cache or search engine results.

I have changed my robot.txt file to exclude the URL's -- but this
is only good for future crawls -- not past crawls.

If anyone can help me figure out how to add this information
<meta name="robots" content="noindex, nofollow, noarchive"/>
to the index file of my Drupal / Civicrm file.

According to a number of sources, once I add this content, I
can use Google Web Tools to re-crawl the site and this should
clear the older content.

All I see on Drupal is an index.php file not an index.htm file...

Thanks,

Lori

Careful with the profile visibility settings.

If you want to ask Google to remove certain indexed pages, you can do so.  Google reserves the right to ignore your request.



http://support.google.com/webmasters/bin/answer.py?hl=en&answer=164734