CiviCRM Community Forums (archive)

Hi,

I am trying to prevent Robots to access CiviCRM Event Registration. I want/need my pages to be indexed but I haven't found a way to make them stop at a certain level.

Current sitemap is produced by XMAP and there is no CiviCRM-XMAP plugin to control/expose what I need. I don't want to go back to a fixed robots.txt file because I don't want to be involved each time a new page is added.

For the last months, my logs have been growing at 10Mb per day and after carefull examination yesterday I discovered that GoogleBot was trying to register and tell a friend about the incoming golf activity for my organisation. I need the Golf page to be indexed but I am pretty sure that GoogleBot won't be playing golf this summer with us.

The only way I found to solve this was to force exit when a BOT is detected:
/components/com_civicrm# diff civicrm.php.original civicrm.php
14a15,17
> /* Stop BOTs */
> if ( preg_match('/(bot|spider|yahoo)/i', $_SERVER[ "HTTP_USER_AGENT" ] )) exit();
>

With this patch, the NOTHING showed in the logs from CiviCRM for the last 48 hours. The logs are empty, system is quiet and I can sleep. ;-)

Any suggestion?

Thanks,

Yves

They don't submit but they sure try. Without this patch CiviCRM logs are filled with this:

May 02 09:40:14  [info] $Fatal Error Details = Array
(
    [message] => Vous n'avez pas les droits nécessaires pour accéder à cette page.
    =>
)
May 02 09:40:14  [info] $backTrace = #0 /administrator/components/com_civicrm/civicrm/CRM/Core/Error.php(315): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /administrator/components/com_civicrm/civicrm/CRM/Utils/System/Joomla.php(551): CRM_Core_Error::fatal("Vous n'avez pas les droits nécessaires pour accéder à cette page.")
#2 /administrator/components/com_civicrm/civicrm/CRM/Utils/System.php(220): CRM_Utils_System_Joomla->permissionDenied()
#3 /administrator/components/com_civicrm/civicrm/CRM/Event/Page/EventInfo.php(63): CRM_Utils_System::permissionDenied()
#4 /administrator/components/com_civicrm/civicrm/CRM/Core/Invoke.php(296): CRM_Event_Page_EventInfo->run((Array:3), NULL)
#5 /administrator/components/com_civicrm/civicrm/CRM/Core/Invoke.php(72): CRM_Core_Invoke::runItem((Array:15))
#6 /administrator/components/com_civicrm/civicrm/CRM/Core/Invoke.php(54): CRM_Core_Invoke::_invoke((Array:3))
#7 /components/com_civicrm/civicrm.php(78): CRM_Core_Invoke::invoke((Array:3))
#8 /components/com_civicrm/civicrm.php(18): civicrm_invoke()
#9 /libraries/joomla/application/component/helper.php(383): require_once("/components/com_civicrm/civicrm.php")
#10 /libraries/joomla/application/component/helper.php(351): JComponentHelper::executeComponent("/components/com_civicrm/civicrm.php")
#11 /includes/application.php(197): JComponentHelper::renderComponent("com_civicrm")
#12 /index.php(60): JSite->dispatch()
#13 {main}
[/tt]

The robots.txt solution isn't practical, it would require that each time a user post a new activity I have to add the proper link in the file.

Yves