Could not acquire lock, another CiviMailProcessor process is running

[ragnarkurm]

Hello,

This is error message I get very often,
about 5 per 24h or so.

I noticed it happens more frequently when backup runs,
but other times as well (maybe high load times).

Here's the full error message:

<br />
<b>Fatal error</b>:  Uncaught exception 'Exception' with message 'Could not acquire lock, another CiviMailProcessor process is running' in /www/xxx/html/sites/all/modules/civicrm/bin/CiviMailProcessor.php:213
Stack trace:
#0 {main}
  thrown in <b>/www/xxx/html/sites/all/modules/civicrm/bin/CiviMailProcessor.php</b> on line <b>213</b><br />

And here is my crontab:

10,40 * * * *   /usr/bin/wget -O - -q -t 1 'http://www.xxx.org/sites/all/modules/civicrm/bin/civimail.cronjob.php?name=...&pass=...&key=...' 'http://www.xxx.org/sites/all/modules/civicrm/bin/CiviMailProcessor.php?name=...&pass=...&key=...'

I havent found anything by google about this problem or by civicrm wiki.

I suspect that during high load times the locks will not be released,
probably because of server/php timeout?

Does anyone has idea how to debug it further?

Does anyone know a good documentation about cron timing setup?
How often should/can I run civimail.cronjob.php and how often CiviMailProcessor.php?
Does cron frequency depend on (Mailer Spool Period, Mailer Batch Limit, Mailer Spool Limit)?
How should I configure apache timeouts (if there are any relevant ones) and same in PHP default_socket_timeout or max_execution_time? So that server wont die away too quickly.

Thanks,
Ragnar.

[davej]

Hi,

Same problem here. I think it's due to the way locking is done. CiviMailProcessor.php uses...

Code: [Select]

$lock = new CRM_Core_Lock('CiviMailProcessor');
...which sends this query to MySQL:

Code: [Select]

SELECT GET_LOCK( 'CiviMailProcessor', 1 )
MySQL handles locks per-server rather than per-database. So if a CiviMailProcessor.php job is running on another site on the same server at the same time, it's competing for the same lock, so one or other will get the message "Could not acquire lock, another CiviMailProcessor process is running".

MySQL docs say:

"Names are locked on a server-wide basis. If a name has been locked by one client, GET_LOCK() blocks any request by another client for a lock with the same name. This allows clients that agree on a given lock name to use the name to perform cooperative advisory locking. But be aware that it also allows a client that is not among the set of cooperating clients to lock a name, either inadvertently or deliberately, and thus prevent any of the cooperating clients from locking that name. One way to reduce the likelihood of this is to use lock names that are database-specific or application-specific. For example, use lock names of the form db_name.str or app_name.str."

I'd have thought this needs fixing in CRM_Core_Lock: it should prepend a string unique to the site*, such as the base URL, to the lock name that it passes to MySQL. I'll check with the team and file an issue if appropriate.

* Need to think about multi-org installations. I think locks should operate per-site (e.g. using base URL) rather than per-installation (e.g. using database name): it's likely that several sites within a multi-org installation will want to use CiviMail and these should not interfere with each other.

A workaround would be to change the times at which the cron jobs run, to avoid clashes.

Dave J

[davej]

Discussed with Lobo on IRC and filed issue with patch: http://issues.civicrm.org/jira/browse/CRM-6030 .

Dave J