Cannot seem to run parallel jobs for CiviMail process_mailing

[dmdude]

Hi -
  No matter what combination of batch limit, cron job limit, and job size I try, I cannot seem to run parallel cron jobs that do process_mailing (send civimail bulk emails).  This is the error I get:

lock aquisition for my_civi.1.civimail.job.207 attempted when my_civi.1.civimail.job.206 is not released

  This is CiviMail 4.5.1 with Drupal 7, but I think the same code is in 4.5.2.  Has anybody seen this before?  Any ideas about what to try to get parallel process_mailing cron jobs to run?

[dmdude]

To answer my own question, I found the comment below in Lock.php.  It is telling me that parallel processing was disabled to prevent duplicate emails from being sent.  Perhaps this is OK though; I can work around it.  Should I open an issue ticket for this?

   * CRM-12856 locks were originally set up for jobs, but the concept was extended to caching & groups without
   * understanding that would undermine the job locks (because grabbing a lock implicitly releases existing ones)
[... Removed 2 lines]
   * if we are running in the context of the cron log then we would rather die (or at least let our process die)
   * than release that lock - so if the attempt is being made by setCache or something relatively trivial
   * we'll just return TRUE, but if it's another job then we will crash as that seems 'safer'

[Michael McAndrew]

Not an answer to your question - just a request for a bit more background - why do you want to run parallel jobs for CiviMail processing?

[dmdude]

I wanted parallel jobs to increase throughput.  Right now, it takes me about 40 minutes to send 10K emails, and that's in one batch (cron job limit = 0).  Our goal is to have a 100K batch of emails processed in less than an hour.  We think our CPU's, network, and SMTP servers can handle that load.
 

[Upperholme]

Not clear why you'd want/need to get all that mail out so quickly, but that's probably a separate issue.

[lcdweb]

We are using the multi-threading options for CiviMail and have not had any issues with it -- it's working as expected (and agree it's an important functionality for increasing mailing throughput).

Try some different settings:
batch limit = 1000
throttle time = 0
job size = 1000
cron job limit = 10

when the mailing is first picked up by the cron job, you should see it split into subjobs in the civicrm_mailing_job table. You'll see multiple rows with the same mailing ID, most of which will have job_type = child with a job_limit of 1000 (the mailer job size setting).

set you're cron jobs to run frequently (at least every 5 min) and then monitor the cron logs and the relevant rows in mailing_job for this mailing ID. you should see an additional child job picked up for processing every time the cron runs (in the mailing_job table note the start_date column). (I'm going on memory here, so it's possible all child jobs are picked up in a single run, I don't recall)

if you're not seeing this happening, you might need to investigate your SMTP server. most servers set limits on how many mailing socket connections can be opened simultaneously. you may be hitting limits on that side, preventing civimail from obtaining a lock.

[jamie]

Are you using the mail() function or relaying via SMTP? I haven't run any tests, but I imagine you will get better performance using smtp.

[davej]

  No matter what combination of batch limit, cron job limit, and job size I try, I cannot seem to run parallel cron jobs that do process_mailing (send civimail bulk emails).  This is the error I get:

lock aquisition for my_civi.1.civimail.job.207 attempted when my_civi.1.civimail.job.206 is not released

  This is CiviMail 4.5.1 with Drupal 7, but I think the same code is in 4.5.2.

We're getting the same issue in 4.4.14 with these settings:

batch limit = 2500
throttle time = 5
job size = 2500
cron job limit = 2
civimail_server_wide_lock = off

Civi cron is running every minute. Mailings scheduler is the only scheduled job that runs on every cron run (others are hourly or daily). The mailing is correctly split into sub-jobs of 2500 mails, however these are processed consecutively. Each batch takes about 6 minutes to run. Every time a new instance of the scheduled job tries to start while an existing instance is running, we get the same error that dmdude reports:

Finished execution of Mailings scheduler with result: Failure, Error message: lock aquisition for mydb.1.civimail.job.12415attempted when mydb.1.civimail.job.12414is not released

Hence only one instance runs at a time. As dmdude says, it seems that CRM_Core_Lock::hackyHandleBrokenCode(), which generates the error message above, prevents multiple instances running simultaneously. Looking at CRM_Mailing_BAO_Mailing::processQueue(), the code for $config->mailerJobsMax > 1 seems to work on the assumption that a CRM_Core_Lock can be created independently for each of the mailerJobsMax cron jobs. But since CRM-12856, CRM_Core_Lock::__construct() disallows multiple locks with names containing "civimail.job." . Which, as far as I can see, prevents parallel mail jobs. (This wouldn't have worked correctly in any case, given that each MySQL connection can only acquire one lock: acquiring a second, even if it has a different name, releases the first: https://dev.mysql.com/doc/refman/5.5/en/miscellaneous-functions.html#function_get-lock)

But if that's right, how come lcdweb is seeing this work correctly - can anyone confirm that mail jobs are indeed sending in parallel in their post-CRM-12856 Civi?

[Added 22 April 2015:] lcdweb has found instances of the locking error in civicrm_job_log and is filing an issue. See also http://civicrm.stackexchange.com/questions/351/can-civimail-send-multiple-mailing-jobs-concurrently .

Dave