CiviCRM Community Forums (archive)

We did our first large e-mail using CiviMail and a 3rd party MTA yesterday.  Our total list size was 62,236 e-mails.  We mailed in a few batches and did a few experiments and discovered what I think could be a great opportunity to increase overall throughput.

Our first batch was to 8426 e-mails.  The process took a total of 93 minutes.  Averaging 90.6 emails/minute.  This was just one single e-mail running during one single process. 

The total delivery time in the first batch was not ideal so we decided in the next batch of 53,810 e-mails to split it into two e-mails and send them both at the same time.

What's interesting is that when we did this each individual job still processed at 90.6 emails/minute but at the same time.  What this resulted in was a faster overall send rate.  For 53,810 we were able to deliver all of them within 372 minutes at a rate of 144.65 e-mails a minute.

Based on this data I'm wondering whether it would make sense to have a setting somewhere in CiviMail that allows you to define the number of threads per job.  Then when you send a mailing in CiviMail it takes the queued list size and splits it into separate jobs each running on a separate process.  From a reporting standpoint it would all be seen as one e-mail but just offer a way to process mail quicker.

Has anyone looked into mutli-threading individual jobs before?

Yes using a local server, even to relay would be faster but our issues is access to the local server.  Most web hosts don't offer this and the Cloud architecture that we're using doesn't either.

The sub_job_id concept seems like it could be more scalable for a variety of hosting solutions.

We don't quite get the same access as other VPS.  This is the system we're running:
http://www.rackspacecloud.com/cloud_hosting_products/sites

We might take a crack at developing the code to split the job.  If we can get it up and running we'll post the commits so you guys can see and if it make sense.  I agree not a 3.2 change... there's already enough on that plate.

Any advice on what files we should be looking at?

I'm looking at the code from CRM\Mailing\BAO\Job.php

In runJobs

I see that the first query basically get the jobs that has been submitted or scheduled to be run at the point the cron is ran

It then puts a lock on the job or jobs and set their status as 'ran'.

It then queues up all the email addresses then delivers them.

So I guess my question is, basically instead of creating one lock for each job to be run by one cron process, we will need to use some logic (like a partition of N number of email) to create multiple locks therefore resulting in multiple queue and deliver methods to be called for a single job.


I also think I'm way over my head too

Hi Lobo:

So I studied the code a little bit more and found the following:

Although the queue and deliver method in Job.php are based on job_id, the problem is that virtually the relationship between civicrm_mailing and civicrm_mailing_event_job is one to one. Excluding the test mail, you virtually have 1 job for each mailing.

Moreover, it looks like to me the key lies in

CRM/Mailing/BAO/Mailing.php

in particular  function &getRecipients($job_id, $includeDelivered = false, $mailing_id = null)

It basically does the processing of inclusion and exclusion of recipients for each job (But essentially each mailing).


So I guess the following steps could be used to split up the mailing.


1. When mailing is submitted, it creates multiple jobs based on how many records a job can hold (say 50000)
2. There will be an additional or two columns in the civicrm_mailing_event_job table to signify the job limit and offset
3. In function &getRecipients($job_id, $includeDelivered = false, $mailing_id = null), introduce new parameter for limit and offset so it only returns a portion of the emails from the mailing
4. Locks can still be set at job level and not too much code will need to be changed from job.php


What do you think.

Quick question,

If I alter the table to add more columns, can I still use the DAO methods of the entity (for example in this case use something like this)

<?php
                $saveJob = new CRM_Mailing_DAO_Job( );
                $saveJob->start_date = date('YmdHis');
                $saveJob->status     = 'Running';
                
                // new fields I added to civicrm_mailing_job table
                $saveJob->type = 'child';
                $saveJob->parentid = $job->id;

                $saveJob->save();
?>
Right now I'm just using CRM_Core_DAO object and using the query method to write insert SQL. I'm guessing that I can't use the entity DAO because those new fields are not wrapped.


Also, what's the different between instantiate CRM_Core_DAO vs entity DAO objects?


Thanks,


Chang

Lobo, thanks for pointing out the issues with my code on IRC

So my question now is that I have the "child jobs" queue up part of the mailings based on limit and offset. I'm still firing the deliever() method for each child job and update the child status upon the deliever() returns $IsComplete.

I also now have a runJobs_post() that queries all running "parent" jobs and look up to see if all child jobs of the parent job are complete then update the status of the parent job as well as mark the mailing as complete.

Will the new relationship between mailing and mailing jobs affect the mailing report etc?

It turned out that the queries themselves were in place already, I took look at every query that was used in the mailing report and the presentation logic. It actually "sums" up all the statistics from multiple jobs for each mailing already. The only thing I had to do was modify the template to use the "sum" variable rather than the job.0

However, my question now turns if this new delivery system will affect the CiviMailProcessor.php and bounce processing?