Author Topic: How do we maximize civimail.cron through put (Read 871 times)

dschafer · January 30, 2012, 04:27:27 pm

Hi
I'm looking for some insight into how mails are sent via civimail.cron.php. in 3.3.5

We use SMTP.com for mail relay since the client is looking for high outbound speed.
Problem is we can't seem to get Civi to send more that a few ten's of messages a minute.

The mailer batch limit and job limit are both set to 0. My expectation was that this would result in the mailer opening a connection to smtp.com and sending all the message over that connection until it was complete.

When monitoring net stat is appears that the process may actually be establishing a new connection for each message? Is that correct?

I tried setting the batch limit to 0 and mailer job size to 1000. I expect (assuming threading..) would result in multiple processes each running in parallel each tasked with sending 1000 messages over a single connection.

The actual result was 1 running job in civicrm_mailing_job with a limit of 1000 and a pending job with a limit of 1000 and offset of 1000. The was not parallelism or threading.

So does the process actual open a new connection for each outbound message and how is "threading" actually intended to function.

Thanks
Dave

Donald Lobo · January 30, 2012, 04:41:18 pm

hey dave:

1. Wierdly we did not notice the "open smtp connection" on every email till very very recently, This has been fixed in 4.1 and we now keep the connection persistent

http://issues.civicrm.org/jira/browse/CRM-9349

2. with the number below u wont get any parallelism. I kinda like the below values for decent sized mailings:

job limit = batch limit = 5000
number of cron jobs = 5 or 10. this might be a 4.1 feature.

have cron running every min. Assume mailing size of 50K and cron job limit == 5

so basically the mailing is split into 10 sub-jobs. Assume the machine can deliver 5K emails in 2 mins. Have cron start every min.

U'll have 5 cron job in parallel after the 5th minute all of them delivering a different child job. so basically in 12-15 mins your mailing should be delivered

lobo

dschafer · January 31, 2012, 10:29:55 am

Thanks lobo -

I was surprised to see the multiple connections. It is not something that is easily noticed if all the traffic is local but we clearly see a performance hit going to the external relay.

Any idea on which file needs the changes.

I see the following at line 333 in /civicrm/packages/Mail/smtp.php

Code: [Select]

        /* If persistent connections are disabled, destroy our SMTP object. */
        if ($this->persist === false) {
            $this->disconnect();
        }

It looks like false is the default state.

At a minimum I guess we could force it to true the send() method in this file and just override whatever is passed by the caller.

Do you know if a new smtp object is created for each message or is it only one per mailing job?

Do you know if the destructor will get called reliably so we are sure the connections get closed?

Thanks

Donald Lobo · January 31, 2012, 12:03:39 pm

hey the patch should be linked to the issue i mentioned. check the fisheye tab

It was one smtp object per mailing job, but the connection was re-established for every mail

yes, i do think the destructors are called reliably. However the connection should be closed when the job dies and hence i do think its a good idea to have a batch limit of 2500 or 5000

lobo

dschafer · February 03, 2012, 01:19:50 pm

Hi
Nothing is posted on the fisheye tab for the issue.

I'd like to patch this back to 3.3.5 if possible.

I'll add a comment on the issue.

- Dave

CiviCRM Community Forums (archive)

News:

Author Topic: How do we maximize civimail.cron through put (Read 871 times)

dschafer

How do we maximize civimail.cron through put

Donald Lobo

Re: How do we maximize civimail.cron through put

dschafer

Re: How do we maximize civimail.cron through put

Donald Lobo

Re: How do we maximize civimail.cron through put

dschafer

Re: How do we maximize civimail.cron through put