set amount of mails per second with CiviMAIL scheduler

[koen van hees]

Hello

I thought this would be a common problem, but I can't find any info on it.

I started a new CiviCRM install (4.0.8 with Drupal 7.10) with siteground.com, since they claimed to be the greatest civi host around for a very reasonable price. The price is very reasonable, but I'm finding their hosting formula are not civi minded at all.

2 out of 3 mails I schedule are bounced ("syntax"), not because the addresses are not correct (mine was bounced as well) but because the mail server doesn't accept more than 10 mails / second.

Is there any way to make CiviMail send e.g. 5 mails/second?
(other limits: 1cronjob/hour and <= 400mails/hour)

[Donald Lobo]

1. might want to switch to a better host

2. The only limit we have right now is mailings sent per cron run. You can make the cron go every hour and set the mailng limit to the hourly limit. However it might deliver all that email in the first few seconds / minutes (depending on how fast your machine is)

lobo

[koen van hees]

Thinking hard about option 1.
Have already done option 2. I asked for intervention from their programmers and they tried to limit the amount of mails per second, which didn't work. They thought the large amount of mails was due to multi-threading, but it wasn't. Which sort of brings me back to option one.

Now they suggest using gmail smtp. I'm not sure they'd allow that. Is there a free mail server you can use like that? Why would they allow that?

[val]

2. The only limit we have right now is mailings sent per cron run. You can make the cron go every hour and set the mailng limit to the hourly limit. However it might deliver all that email in the first few seconds / minutes (depending on how fast your machine is)

The mail limit applied this way is not very useful if the application tries to enqueue all e-mail messages at once. The mail per second limitation on our (Siteground) shared servers is applied in attempt to ensure that a single user will not fill-in the mail server queue and thus affect all other users on the same server. This is especially valid when dealing with mailing lists where the number of recipients is usually in the thousands range and naturally, having all the messages enqueued at once in such cases, is not a good idea.

The limit is also not something unique for SiteGround. I know (from my own personal experience) that many other hosting companies, such as APlus, IPowerWeb, Bluehost, LunarPages have the same or similar mail service limitations and it is there for a reason.

What you have recommended as first solution (i.e a change to a different hosting provider) might solve the problem for Koen but it is just a workaround in the long term and there will be other users of CiviCRM who will  be struggling with the same issue. I know that most mailing list scripts and applications (phpList, DaDa Mail and a whole bunch of commercial scripts) have a configurable option where you can instruct the script to send x number of messages every second or send the messages with X milliseconds delay in between. I was rather surprised to see CiviCRM has no such option, as opposed to the otherwise excellent functionality of the app.

Perhaps it would be suitable to implement a simple mail throttling feature? I do not imagine it to be  very difficult to code and it will surely fix the issue for good and for all CiviCRM users.

[Donald Lobo]

Perhaps it would be suitable to implement a simple mail throttling feature? I do not imagine it to be  very difficult to code and it will surely fix the issue for good and for all CiviCRM users.

Would be great and helpful to the community, if you can investigate the other packages and submit a patch to do something similar on CiviCRM. Ping us on IRC if u need help getting started

lobo

[val]

Well, I'm not a full time developer and I don't fancy myself one, but I looked at the files and gathered a small patch file, which I've attached to this message. The patch file edits 4 files of the application and I've listed the changes below (unfortunately the code tag is considered as link by the forum and I couldn't use it) :

File: CRM/Mailing/Config.php
Purpose of the edit: Add a configurable value for the throttling time to the config object.
The actual code:
-----------
     /**
      * How long (in micro seconds) should we wait in between each mail delivery? Setting this to 0 disables it.
      *
      * @var int
      */
     public $mailThrottleTime = 120000;
-----------
The above sets the mailThrottleTime to 120,000 which is about 120ms and having in mind  the limit of 10 messages per second rate, it will be just about enough to avoid any issues with it. The second file to edit is

File: CRM/Admin/Form/Setting/Mail.php
Purpose: Add a form element in the mailer configuration page
Code:

-----------
$this->addElement('text','mailThrottleTime', ts('Mailer Throttle Time'));
-----------

Next file is

File: templates/CRM/Admin/Form/Setting/Mail.tpl
Purpose: Render the above element in the config form.
Code:
-----------
  <tr class="crm-mail-form-block-mailThrottleTime">
      <td class="label">{$form.mailThrottleTime.label}</td><td>{$form.mailThrottleTime.html} <br />
      <span class="description">{ts}The time to (u)sleep in between each e-mail in micro seconds. Setting this above 0 allows you to control the rate at which e-mail messages are sent to the mail server, avoiding filling up the mail queue very quickly. Set to 0 to disable.{/ts}</span></td>
  </tr>
-----------

And the last file to edit

File: CRM/Mailing/BAO/Job.php
Purpose: Throttle each e-mail delivery for the configured micro seconds
Code (around 573 line after the hack to stop mailing job at run time)

-----------
  // If we have enabled the Throttle option, this is the time to enforce it.
  $config = CRM_Core_Config::singleton();
  if($config->mailThrottleTime > 0){
    usleep((int)$config->mailThrottleTime);
  }
-----------

It is in no way perfect, but as mentioned I'm not a full time developer and I don't really know CiviCRM code that well (as you did). Let me know if you have any comments/suggestions about it.
 

[Donald Lobo]

hey val:

Thanx for the patch. much appreciated definitely helps us include it in the next version of civicrm

can u please create an issue here: http://issues.civicrm.org/ and attach the patch there. we'll tweak it a bit and apply it to 4.2

lobo

[osep]

Hello,
I'm using CiviCRM 4.1.5 and I'm interested in the Val's patch.

Someone he tested? Does it work?

Thank you for your answers!

[Donald Lobo]

search the issue queue http://issues.civicrm.org/

that patch was committed, i think for 4.1

lobo

[osep]

hello,

I have found : http://issues.civicrm.org/jira/browse/CRM-9698

In the file:CRM/Mailing/BAO/Job.php : where do I put these lines:

  // If we have enabled the Throttle option, this is the time to enforce it.
  $config = CRM_Core_Config::singleton();
  if($config->mailThrottleTime > 0){
    usleep((int)$config->mailThrottleTime);
  }

Val to put them here : Code (around 573 line after the hack to stop mailing job at run time)

I put it at the end of the function :
public function deliverGroup(&$fields, &$mailing, &$mailer, &$job_date, &$attachments) {

like this :
          // hack to stop mailing job at run time, CRM-4246.
          // to avoid making too many DB calls for this rare case
          // lets do it when we snapshot
          $status = CRM_Core_DAO::getFieldValue('CRM_Mailing_DAO_Job',
            $this->id,
            'status'
          );
          if ($status != 'Running') {
            return FALSE;
          }
        }
      }
  // If we have enabled the Throttle option, this is the time to enforce it.
  $config = CRM_Core_Config::singleton();
  if($config->mailThrottleTime > 0){
    usleep((int)$config->mailThrottleTime);
  }
      unset($result);

      // seems like a successful delivery or bounce, lets decrement error count
      // only if we have smtp connection errors
      if ($smtpConnectionErrors > 0) {
        $smtpConnectionErrors--;
      }
    }

    $result = $this->writeToDB($deliveredParams,
      $targetParams,
      $mailing,
      $job_date
    );

    return $result;
  }

Is that correct ?


Thank you for your help

[awasson]

@osep: I believe you'll find the patch has been applied to recent versions of civiCRM 4.x. Check out the Mailer Settings (Administer -> civiMail -> Mailer Settings). You'll find the fields there.

Andrew

[osep]

euh ?

Is this parameter : Mailer Batch Limit - Throttle email delivery by setting the maximum number of emails sent during each CiviMail run (0 = unlimited) ??

But what means  this sentence : the maximum number of emails sent during each CiviMail run

I need more than one cron job

I would like a few details ...

Thanks

[Donald Lobo]

1. large mails take a long time to deliver

2. each run delivers a certain number of mails as specified

3. cron executes periodically (every 5 mins, every hour etc) based on how u set it up

4. the limit is observed for each run

lobo