Author Topic: CiviMail Bulk mail Memory Exhaust (Read 2781 times)

roswell2011 · November 07, 2011, 11:43:11 am

One of our site uses CiviMail to send blast of size 10k+ every now and then. It used to take about 2 hours to send a blast of this size. We were not having issue before, but since few weeks the site crashes during such kind of blasts. Normally the memory use is about 45-50% but during the blast it peaks up to 100% and the site crashes. When the server restarts the cron runs again and sends the emails again since the mail is not marked as delivered (which is solved in the newer version http://issues.civicrm.org/jira/browse/CRM-8678 ).

Currently we are in CiviCRM v 3.4.5 and wondering if it is an issue in this version which I doubt since it was working fine before.

xavier · November 07, 2011, 12:32:43 pm

Hi,

are you using wget to run the cron? if you do, try using php-cli, that's way more performant (and you can set a different memory setting for the cli and apache)

X+

Hershel · November 09, 2011, 12:07:51 pm

In addition to xavier's suggestion, perhaps you can adjust the number of emails which go out each time? Perhaps if it's slower that will help.

ken · November 09, 2011, 02:25:28 pm

Xavier,

I use the php-cli and set the nice etc so the job is light on resources. I find that the civimail.cronjob.php chews up memory, forcing the server to swap and sometimes crashes.

My PHP settings for CLI and Apache are the same: Max Mem = 128M; Max Execution time = 600s; and these were set high for imports etc.

Any suggestions on what I should set the CLI parameters to?

Thanks,

Ken

Donald Lobo · November 09, 2011, 03:12:38 pm

i would ensure that you hve set the batch limit in your mailer settings. Depending on your host etc, a value between 1K - 10K is appropriate

If you have lots of compute / db power, u can run multiple parallel jobs for faster delivery of mail

lobo

ken · November 10, 2011, 02:05:20 am

Lobo,

I have the Mailer Batch Limit set to 60. It was 300 but my server would run out of memory and crash. Even at 100 the server is unstable. When sending emails the memory usage of the process grows by about 100Mb per 100 emails.

My VPS has 1024Mb of RAM with swap of another 1024Mb. One aspect of the problem is that the web server is also our email server, so things get locked up as the mail server competes with the cron job.

BTW, I have the Mailer Cron Job Limit set to '1' since the civimail.cronjob.php is such a memory hog. The fix for CRM-8460 fails to handle this case at line 52 of bin/civimail.cronjob.php. The "> 1" should be a ">= 1".

Ken

xavier · November 10, 2011, 05:57:38 am

Hi,

Got regularly mailings of several 10k, and damn sure I don't have 10Mb of memory available just for that process

you are on php 5.3.3 on this VPS?

X+

Donald Lobo · November 10, 2011, 06:11:51 am

The last time i monitored memory usage on civimail (maybe in 3.4.x), it was fairly constant. We have quite a few orgs sending fairly large volumes of email (batch limit = 2.5 - 5K) and i dont think they see the below behavior

lobo

ken · November 10, 2011, 02:28:29 pm

Xavier,

We're running PHP 5.3.3 on Ubuntu 10.10, and the email sending has been giving us grief since 3.4.2 (we're now on 3.4.7).

All suggestions are welcome!

Ken

roswell2011 · November 11, 2011, 08:11:19 am

Thanks for the pointer Xavier. I had the batch limit set to 300 but after the crashes I lowered it down to 200 and waiting for another large blast to monitor the system. I also upgrade to 3.4.7 which should fix the multiple email problem in case Apache crashes again. My VPS has 1024 MB RAM with avg utilization of 45-50%. Like Ken's our web server is our mail server as well but my problem started fairly about a month ago. Also there are constant Brute Force Attack going on the server for which we are using cPHulk Brute Force Protection (not sure if it matters).

ken · November 12, 2011, 09:11:23 pm

I've instrumented CRM_Mailing_BAO_Job::deliverGroup() with calls to CRM_Utils_System::memory(). Here are the last few lines from a run where civicrm.cronjob.php is sending 412 emails ...

Quote

Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1645872, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup $key % 101 == 0 = 1649672, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1649680, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1649700, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1653492, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1653504, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1657308, 1321159026
Nov 13 15:37:06 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1657308, 1321159026
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1661108, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1661108, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1664916, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1664916, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1668716, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup call bulkCreate = 1668748, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1668748, 1321159027
Nov 13 15:37:07 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1672520, 1321159027
Nov 13 15:37:08 [info] $CRM_Mailing_BAO_Job::deliverGroup calling compose = 1672552, 1321159028
Nov 13 15:37:08 [info] $CRM_Mailing_BAO_Job::deliverGroup calling send = 1676324, 1321159028
Nov 13 15:37:08 [info] $CRM_Mailing_BAO_Job::deliverGroup exit = 1676712, 1321159028

The memory usage is 1.67Gb
Each call to CRM_Mailing_BAO_Mailing::compose() adds about 3.7Mb

Donald Lobo · November 13, 2011, 07:49:38 am

r u using attachments in the mailing?

Can you attach your mailing text / html to this forum topic, so we can run a similar test on te same mailing

lobo

mcarson · November 13, 2011, 08:59:33 am

Not sure if this is relevant, but I have had some recent issues with out-of-memory issues when sending bulk emails.

On 3.3.5, I had configured the batch limit to about 400 and had no issues. Recently after moving to a much more powerful VPS and upgrading to 3.4.6, the server crashed (a lot to do with how I had configured Parallels). After digging into the logs for both the web and the mail server, I left the batch limit at 400 but changed the configuration of my mail server (Postfix) to throttle the number of processes that are allowed to start. I am using cli under fcgi rather than wget/curl.
After some fine-tuning, I am having no problems.

ken · November 18, 2011, 07:50:20 pm

I've found the cause and created a patch - see http://forum.civicrm.org/index.php/topic,22359.msg93714.html

CiviCRM Community Forums (archive)

News:

Author Topic: CiviMail Bulk mail Memory Exhaust (Read 2781 times)

roswell2011

CiviMail Bulk mail Memory Exhaust

xavier

Re: CiviMail Bulk mail Memory Exhaust

Hershel

Re: CiviMail Bulk mail Memory Exhaust

ken

Re: CiviMail Bulk mail Memory Exhaust

Donald Lobo

Re: CiviMail Bulk mail Memory Exhaust

ken

Re: CiviMail Bulk mail Memory Exhaust

xavier

Re: CiviMail Bulk mail Memory Exhaust

Donald Lobo

Re: CiviMail Bulk mail Memory Exhaust

ken

Re: CiviMail Bulk mail Memory Exhaust

roswell2011

Re: CiviMail Bulk mail Memory Exhaust

ken

Re: CiviMail Bulk mail Memory Exhaust

Donald Lobo

Re: CiviMail Bulk mail Memory Exhaust

mcarson

Re: CiviMail Bulk mail Memory Exhaust

ken

Re: CiviMail Bulk mail Memory Exhaust