Author Topic: Time out on IMAP processing (Read 7360 times)

goran · December 19, 2008, 04:09:12 am

I have already successfully been using CiviMailProcessor.php (set it up on IMAP account) and it worked quite nice. I had to increase max_execution_time in php.ini to process my catchall account once it started to get filled up (it is a catchall for domain with several hundred mailboxes).

I have increased it to 600 seconds, but even that had not been enough.
Is there a way to increase max_execution_time for a single script or any other solutions for this?

Thanks.

------------------------------------
I need a tag line that praise CiviCRM

Donald Lobo · December 19, 2008, 07:02:17 am

I think we'll need to tweak the script to process incoming mail in a "batch size" rather than all incoming mails. This might be faster and more memory efficient. I'll discuss this with piotr

The script should also try to make execution time unlimited if possible using the following code (from import)

// lets get around the time limit issue if possible, CRM-2113
if ( ! ini_get( 'safe_mode' ) ) {
set_time_limit( 0 );
}

lobo

goran · December 19, 2008, 12:52:32 pm

Thank you.

goran · January 16, 2009, 06:43:59 am

With the increase of spam/e-mails during holidays the problem came back... to stay.

There seems to be a certain number of mails after which the CiviMailProcessor.php just does not scale anymore. After looking a bit at the code I would say that if anything it should be made more resilient than (potentially) efficient. In another words, as I understand it - it currently tries to do all at once and if it fails nothing is done, where the opposite would be much more appreciated for anything that should go into cron - do as much as you can until it fails (so in this case it is better to do one by one e-mail).

P.S. I am trying to hack through this as I am already using it live and am trying to fix this.

P.S.S. Is there a reason why eZ Components used in civicrm are such an old version?

Donald Lobo · January 16, 2009, 08:03:23 am

can you please file an issue and we'll fix it for 2.2.

Here's a first attempt at a patch:

Code: [Select]

Index: CRM/Mailing/MailStore.php
===================================================================
--- CRM/Mailing/MailStore.php	(revision 19381)
+++ CRM/Mailing/MailStore.php	(working copy)
@@ -95,6 +95,24 @@
         return $mails;
     }
 
+    function someMails( $offset = 1 , $limit = 50 )
+    {
+        try {
+            $set = $this->_transport->fetchFromOffset( $offset, $limit );
+        } catch ( ezcMailOffsetOutOfRangeException $e ) {
+            return false;
+        }
+
+        $mails = array();
+        $parser = new ezcMailParser;
+        foreach ($set->getMessageNumbers() as $nr) {
+            if ($this->_debug) print "retrieving message $nr\n";
+            $single = $parser->parseMail($this->_transport->fetchByMessageNr($nr));
+            $mails[$nr] = $single[0];
+        }
+        return $mails;
+    }
+
     /**
      * Point to (and create if needed) a local Maildir for storing retrieved mail
      *
Index: bin/CiviMailProcessor.php
===================================================================
--- bin/CiviMailProcessor.php	(revision 19381)
+++ bin/CiviMailProcessor.php	(working copy)
@@ -58,83 +58,89 @@
         // retrieve the emails
         require_once 'CRM/Mailing/MailStore.php';
         $store = CRM_Mailing_MailStore::getStore($name);
-        $mails = $store->allMails();
-
         require_once 'api/Mailer.php';
-        foreach ($mails as $key => $mail) {
 
-            // for every addressee: match address elements if it's to CiviMail
-            $matches = array();
-            foreach ($mail->to as $address) {
-                if (preg_match($regex, $address->email, $matches)) {
-                    list($match, $action, $job, $queue, $hash) = $matches;
-                    break;
-                // FIXME: the below elseifs should be dropped when we drop legacy support
-                } elseif (preg_match($commonRegex, $address->email, $matches)) {
-                    list($match, $action, $_, $job, $queue, $hash) = $matches;
-                    break;
-                } elseif (preg_match($subscrRegex, $address->email, $matches)) {
-                    list($match, $action, $_, $job) = $matches;
-                    break;
-                }
+        while ( 1 ) {
+            $mails = $store->someMails( 1, 50 );
+            if ( $mails == false ) {
+                break;
             }
 
-            // if $matches is empty, this email is not CiviMail-bound
-            if (!$matches) {
-                $store->markIgnored($key);
-                continue;
-            }
+            foreach ($mails as $key => $mail) {
 
-            // get $replyTo from either the Reply-To header or from From
-            // FIXME: make sure it works with Reply-Tos containing non-email stuff
-            $replyTo = $mail->getHeader('Reply-To') ? $mail->getHeader('Reply-To') : $mail->from->email;
+                // for every addressee: match address elements if it's to CiviMail
+                $matches = array();
+                foreach ($mail->to as $address) {
+                    if (preg_match($regex, $address->email, $matches)) {
+                        list($match, $action, $job, $queue, $hash) = $matches;
+                        break;
+                        // FIXME: the below elseifs should be dropped when we drop legacy support
+                    } elseif (preg_match($commonRegex, $address->email, $matches)) {
+                        list($match, $action, $_, $job, $queue, $hash) = $matches;
+                        break;
+                    } elseif (preg_match($subscrRegex, $address->email, $matches)) {
+                        list($match, $action, $_, $job) = $matches;
+                        break;
+                    }
+                }
 
-            // handle the action by passing it to the proper API call
-            // FIXME: leave only one-letter cases when dropping legacy support
-            switch($action) {
-            case 'b':
-            case 'bounce':
-                if ($mail->body instanceof ezcMailText) {
-                    $text = $mail->body->text;
-                } elseif ($mail->body instanceof ezcMailMultipart) {
-                    foreach ($mail->body->getParts() as $part) {
-                        if (isset($part->subType) and $part->subType == 'plain') {
-                            $text = $part->text;
-                            break;
+                // if $matches is empty, this email is not CiviMail-bound
+                if (!$matches) {
+                    $store->markIgnored($key);
+                    continue;
+                }
+
+                // get $replyTo from either the Reply-To header or from From
+                // FIXME: make sure it works with Reply-Tos containing non-email stuff
+                $replyTo = $mail->getHeader('Reply-To') ? $mail->getHeader('Reply-To') : $mail->from->email;
+
+                // handle the action by passing it to the proper API call
+                // FIXME: leave only one-letter cases when dropping legacy support
+                switch($action) {
+                case 'b':
+                case 'bounce':
+                    if ($mail->body instanceof ezcMailText) {
+                        $text = $mail->body->text;
+                    } elseif ($mail->body instanceof ezcMailMultipart) {
+                        foreach ($mail->body->getParts() as $part) {
+                            if (isset($part->subType) and $part->subType == 'plain') {
+                                $text = $part->text;
+                                break;
+                            }
                         }
                     }
+                    crm_mailer_event_bounce($job, $queue, $hash, $text);
+                    break;
+                case 'c':
+                case 'confirm':
+                    crm_mailer_event_confirm($job, $queue, $hash);
+                    break;
+                case 'o':
+                case 'optOut':
+                    crm_mailer_event_domain_unsubscribe($job, $queue, $hash);
+                    break;
+                case 'r':
+                case 'reply':
+                    // instead of text and HTML parts (4th and 6th params) send the whole email as the last param
+                    crm_mailer_event_reply($job, $queue, $hash, null, $replyTo, null, $mail->generate());
+                    break;
+                case 'e':
+                case 're':
+                case 'resubscribe':
+                    crm_mailer_event_resubscribe($job, $queue, $hash);
+                    break;
+                case 's':
+                case 'subscribe':
+                    crm_mailer_event_subscribe($mail->from->email, $job);
+                    break;
+                case 'u':
+                case 'unsubscribe':
+                    crm_mailer_event_unsubscribe($job, $queue, $hash);
+                    break;
                 }
-                crm_mailer_event_bounce($job, $queue, $hash, $text);
-                break;
-            case 'c':
-            case 'confirm':
-                crm_mailer_event_confirm($job, $queue, $hash);
-                break;
-            case 'o':
-            case 'optOut':
-                crm_mailer_event_domain_unsubscribe($job, $queue, $hash);
-                break;
-            case 'r':
-            case 'reply':
-                // instead of text and HTML parts (4th and 6th params) send the whole email as the last param
-                crm_mailer_event_reply($job, $queue, $hash, null, $replyTo, null, $mail->generate());
-                break;
-            case 'e':
-            case 're':
-            case 'resubscribe':
-                crm_mailer_event_resubscribe($job, $queue, $hash);
-                break;
-            case 's':
-            case 'subscribe':
-                crm_mailer_event_subscribe($mail->from->email, $job);
-                break;
-            case 'u':
-            case 'unsubscribe':
-                crm_mailer_event_unsubscribe($job, $queue, $hash);
-                break;
+
+                $store->markProcessed($key);
             }
-
-            $store->markProcessed($key);
         }
     }

I dont know the code well enuf, but i think its pretty close. i dont have a setup right now, so u'll need to test it

We are using a prety recent version (2008.1 i think). their version numbering system is a bit messed up (IMO)

lobo

goran · January 16, 2009, 08:42:56 am

Thanks for the quick reply. Will be testing it.

As for the versions - I compared versions mentioned in files, for example ezcMailImapTransport class in imap_transport.php (distributed with civicrm 2.1.4) declares Mail package version 1.3. Latest stable from ezcomponents.org is at Mail package version 1.6 (see http://ezcomponents.org/docs/api/latest/Mail/_Mail---1.6---src---transports---imap---imap_transport.php.html)

From http://ezcomponents.org/download/dl_components I think that Mail 1.3 corresponds to 2007.1 version, Mail 1.6 corresponds to 2008.2 version

I believe there were several not so small bugs corrected in between.

goran · January 16, 2009, 09:00:16 am

Opened http://issues.civicrm.org/jira/browse/CRM-4002

Donald Lobo · January 16, 2009, 02:17:14 pm

http://fisheye.civicrm.org/changelog/CiviCRM/trunk/packages/ezc

the version in 2.2 is v2008.1. I'l check with piotr and we might upgrade to the latest version before the next alpha release

lobo

Piotr Szotkowski · January 19, 2009, 09:07:51 am

As for versions, CiviCRM 2.1 didn’t initially ship CiviMailProcessor, and so we didn’t check eZ Components version throughly (and it was a bit too late to upgrade past the release of 2.1.0). For 2.2 we have eZ Components 2008.1, which was the latest stable until two weeks ago.

Is there anything particular in 2008.2 which would make it worthwhile to upgrade eZ Components on the 2.2 branch?

(I’ll be working on the real issue behind this thread today and tomorrow.)

goran · January 19, 2009, 10:11:36 am

Re versions: No, I have no idea what are the changes between 2008.1 and 2008.2. Sorry for the noise - I was referring to diff between 2007.1 and 2008.2...

To compensate for the noise, here's what I've found regarding diff on Mail component between 2008.1 and 2008.2
(paste from http://ezcomponents.org/resources/news/news-2009-01-05)
1.6 - Monday 05 January 2009

* Fixed issue #14220: File attachments mess up emails without body text.

1.6rc1 - Monday 15 December 2008

* Fixed issue #14025: Problem with ezcMailComposer::addAttachment when use the fifth param to change the file name.

1.6beta1 - Monday 01 December 2008

* Fixed issue #14009: ezcMailTools::validateEmailAddressMx() uses wrong HELO domain name.
* The function ezcMailTools::validateEmailAddressMx() throws an exception if there is no support for getmxrr() and checkdnsrr().
* Altered the ezcMailTools::validateEmailAddress() regexp to protect against locale issues.

1.6alpha1 - Monday 10 November 2008

* Implemented issue #13383: Add a method to extract/change/replace entities in HTML mail with the CID elements replaced.
* Implemented feature feature #13539: Add new mail parser option fileClass.
* Fixed issue #13878: Endless loop in ezcMailParser.

Piotr Szotkowski · January 23, 2009, 09:58:46 am

I implemented an initial version of this on the v2.2 branch. It currently throws a cryptic exception on exit, but otherwise seems to work – will fix and test thoroughly on Monday.

(I also updated eZ Components to 2008.2.)

goran · January 23, 2009, 12:18:03 pm

I would like to test over the weekend, but my problem is that the test machine is on 2.1.4 and can't upgrade to 2.2 from trunk (no time to go into that). Do you think I can test the patches on 2.1.4 (I see the settings have moved to the db... so it might not be practical)?

Piotr Szotkowski · January 27, 2009, 05:32:49 am

I’m sorry, but I don’t think that trying to patch CiviCRM 2.1’s CiviMailProcessor with all the stuff that went into this for CiviCRM 2.2 is a viable option; we even upgraded eZ Components (the mail framework we use to connect to IMAP and POP servers) between 2.1 and 2.2.

FWIW, the final implementation of the batch polling is commited and will be a part of the upcoming 2.2.alpha4 (or beta1, depending on what we call it).

goran · January 27, 2009, 06:25:41 am

Upgrading ez Components is not a problem for me (was my suggestion, anyway

) and is already done. Apart from that I would update only CiviMailProcessor and MailStore related files (which was going well; btw, are there dependencies of the other parts of the system on MailStore or is it used only for mail processor?). I got stuck with the change on the database structure and don't know where to get CREATE TABLE for the mail settings without going through with the actual install of 2.2?

Piotr Szotkowski · January 28, 2009, 05:57:46 am

Quote from: goran on January 27, 2009, 06:25:41 am

Apart from that I would update only CiviMailProcessor and MailStore related files (which was going well; btw, are there dependencies of the other parts of the system on MailStore or is it used only for mail processor?).

It’s currently used only for CiviMailProcessor (we have plans to change that in CiviCRM 2.3+).

Quote

I got stuck with the change on the database structure and don't know where to get CREATE TABLE for the mail settings without going through with the actual install of 2.2?

You can get it either from 2.2 tarball’s sql/civicrm.mysql file or from CRM/Upgrade/Incremental/sql/2.2.alpha1.mysql.tpl:

Code: [Select]

-- CRM-3851: migrate civicrm_domain.email_domain and .email_return_path to civicrm_mail_settings
SELECT email_domain, email_return_path FROM civicrm_domain LIMIT 1 INTO @domain, @return_path;


CREATE TABLE `civicrm_mail_settings` (
  `id` int(10) unsigned NOT NULL auto_increment COMMENT 'primary key',
  `name` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'name of this group of settings',
  `is_default` tinyint(4) default NULL COMMENT 'whether this is the default set of settings for this domain',
  `domain` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'email address domain (the part after @)',
  `localpart` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'optional local part (like civimail+ for addresses like civimail+s.1.2@example.com)',
  `return_path` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'contents of the Return-Path header',
  `protocol` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'name of the protocol to use for polling (like IMAP, POP3 or Maildir)',
  `server` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'server to use when polling',
  `port` int(10) unsigned default NULL COMMENT 'port to use when polling',
  `username` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'username to use when polling',
  `password` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'password to use when polling',
  `is_ssl` tinyint(4) default NULL COMMENT 'whether to use SSL or not',
  `source` varchar(255) collate utf8_unicode_ci default NULL COMMENT 'folder to poll from when using IMAP, path to poll from when using Maildir, etc.',
  PRIMARY KEY  (`id`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

INSERT INTO civicrm_mail_settings (name, is_default, domain, return_path) VALUES ('default', true, @domain, @return_path);

CiviCRM Community Forums (archive)

News:

Author Topic: Time out on IMAP processing (Read 7360 times)

goran

Time out on IMAP processing

Donald Lobo

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

Donald Lobo

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

Donald Lobo

Re: Time out on IMAP processing

Piotr Szotkowski

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

Piotr Szotkowski

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

Piotr Szotkowski

Re: Time out on IMAP processing

goran

Re: Time out on IMAP processing

Piotr Szotkowski

Re: Time out on IMAP processing