CiviCRM Community Forums (archive)

I can't get CiviMail to handle bounces.  The CiviMail Processor (return channel) runs normally and will properly process replies and other verp codes, but it never recognizes the bounces which all wind up in an "ignored" folder on our server.  We operate only a single site/server and use gmail for our smtp and incoming pop mail.  When I look at the email that comes back from the bounce, I notice 2 "Return-Path:" entries in the header (see original message below), one of which has the +b localpart supplement to the email address and one of which doesn't.  Could this be our problem?  If so, any ideas about why that might happen?

Any suggestions would be appreciated!

Below is the full message pulled from the "ignored" box (not reflected as a bounce) with names changed.  Note that college.edu was a valid institution but ztest was not a valid username there.

Delivered-To: group@mygroup.com
Received: by 10.231.168.210 with SMTP id v18cs115849iby;
        Thu, 19 Nov 2009 07:59:30 -0800 (PST)
Received: by 10.224.60.3 with SMTP id n3mr90082qah.245.1258646370242;
        Thu, 19 Nov 2009 07:59:30 -0800 (PST)
MIME-Version: 1.0
Return-Path: <>
Received: by 10.224.60.3 with SMTP id n3mr122681qah.245; Thu, 19 Nov 2009
   07:59:30 -0800 (PST)
From: Mail Delivery Subsystem <mailer-daemon@googlemail.com>
To: group@mygroup.com
X-Failed-Recipients: ztest@college.edu
Subject: Delivery Status Notification (Failure)
Message-ID: <00c09fa21d41cbc75b0478bb7148@google.com>
Date: Thu, 19 Nov 2009 15:59:30 +0000
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Delivery to the following recipient failed permanently:

     ztest@college.edu

Technical details of permanent failure:=20
Google tried to deliver your message, but it was rejected by the recipient =
domain. We recommend contacting the other email provider for further inform=
ation about the cause of this error. The error that the other server return=
ed was: 550 550 No such user (ztest@college.edu) (state 14).

----- Original message -----

Received: by 10.224.60.3 with SMTP id n3mr90046qah.245.1258646368168;
        Thu, 19 Nov 2009 07:59:28 -0800 (PST)
Return-Path: <group@mygroup.com>
Received: from civicrm.mygroup.com (cpe-74-76-10-87.ny=
cap.res.rr.com [74.76.10.87])
        by mx.google.com with ESMTPS id 23sm522810qyk.11.2009.11.19.07.59.2=
7
        (version=3DSSLv3 cipher=3DRC4-MD5);
        Thu, 19 Nov 2009 07:59:27 -0800 (PST)
Message-ID: <4b056b5f.9753f10a.610e.1e03@mx.google.com>
MIME-Version: 1.0
Reply-To: group+r.11.39.147f08201e12660f@mygroup.com
Return-Path: group+b.11.39.147f08201e12660f@mygroup.com
From: "My Group" <group@mygroup.com>
Subject: Bounce test
List-Unsubscribe: <mailto:group+u.11.39.147f08201e12660f@mygroup.com>
To: "Kenneth test" <ztest@college.edu>
Content-Type: multipart/alternative;
   boundary=3D"=3D_6d2c6785daeb3180cf6a48f9053e8afe"
Date: Thu, 19 Nov 2009 10:59:27 -0500

Sample Header for HTML formatted content.=20

   Trying a new mailing rather than resending an existing one to see if
only 1 return path is generated.=20

   <div class=3D"location vcard"><span class=3D"adr"><span class=3D"street-ad=
dress">P.O. Box 86</span>
<span class=3D"locality">Localville</span>, <span class=3D"region">NY</span> =
<span class=3D"postal-code">11111</span>
<span class=3D"country-name">United States</span></span></div>=20

   http://civicrm.mygroup.com/drupal/index.php?q=3Dcivic=
rm/mailing/optout&reset=3D1&jid=3D11&qid=3D39&h=3D147f08201e12660f Sample F=
ooter for HTML formatted content.

The cron job runs daily and it brings back any replies, unsubscribes and others successfully, but it never catches any bounces.  It brings them back into the "ignored" area which is what it will do with any email it doesn't recognize.  I believe this may be due to having 2 "Return-Path" headers in the original message but I'm not certain about that.  The original message that bounced had the following headers in it (note the 2 Return-Path headers):

Received: by 10.224.110.71 with SMTP id m7mr3171447qap.127.1258146504750;
        Fri, 13 Nov 2009 13:08:24 -0800 (PST)
Return-Path:
Received: from xxx.com (cpe-74-76-10-87.nycap.res.rr.com [74.76.10.87])
        by mx.google.com with ESMTPS id 22sm1555315qyk.10.2009.11.13.13.08.23
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 13 Nov 2009 13:08:24 -0800 (PST)
Message-ID: <4afdcac8.9653f10a.40b2.7817@mx.google.com>
MIME-Version: 1.0
Reply-To: r.8.37.bdfe7a65b41c2a5b@xxx.com
Return-Path: b.8.37.bdfe7a65b41c2a5b@xxx.com
From: "xxx"
Subject: Test
List-Unsubscribe:
To: "Test User"
Content-Type: multipart/alternative;
   boundary="=_b6e06ebb65953e7f7cd28726dfbb126f"
Date: Fri, 13 Nov 2009 16:08:31 -0500

Any ideas why there might be 2 Return-Path headers in the messages?  We're just generating a simple mass mailing in civimail which goes out normally through the cron.  Emails with good addresses are delivered with no problem and replies and other types are handled normally.  I should have mentioned that we're using civicrm 3.0 and drupal 6.14.  The "Return Path" entry for our civi Mail Accounts is blank (value is null).

Thanks for any help you can offer!

Ken

I verified that the "Return Path" in the "Mail Account" was truly null - put the cursor there and hit backspace a bunch of times and then del many times.  The cursor was always at the first position in the field.  In addition when I select it out of the database table with phpmyadmin, it shows as null.

The send mail to contact function just opens a client window on my computer so it wouldn't test sending from the server.  I did send a test message at the end of a new mailing to a bad email address hoping that would be the same as a send mail from contact.  The result that came back was the same (see below).  However, when I send a test message to my correct email address I get an email that shows no "Return-Path" headers at all.

Delivery to the following recipient failed permanently:

     kqapeman@skidmore.edu

Technical details of permanent failure:=20
Google tried to deliver your message, but it was rejected by the recipient =
domain. We recommend contacting the other email provider for further inform=
ation about the cause of this error. The error that the other server return=
ed was: 550 550 No such user (kqapeman@skidmore.edu) (state 14).

----- Original message -----

Received: by 10.224.42.83 with SMTP id r19mr209652qae.35.1259284090141;
        Thu, 26 Nov 2009 17:08:10 -0800 (PST)
Return-Path:
Received: from civicrm.xxx.com (cpe-74-76-10-87.ny=
cap.res.rr.com [74.76.10.87])
        by mx.google.com with ESMTPS id 8sm3379671qwj.13.2009.11.26.17.08.0=
8
        (version=3DSSLv3 cipher=3DRC4-MD5);
        Thu, 26 Nov 2009 17:08:09 -0800 (PST)
Message-ID: <4b0f2679.48c3f10a.76d5.fffffe3f@mx.google.com>
MIME-Version: 1.0
Reply-To: spc+r.19.46.fe4c46bf4b537c98@xsxxx.com
Return-Path: spc+b.19.46.fe4c46bf4b537c98@xxx.com
From: "CiviCRM Test Mailer (xxx)"
Subject: Test Mailing: test of send mail to contact
List-Unsubscribe:
To: "xxx"

OK - I just used the send email to contact function to an invalid email address and it came back with the blank "Return-Path" header, so I guess google must be adding it.  We're using a gmail ssl smtp server - I couldn't figure out how to ping it.  Our domain server just uses the standard civi cron process to send the mail to gmail.

On the positive side I did implement a workaround that will be fine for us.  Since the original return path header only shows up in the "original message" text of a bounced email, I added some code to civimailprocessor to look at the entire message for the verp return path string only if all other verp tests fail.  I just did a test now and it picked up the bounced email as a bounce and put my email address on hold.  I also found a needed small change to one row in the table "civicrm_mailing_bounce_pattern", "ID" 46:
change from:
delivery to the following recipients failed
to:
delivery to the following recipient(s)? failed

to accommodate delivery failure to only 1 recipient

The code changes I made to civimailprocessor.php in case they might be useful to someone else were:

add after line 87 ($regex = ...) the following line:
        $spcRegex = '/Return-Path: ' . preg_quote($dao->localpart) . 'b' . $twoDigitString . '([0-9a-f]{16})@' . preg_quote($dao->domain) . '/';

copy lines 130-149 (starting with text = ""; through the end of the outer "if" statement, within the case b) - do not delete these lines from their existing location
and insert the copied lines immediately after line 116 (if (!matches) {)
Then add the following lines right after the last of the lines you just inserted:
                    if (preg_match($spcRegex, $text, $matches)) {
                        list($match, $job, $queue, $hash) = $matches;
                    }
                    if ($matches) {
                      $action = "";
                      crm_mailer_event_bounce($job, $queue, $hash, $text);
                    } else {
                      $store->markIgnored($key);
                      continue;
                    }

To sum that up, you would then have the following starting with line 87 up to the unchanged switch ($action);
        $regex = '/^' . preg_quote($dao->localpart) . '(b|c|e|o|r|u)' . $twoDigitString . '([0-9a-f]{16})@' . preg_quote($dao->domain) . '$/';
        $spcRegex = '/Return-Path: ' . preg_quote($dao->localpart) . 'b' . $twoDigitString . '([0-9a-f]{16})@' . preg_quote($dao->domain) . '/';

        // retrieve the emails
        require_once 'CRM/Mailing/MailStore.php';
        $store = CRM_Mailing_MailStore::getStore($name);

        require_once 'api/Mailer.php';

        // process fifty at a time, CRM-4002
        while ($mails = $store->fetchNext(50)) {
            foreach ($mails as $key => $mail) {

                // for every addressee: match address elements if it's to CiviMail
                $matches = array();
                foreach ($mail->to as $address) {
                    if (preg_match($regex, $address->email, $matches)) {
                        list($match, $action, $job, $queue, $hash) = $matches;
                        break;
                    // FIXME: the below elseifs should be dropped when we drop legacy support
                    } elseif (preg_match($commonRegex, $address->email, $matches)) {
                        list($match, $action, $_, $job, $queue, $hash) = $matches;
                        break;
                    } elseif (preg_match($subscrRegex, $address->email, $matches)) {
                        list($match, $action, $_, $job) = $matches;
                        break;
                    }
                }

                // if $matches is empty, this email is not CiviMail-bound
                // but for spc first check to see if a bounce return path is anywhere in the original message header
                if (!$matches) {
                    $text = '';
                    if ($mail->body instanceof ezcMailText) {
                        $text = $mail->body->text;
                    } elseif ($mail->body instanceof ezcMailMultipart) {
                        if ($mail->body instanceof ezcMailMultipartRelated) {
                           foreach ($mail->body->getRelatedParts() as $part) {
                               if (isset($part->subType) and $part->subType == 'plain') {
                                   $text = $part->text;
                                   break;
                               }
                           }                           
                        } else {   
                           foreach ($mail->body->getParts() as $part) {
                               if (isset($part->subType) and $part->subType == 'plain') {
                                   $text = $part->text;
                                   break;
                               }
                           }
                        }
                    }
                    if (preg_match($spcRegex, $text, $matches)) {
                        list($match, $job, $queue, $hash) = $matches;
                    }
                    if ($matches) {
                      $action = "";
                      crm_mailer_event_bounce($job, $queue, $hash, $text);
                    } else {
                      $store->markIgnored($key);
                      continue;
                    }
                }

                // get $replyTo from either the Reply-To header or from From
                // FIXME: make sure it works with Reply-Tos containing non-email stuff
                $replyTo = $mail->getHeader('Reply-To') ? $mail->getHeader('Reply-To') : $mail->from->email;

                // handle the action by passing it to the proper API call
                // FIXME: leave only one-letter cases when dropping legacy support
                switch ($action) {

Ken

Sure - I'll send the file to your email address since it's not a file type that the forum attach seems to support.

Thanks for your help!

Ken

Hi, I managed to rework your patch to be a bit simpler (and, thus, better suited for inclusion in the v3.0 branch) – can you please (a) take a look at r25337 and (b) verify that the CiviMailProcessor.php from the v3.0 branch’s bin directory works in your case?

I simplified the patch by scanning the whole body of the soft bounce (without splitting it into parts) for the Return-Path: bounce pattern and by keeping the other logic intact.