CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Demystifying Bounce Detection - help
Pages: [1]

Author Topic: Demystifying Bounce Detection - help  (Read 1526 times)

mcgeehon

  • I post occasionally
  • **
  • Posts: 76
  • Karma: 0
  • CiviCRM version: 4.x
  • CMS version: 7.x
  • MySQL version: 5.x
  • PHP version: 5.2.x
Demystifying Bounce Detection - help
February 08, 2013, 11:41:21 am

First, I have read both http://book.civicrm.org/user/current/initial-set-up/email-system-configuration/ and http://forum.civicrm.org/index.php/topic,16430.0.html.

CiviMail works perfectly for sending mail and I have no problem using my corporate gmail account for use as an inbox for bounced email.

My problem is that I cannot get Civi to recognize a single email as bounced.  I have verified that when an email goes out, it contains several instances of the myEmailAddress+b.14.8.03c664355795e0a1@myDomain.com instances in the header envelope and other places.

  • The return email never contains the unique email string above except for in the attached original email.  I wasn't sure if Civi was expecting to find this unique email string in a specific part of the returned email body?
  • About 90% of my bounced emails contain a subject similar to 'Delivery Status Notification (Failure)'.  I wasn't sure if Civi is matching the patterns in the `civicrm_mailing_bounce_pattern` against the subject of the returned email, or against the entire email itself?

Anyway, I have verified that the received messages contain both the unique email and an assumed matched civicrm bounce pattern, however all messages keep getting placed in the ignored folder.

Any better idea on what the bounce algorithm specifically is looking for would be appreciated.  I may have to dig into code otherwise.

Attached is a sanitized returned email as an example of something that CiviCRM completely ignores, even after I have added 'Delivery Status Notification' as a bounced patter with event type 6.

Thanks

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Demystifying Bounce Detection - help
February 08, 2013, 02:57:53 pm

are you overriding verp in the civimail settings?

if you look at your email message, the Reply-to header is a normal one (and hence gmail sends the reply there) rather than something like

no-response+r.1.3.34......

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

mcgeehon

  • I post occasionally
  • **
  • Posts: 76
  • Karma: 0
  • CiviCRM version: 4.x
  • CMS version: 7.x
  • MySQL version: 5.x
  • PHP version: 5.2.x
Re: Demystifying Bounce Detection - help
February 10, 2013, 12:22:39 pm
Thanks.  The process for sending mail on my server is:

CiviCRM -> mail() -> Exim -> Gmail -> Recipient.   

My emails pick up the identity of my Gmail account, which is where the reply over-ride is occuring.  I can take a look my Gmail setup to see if there is a way to have it maintain the original reply to that is sent, though this may not work. 

Has anyone had any success in using Gmail as a gateway to send Civimail without having gmail over-ride your headers?

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Demystifying Bounce Detection - help
February 10, 2013, 12:31:12 pm

any specific reason u r not doing:

civicrm -> gmail mail servers (via smtp) -> recipient

why is mail / exim in there

lobo

A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Demystifying Bounce Detection - help
February 11, 2013, 01:39:46 am
Don't use gmail as your smtp server for mass mailing. They cut you after a handful (well, might be a few hundreds?) of mails sent.

Use either a specialised company (eg. civismtp) or exim directy

X+
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

mcgeehon

  • I post occasionally
  • **
  • Posts: 76
  • Karma: 0
  • CiviCRM version: 4.x
  • CMS version: 7.x
  • MySQL version: 5.x
  • PHP version: 5.2.x
Re: Demystifying Bounce Detection - help
February 12, 2013, 02:00:57 pm

At the moment, we are using gmail because we only send out around 50 emails at most a day.  We use Constant Contact for a majority of our mailing needs.  There is no specific reason that we send to Exim before Gmail, however I will look into setting up Exim or Postfix as our smtp server as we try to assume move more of our mailing to Civi.

I will update this thread once I have a chance to test this out.  Thanks.


mcgeehon

  • I post occasionally
  • **
  • Posts: 76
  • Karma: 0
  • CiviCRM version: 4.x
  • CMS version: 7.x
  • MySQL version: 5.x
  • PHP version: 5.2.x
Re: Demystifying Bounce Detection - help
February 19, 2013, 02:30:36 pm

I changed my send process as follows:

CiviMail -> postfix -> recipient

This works great.  I ran into a second problem whereby email from my server was being flagged as SPAM.  I ended up having to create an SPF record, fix my hostname, setup reverse DNS and the rest of the shabang to get unflagged. 

My only remaining issue is that Civibounce does not detect a decent amount of my email rejections as bounces.  I looked at the civicrm_mailing_bounce_pattern list and am not sure where Civi uses that patterns against?  Does Civi test against the header, subject or body of the email?

For example, the following email gets processed as ignored.  I cannot figure out how to get Civi to recognize that it is a bounce:

SUBJECT:  Undeliverable: test email subject

Code: [Select]
Delivery has failed to these recipients or groups:

bob@email.com<mailto:bob@email.com>
The e-mail address you entered couldn't be found. Please check the recipien=
t's e-mail address and try to resend the message. If the problem continues,=
 please contact your helpdesk.







Diagnostic information for administrators:

Generating server: CSIS365.onmicrosoft.com

bob@email.com
#550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ##rfc822;bob@email.com

The email then contains the original header and contents that were sent.  I cannot get Civi to process this as a bounce.  I have added entries of type '6' with the pattern 'Undeliverable' and of pattern 'Delivery has failed to these recipients', however neither appears to match.

Any idea's?




Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Demystifying Bounce Detection - help
February 19, 2013, 03:59:19 pm

Took a quick look at the code and seems like Civi tests mainly against the body. Would be good if you can add some debug statements here:

CRM/Utils/Mail/EmailProcessor.php (search for event_bounce)

and

CRM/Mailing/BAO/BouncePattern.php, function &match (which matches to a bounce pattern)

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Demystifying Bounce Detection - help

This forum was archived on 2017-11-26.