CiviCRM Community Forums (archive)

Hi,

I have limited experience with CiviCRM (only with version 3.0x) and have come to some question about the EmailProcessor and the Bounce Process.

I have read all the wiki and went through all the forum posts I can find regarding to the issue, but some inner workings are still not explained.

My situation:

I have a Drupal 6.x/CiviCRM 3.03 Set up on http://www.mydomainname.com
I use a third party SMTP service (Not CiviSmtp) for sending out emails

For testing emails I use the web hosts SMTP http://mail.mydomainname.com

I have a catchall mail account set up for the return channel

catchall@mydomainname.com

Setting:

Name: mail handler
Server: mail.mydomainname.com
Username: username
Password: password
Localpart:
Email Domain: mydomainname.com
Return-Path:
Protocol: IMAP
Source:
Use SSL? (unchecked)
Default Option: (checked)


I ran /sites/all/modules/civicrm/bin/EmailProcessor.php?username=[username]&pass=[password]&key=[key]

Everything is parsed currently and I get the correct number of bounces from the mailing.

However, it seems the MailProcessor cannot distinguish between 'Hard Bounce' (i.e. Bad Domain name, no such name, etc) and 'Soft Bounce' (away from office message, etc).

All of the bounced email are marked as 'Invalid Email address' which seems to be the default bounce type CiviMail falls back on. The problem with that is all the bounced email from that mailing will be put on hold regardless of the bounce type.

I do not know exactly how the mail processor works but I assume that it looks for information in the email headers and if it is a hard fail, it will send it back to the 'return path', where as if it is a soft bounce (automated reply), it should send it back to the 'reply-to' address and that's how the email processor distinguish between a hard or soft bounce.

How does CiviCRM determine which type of bounce it is? (all bounce type listed in http://wiki.civicrm.org/confluence/display/CRMDOC/Bounce+Handling)

If someone can explain that to me it would be much appreciated!


Cheers!

Take a look at the following tables in your CiviCRM database:
— civicrm_mailing_bounce_type lists the various bounce types with the relevant hold_threshold – i.e., the number of such bounces needed for a given email address to be put on hold;
— civicrm_mailing_bounce_pattern lists regular expression patterns that are matched against the bounce body – if the given bounce matches any of these patterns, the relevant bounce type is assumed (and its hold_threshold is consulted whether to put the email on hold or not).

If you look now at the CRM_Mailing_Event_BAO_Bounce::create() function (in the CRM/Mailing/Event/BAO/Bounce.php file), you will see that if there’s no match, the bounce type of 6 is assumed.

Help with expanding the civicrm_mailing_bounce_pattern table to cover more cases would be most appreciated.

Thanks for the explanation, that is a very smart way of handling different types of bounces. I went back to the failed delivery email again and surely because it did not contain any of the bounce pattern in the database, it was categorized as type 6.

I would be more than happy to help out in adding more bounce pattern and hold_threshold rules.


However, my other question is regarding to the 'return path' and 'reply to' address that civiCRM generates.

For example, a mailing I send out will have

   
b.18.101.44350aeb45d04757@mydomainname.com in the 'return path'

and

r.11.64.b929a2f9ea924247@mydomainname.com in the 'reply to'

in the email header.

I assume these masks contains the mailing id plus a hash of the email address that was sent to or something in the similar fashion.


However, my SMTP server is not able to send the email back to the 'return path' or 'reply to' once the email bounces, they will only be able to send the failure message back to the original sender.

Even if I setup that sender email as a catchall and the email account in CiviCRM, I assume the mail processor will ignore any email comes directly to the sender say to (sender@mydomainname.com)


In any case, I appreciate all of your help!


Cheers!

Thanks for all the timely helps 

I will definitely work on adding more bounce pattern and hold_threshold rules when I get our CiviCRM deployment running on production.

I guess the issue came down to this:

In a usual case a typical civiCRM deployment will use the website host SMTP to send out emails, usually in these cases the hosts SMTP server behaves accordingly (sending failures back to return path, sending auto reply to reply-to email headers). However, as you know, the host SMTP just does not cut it when you have large number of mailings. If an organization needs more SMTP capability they can even use CiviSMTP which I believe is a great plus for the platform.

However, in a way I'm testing to see if CiviCRM's mailing and other capability can be pushed into enterprise level, that's the reason we have an external SMTP service provider that's capable to send up in the millions of emails on a weekly or monthly basis. However, our external SMTP provider rewrites all the return-path of emails to route them to their email server so they can process bounce, generate reports etc.

In any case thank you very much for your help and expect me to pop up more on the forums.

I believe these are great discussions to have that will help the platform grow, I wouldn't even mind to put the inner workings of how the bounce process and return channel works on a wiki once I gain the complete understanding of them.


Thank you,
Cheers!