"Unknown bounce type: Could not parse bounce email"

[Piotr Szotkowski]

1. This case I described did not happen to us in previous cases, this was the only time. (Very isolated) As matter of fact, I did another mailing this afternoon with the same 50,000 emails and they were all injected fine.

Ok, then it’s not the case; the bugs fixed on the 3.1 branch would manifest themselves every time (i.e., they were 100% reproducible for a given set of contact names).

It makes me wonder if it had anything to do with MySQL connection issues (MySQL server gone away or etc) or script time out issues. However, those were not shown in my php logs. Also we had sent 5 or 6 mailings to this email groups before (sent successfully) so I would think that our SMTP server does not dislike the CiviCRM email headers and the mailing logic were fine?

If you had a successful delivery with the same group and the same CiviCRM version then it would suggest that there were some non-CiviCRM sources of the problems – MySQL connection timeouts, PHP script execution timeouts, SMTP/network issues, etc. If PHP and SMTP logs don’t show anything, then it’s a bit hard to diagnose; you can try checking whether the problem persists when you send slower and in smaller batches (maybe some target servers throttle incoming emails after a large blast?).

Can what you described happen on some occasions but not others?

No, not the issues fixed on the 3.1 branch.

2. It would be a nice feature if CiviCRM can create some kind of log of those error emails due to the PEAR mailing library or even just create some sort of action available in the mailing report so you can add all the unknown bounce type error into a group, remove their hold and resend the emails?

That’s a good idea! Can you please file an issue for this? (Also, as Lobo says, a code contribution would be most welcome, of course.)

What are some other options we can use to debug/trace this error?

You can try comparing the bounce list with your SMTP logs to see which emails did not even get to the mail server; that would suggest CiviCRM’s fault.

i will update to CiviCRM 3.15 and let you know if this continue to persist.

Thanks!

[xcf33]

Hi Piotr,

Thanks for all the update, it's hard to diagnose because those email did not show up in the SMTP server, I will file an issue for this. It could have potentially been a time out issue but it would be hard to diagnose because the mailing cron is being run by shell_exec with CURL and we don't have access to the apache or CURL related logs.

[bcobin]

I'm dealing with the same issue - many bounced emails with the "unable to parse" message. Just to clarify: is it expected that 3.15 solves the issue or is there something else that needs to be done? Thank you for dealing with the issue!

[xavier]

Hi,

What do you mean by many ? How many bounce did you get and on a mailing and how many were unknown ?

You might need to do a bit of detective work and see what is the text in the bounce email, so CiviCRM can add contain a new rule to handle your specific format of text in the bounce email.

Unfortunately, every single mail server seems to have a different way of saying the same things, so they are a lot of opportunities to no know the bounce type.

@shot, is the way off testing and adding a new parser regex documented somewhere ?

[bcobin]

Thanks, Xavier - I'm looking at a mailing where 7608 out of about 13,000 were bounced - looking at the first report page, I'd say about 90% were "Invalid" and "Unknown bounce type: could not parse bounce email."

A lot of the addresses seem to be large-volume providers like AOL, MSN and Yahoo. And pretty much all of the recipients have names (meaning not just mail addresses.)

I discovered the post on aaaMailBounceResend and Custom Searches, so I can reactivate the accounts that are "on hold" and, I suppose, manually deactivate the Opt-out/unsubscribe requests; my question relates to whether or not it's expected that 3.15 solves the original problem.

Thanks for the quick response!

[bcobin]

Upgraded to 3.1.5 on client site and when I check the Drupal Status Report, I now get:

Fatal error: Cannot redeclare class pear in /home/[site]/public_html/sites/all/modules/civicrm/packages/PEAR.php on line 87

Mail isn't being sent. [Correction - mail was successfully sent, it seems, but Civi is one hour off time-wise.]

5,000+ email completed, few bounces and much better reporting; only a few "could not parse bounce email." Good!

But the fatal error is a show-stopper - being unable to check the site status report is very bad.

Oh, heavens... any ideas here? Thanks!

[Piotr Szotkowski]

@shot, is the way of testing and adding a new parser regex documented somewhere ?

I don’t think so; it’s basically ‘add a new entry to the relevant table(s) and see if it works’. Docs welcome, of course.

[Piotr Szotkowski]

Upgraded to 3.1.5 on client site and when I check the Drupal Status Report, I now get:

Fatal error: Cannot redeclare class pear in /home/[site]/public_html/sites/all/modules/civicrm/packages/PEAR.php on line 87

This means that PEAR is included more than once somewhere – are you sure that (a) your PHP stack does not include a system version of PEAR by itself and (b) there isn’t a copy of the previous CiviCRM install left over somewhere in the same Drupal install?

[bcobin]

Thank you, Piotr - it was a conflict with another module and I found it - solution posted at http://forum.civicrm.org/index.php?topic=14372

This probably belongs in another thread, but I'm also using the FixedMailMIME.php file from http://forum.civicrm.org/index.php/topic,10148.0.html, replacing the 3.1.5 file.

Bottom line: it looks like mass email and bounce/error checking is working nicely!

Could you confirm that using the alternate file is a good idea? If so, I'd say we've hit upon a solid - albeit somewhat complicated - CiviMail solution!

Thanks much , as always...

[Piotr Szotkowski]

Thank you, Piotr - it was a conflict with another module and I found it

Great!

This probably belongs in another thread, but I'm also using the FixedMailMIME.php file from http://forum.civicrm.org/index.php/topic,10148.0.html, replacing the 3.1.5 file.

You mean the http://svn.civicrm.org/civicrm/branches/v3.1/CRM/Utils/Mail/FixedMailMIME.php file that Kiran mentions? If so, you might be interested that we actually shipped it with 3.1.5, so it shouldn’t differ with what you overwrote.

Could you confirm that using the alternate file is a good idea?

Is it really alternate?

If so, I'd say we've hit upon a solid - albeit somewhat complicated - CiviMail solution!

You’d be delighted to know that thanks to the upgraded PEAR libraries we finally got away with FixedMailMIME altogether in 3.2 (the first such-fixed version will be 3.2.beta3).

[xavier]

basically ‘add a new entry to the relevant table(s) and see if it works’.

On the next book sprint, I'd vote you in, the chapter on how to configure civimail is going to be a delight

X+

[bcobin]

Using Civi 3.2.0 on this installation.

I'm still having big problems with this; a recent mailing a client sent out reported about 10% bounces to addresses they know are valid. As we know, once a bounce is reported, the address gets put "on hold," so this becomes a major issue as valid client mailing lists gradually get whittled away.

I had my (very helpful) hosting provide look into this and this is what he sent - I'm hoping this will help shed further light on what clearly still is a problem. E-mail addresses have been changed to protect the innocent.

Could this be an issue with mailer settings? Any help greatly appreciated!

-----From hosting provider----

We have investigated Bounces in your script and it appear that many are false positive, email address for which Civicrm is reporting unknown bounce type actually seem to be delivered fine.
For example its showing unknown bounce for [contact]@gmail.com but according to mail server logs mail was delivered with no error.
2010-08-18 20:34:16 1Olu1H-00048A-CF => [contact]@gmail.com R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [74.125.93.27]

Some rejection are correct which are reported in main exim logs, i was checking only reject logs before. For example [contact]@bravadapartners.com is showing bounce in civicrm but the correct error in exim main log is
[contact]@bravadapartners.com
[contact]@bravadapartners.com R=lookuphost T=remote_smtp defer (-46): SMTP error from remote mail server after pipelined DATA: host mail.bravadapartnersrtners.com [67.152.226.138]: 451 Requested action aborted: local error in processing

This is an error from remote server only and there is no problem from your VPS but still civicrm did not show the correct rejection message.

I have made exim settings more conservative, Please try again to send mail to some invalid email address and see if you get correct bounce message in civicrm.

----End message---

[Piotr Szotkowski]

As we know, once a bounce is reported, the address gets put "on hold," so this becomes a major issue as valid client mailing lists gradually get whittled away.

(Not a solution, but a temporary workaround for the above is the new task that allows you to un-hold such addresses in a batch action.)

For example its showing unknown bounce for [contact]@gmail.com but according to mail server logs mail was delivered with no error.
2010-08-18 20:34:16 1Olu1H-00048A-CF => [contact]@gmail.com R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [74.125.93.27]

Uh-oh. What sending method are you using in CiviCRM (PHP’s mail(), SMTP or sendmail)?

Is there a chance that the bounce recorded in CiviCRM was from a latter attempt? (I don’t think so, but asking just in case…)

For example [contact]@bravadapartners.com is showing bounce in civicrm but the correct error in exim main log is
[contact]@bravadapartners.com
[contact]@bravadapartners.com R=lookuphost T=remote_smtp defer (-46): SMTP error from remote mail server after pipelined DATA: host mail.bravadapartnersrtners.com [67.152.226.138]: 451 Requested action aborted: local error in processing

If this means that the delivery is simply deferred and will be re-attempted (while CiviCRM recorded this as a bounce), then either CiviCRM did not interpret the local server’s response properly or it wasn’t communicated properly (the latter depends on the sending method, see my question above). Did Exim in the above case properly communicate that the delivery is only deferred (probably yes, but wanted to clarify)?

[bcobin]

Thank you, Piotr, for responding so quickly (and thoroughly, as usual!)

(Not a solution, but a temporary workaround for the above is the new task that allows you to un-hold such addresses in a batch action.)

Yes, I have installed the aaaMailBounceResend.php, which is a lifesaver; I was surprised this doesn't "ship" with 3.2.1!

Uh-oh. What sending method are you using in CiviCRM (PHP’s mail(), SMTP or sendmail)?

I'm using SMTP.

Is there a chance that the bounce recorded in CiviCRM was from a latter attempt? (I don’t think so, but asking just in case…)

I don't have easy direct access to the Exim logs and the hosting provider (Imageleet) has been very helpful in this area; I will be forwarding this thread to the kind folks over there in the hopes they can get to the bottom of this.

If this means that the delivery is simply deferred and will be re-attempted (while CiviCRM recorded this as a bounce), then either CiviCRM did not interpret the local server’s response properly or it wasn’t communicated properly (the latter depends on the sending method, see my question above). Did Exim in the above case properly communicate that the delivery is only deferred (probably yes, but wanted to clarify)?

Again, I'll be forwarding this over to Imageleet and they should be able to help.

Could when I'm running the cron jobs and/or the Mailer settings have anything to do with it? (Mailer settings are default with Mailer Spool Period 180 seconds.) I'm running civimail.cronjob.php every five minutes and running CiviMailProcessor.php at 11 and 41 minutes after the hour.

Thanks again, Piotr - I'll get back to you as soon as I have the info from the host.

[bcobin]

Piotr,

Below is the response from the host - note that the [contact] I entered was an edit by me so as not to publicly post any email addresses here, so it may be the same case with in the first instance as in the second.

I'm no expert here (you da man!), but it seems that what could be happening here is that when the initial attempt at delivery is unsuccessful, the account is immediately put "on hold," and not taken back off hold when the email is subsequently delivered.

This sounds like it may be a bit of a mess to fix - I hope I'm not ruining your day here. Hopefully this will help!

Thanks again, Piotr... standing by...

---Response from host (with edits for email privacy)----- 

According to exim logs there was no latter attempt for [contact]@gmail.com
2010-08-18 20:34:16 1Olu1H-00048A-CF => [contact]@gmail.com R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [74.125.93.27]

Yes developer is correct regarding second log, delivery is simply deferred and was re-attempted, further down the log mail actually got delivered after few hours to [contact]@bravadapartners.com
2010-08-18 21:49:03 1Olu1H-00048g-Nv => [contact]@bravadapartners.com R=lookuphost T=remote_smtp H=mail.bravadapartners.com [67.152.226.138]
2010-08-18 21:49:03 1Olu1H-00048g-Nv Completed

here is the log before it was delivered in retry
2010-08-18 20:46:25 1Olu1H-00048g-Nv == [contact]@bravadapartners.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host

I guess bounce response by civicrm was for deferred mail but it was later delivered. This also seem to be true for other email address for example [contact]@hoganlovells.com which was deferred with same error and then later delivered.