"Unknown bounce type: Could not parse bounce email"

[Piotr Szotkowski]

Thanks, Piotr - I sent out a 46,000+ email on Tuesday, about 20% were bounces, most of which are "unknown bounce type." All bounced email contacts were put on hold.

Note that with 3.2.4 the bounce reason for previously-known CiviCRM-side problems (like wrong address formatting) is different, so this means we really couldn’t match the bounce reason. Checking (a) the SMTP logs (I’m aware you don’t want to bug your patient hosting provider a lot, and fully understand) and (b) the bounce inbox (might be worth taking a look!) is the only way to see why these addresses are bouncing.

More distressing is that it looks like some emails reported to have been successfully delivered were not received - this was using SMTP.

Once the SMTP server accepts the email as ‘deliverable’ (on SMTP transaction) CiviCRM assumes it’s delivered; it’s assumed to be delivered until the email is later bounced *and* the bounce reaches the bounce inbox *and* the bounce inbox is processed. There might be cases when, say, the receiving end accepts the email, but then silently bins it as spam, or can’t – for some reason – deliver the bounce; in these cases there’s no way for CiviCRM to know the email wasn’t received.

What is the percentage of such non-received ‘successes’?

[Piotr Szotkowski]

I'm still waiting on confirmation that the emails in question weren't received, but in the meantime, I've just sent an email to one of my clients, which I've excerpted below. Please tell me if this is an accurate description of what's going on (or what's not going on!) - thanks!

Very accurate; we don’t have any ‘known’ problems with CiviMail delivery in 3.2.4+, so the only way to figure out the ‘unknown’ bounces is to find out what the actual bounce reason was (from logs/bounce inbox).

[bcobin]

Very accurate; we don’t have any ‘known’ problems with CiviMail delivery in 3.2.4+, so the only way to figure out the ‘unknown’ bounces is to find out what the actual bounce reason was (from logs/bounce inbox ).

Thank you, Piotr - I can read the bounce messages from the sites/default/files/civicrm/civicrm/custom/CiviMail.processed/[year]/[month]/[day]/cur directory - it seems that the thing to do would be to find failed addresses in the bounce report that correspond to "unknown bounce type". We would then be able to correlate the bounce message with the 'unknown' error message, which would give us some answers, right?

I'll be able to do that as soon as http://forum.civicrm.org/index.php?topic=16397 is fixed - with something like 11,000 bounces (out of 46,000+) in the last email that was sent, it's not practical to use "Next" to get to the email address in question in the bounce report. I should be able to give you a lot of good info here once the pager is fixed.

There was also the issue with email addresses being improperly put on hold, even when they were subsequently delivered. I'm now using mail() instead of SMTP, so perhaps that problem has gone away? What do you think?

Regardless, I suppose the thing to do for now would be to dig as deep as possible into the 'unknown' issue and then one could more intelligently decide what to do with bounces, short of unholding all "On Hold" addresses after each email.

Thanks, as always, for all the work you're doing here!

[Piotr Szotkowski]

(Thanks for you patience, again.)

Thank you, Piotr - I can read the bounce messages from the sites/default/files/civicrm/civicrm/custom/CiviMail.processed/[year]/[month]/[day]/cur directory - it seems that the thing to do would be to find failed addresses in the bounce report that correspond to "unknown bounce type". We would then be able to correlate the bounce message with the 'unknown' error message, which would give us some answers, right?

Exactly.

I'll be able to do that as soon as http://forum.civicrm.org/index.php?topic=16397 is fixed

I can’t reproduce this; the pages works for me (in 3.3 at least).

I'm now using mail() instead of SMTP, so perhaps that problem has gone away? What do you think?

Well, IIRC mail() is a front-end to sendmail, so it’s a different delivery method; maybe your server is configured to be a bit more tolerant for emails coming from sendmail (rather than directly received on the SMTP interface).

[bcobin]

Now that the pager is fixed in 3.3 beta2, I was able to correlate the bounce messages (from the sites/default/files/civicrm/custom/CiviMail.processed folder) with the "unknown bounce type" for the contact.

Below is just the first of many (the names have been changed to protect the innocent) - this is for an AOL account.

There are tons of these (it was a mailing to about 45,000) - any ideas? Thanks!

Code: [Select]

Return-path: <>
Envelope-to: b.46.261.a3ceaacda7da9f59@[site].com
Delivery-date: Tue, 02 Nov 2010 04:08:43 -0500
Received: from mailnull by [account].[site].com with local (Exim 4.69)
id 1PDCrC-00011l-S7
for b.46.261.a3ceaacda7da9f59@[site].com; Tue, 02 Nov 2010 04:08:42 -0500
X-Failed-Recipients: [contact]@aol.com
Auto-Submitted: auto-replied
From: Mail Delivery System <Mailer-Daemon@[account].[site].com>
To: b.46.261.a3ceaacda7da9f59@[site].com
Subject: Mail delivery failed: returning message to sender
Message-Id: <E1PDCrC-00011l-S7@[account].[site].com>
Date: Tue, 02 Nov 2010 04:08:42 -0500

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  [contact]@aol.com
    SMTP error from remote mail server after RCPT TO:<[contact]@aol.com>:
    host mailin-01.mx.aol.com [205.188.146.193]: 550 5.1.1 <[contact]@aol.com>:
    Recipient address rejected: aol.com

------ This is a copy of the message, including all the headers. ------

Return-path: <b.46.261.a3ceaacda7da9f59@[site].com>
Received: from 174.36.252.231-static.reverse.softlayer.com ([174.36.252.231] helo=[site].com)
by [account].[site].com with esmtpsa (TLSv1:AES256-SHA:256)
(Exim 4.69)
(envelope-from <b.46.261.a3ceaacda7da9f59@[site].com>)
id 1PDCrB-000110-JE
for [contact]@aol.com; Tue, 02 Nov 2010 04:08:41 -0500
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="=_afa341b832606e00cca3c59eec7a7e38"
Reply-To: "[Site Name]" <info@[site].com>
From: "[Site Name]" <info@[site].com>
Subject: [Subject]
List-Unsubscribe: <mailto:u.46.261.a3ceaacda7da9f59@[site].com>
To: [Contact] <[contact]@aol.com>
Precedence: bulk
X-CiviMail-Bounce: b.46.261.a3ceaacda7da9f59@[site].com
Date: Tue, 02 Nov 2010 05:08:41 -0400

--=_afa341b832606e00cca3c59eec7a7e38
Content-Transfer-Encoding: 8bit
Content-Type: text/plain;
 charset=utf-8

[Piotr Szotkowski]

The way to cater for emails that are bounced because of the rate at which AOL/Yahoo receives them is to adjust the sending rate of your mailings – instead of, say, a twelve-thousand blast sent as quickly as possible, setup CiviMail to send five hundred every half an hour.

The main question is whether these emails they’re actually delayed or bounced; if they’re not delayed, they won’t reach the recipient (albeit might reach them on next mailing).

The way we deal with this in CiviCRM is that we have different bounce counts that put a given address on hold depending on the cause of the bounce. Take a look at the contents of the civicrm_mailing_bounce_type and civicrm_mailing_bounce_pattern tables – the first one says how many bounces are required to put a contact on hold for a given type of bounce, the second says which bounce patterns mean which bounce.

If you could figure out what does a bounce pattern from AOL and Yahoo look like and could add that to civicrm_mailing_bounce_pattern (referencing the relevant bounce type), you can set the bounce threshold to whatever makes sense; we’d also happily include such information in the next CiviCRM versions.

If this got delayed even though a bounce email was sent, then it wasn’t a bounce email but rather a delayed notification email; IMHO these should be mostly ignored, because they mean the delivery will be tried again (and if it eventually fails, a ‘real’ bounce will get send anyway).

[Piotr Szotkowski]

Code: [Select]

SMTP error from remote mail server after RCPT TO:<[contact]@aol.com>:
host mailin-01.mx.aol.com [205.188.146.193]: 550 5.1.1 <[contact]@aol.com>:
Recipient address rejected: aol.com


After reading this thread the remaining question is whether it’s AOL that sends bounces when it temporarily blacklists an IP number, or is the above actually a bounce (in which case it won’t be delivered, but there’s no sign of the fact that it’s temporarily IP blacklisting – it says the email address that you tried to reach doesn’t exist).

[Piotr Szotkowski]

Now that I think about this, it might be that what you’re seeing is that they’re actual bounces, but they’re being registered as both delivered and bounced.

This might be because we register them as delivered as soon as they’re accepted for delivery by your local SMTP server, and only later, when processing bounce emails, we know they in fact bounced.

If this is the case, then we should make sure that we mark them as not delivered when a bounce happens (we might not be doing this at the moment, which would be a bug).

[DougAnderson]

I just sent my first 'successful' bulk mailing with Civimail. However I have 53 bounces vs 353 successful deliveries. The message in each case is "Syntax - Unknown bounce type: Could not parse bounce email"
The ones that bounced seem no different from those that went through and certainly most of them are ones to which I have sent successful emails to within the last few weeks.
Is there any explanation, anything I should be doing differently?

[Donald Lobo]

what smtp server r u using? is it rate limited?

lobo

[DougAnderson]

I'm not sure about your questions - haven't been able to get the answers. I'm not even sure I'm looking in the right places for the answers.

BTW, I was using the "http://durhamclear.ca/index.php?q=civicrm/mailing/queue&reset=1" command url to initiate the mailing.

However, after my initial question I resubmitted the 53 bounces in a new Civimail mailing and 51 delivered successfully. The remaining 2, I submitted to a 3rd mailing and both were delivered successfully.

Inconvenient, but successful. Will those that bounced be removed from subsequent mailings automatically? If so how do I remove the 'tag' that designates that it bounced?

Doug