"Unknown bounce type: Could not parse bounce email"

[Eileen]

Hmm - not sure I can subscribe to this thread without commenting - we are seeing high bounces on some customer but are only in early stages of looking into it (3.1.6 & 3.2.1)

[Piotr Szotkowski]

(Not a solution, but a temporary workaround for the above is the new task that allows you to un-hold such addresses in a batch action.)

Yes, I have installed the aaaMailBounceResend.php, which is a lifesaver; I was surprised this doesn't "ship" with 3.2.1!

Ah, you mean the script from here? (I was talking about the new action avaliable as a search results task from inside CiviCRM’s UI.)

Could when I'm running the cron jobs and/or the Mailer settings have anything to do with it?

No, it seems the problem is with CiviCRM interpreting deferring as a bounce. :|

[Piotr Szotkowski]

Yes developer is correct regarding second log, delivery is simply deferred and was re-attempted, further down the log mail actually got delivered after few hours to [contact]@bravadapartners.com

[…]

I guess bounce response by civicrm was for deferred mail but it was later delivered. This also seem to be true for other email address for example [contact]@hoganlovells.com which was deferred with same error and then later delivered.

That would confirm my suspicion. Can you ask one more question: what is the SMTP code their server returns on deferring? (We will need to handle that response in the _parseResponse() function of the Net_SMTP class.)

[Piotr Szotkowski]

Hmm - not sure I can subscribe to this thread without commenting

Yes, via the ‘notify’ button.

we are seeing high bounces on some customer but are only in early stages of looking into it (3.1.6 & 3.2.1)

Can you check your mailserver’s logs to see the culprit?

[bcobin]

That would confirm my suspicion. Can you ask one more question: what is the SMTP code their server returns on deferring? (We will need to handle that response in the _parseResponse() function of the Net_SMTP class.)

Responding to Piotr:

It looks to me like the error codes are as follows, but I've reached out to the host to confirm:

T=remote_smtp defer (-46): SMTP error from remote mail server after pipelined DATA: host mail.bravadapartnersrtners.com [67.152.226.138]: 451 Requested action aborted: local error in processing

and:

T=remote_smtp defer (-53): retry time not reached for any host

I'll be checking back in to verify this is the information you're looking for - thanks SO much, Piotr!

UPDATE FROM HOST:

SMTP code 451 is normally returned by most servers detecting which mail is sent into queue and tried again later.

According to your logs also exim is sending mail into queue for later delivery whenever it encounter 451

[xavier]

Technical question:

does it mean that the cronjob is waiting on each smtp connection to the each remote server to get the answer or that these error messages comes back as bounces or... ?

Isn't it a big performance hit to do smtp over mail() ? As in in mail() php hands the mail to the local mail server and continues. The mail server then sends whenever they can, do the smtp connections and whatever.

Pro: php doesn't work, Con: you have the error messages that only comes back as mails in the bounces mailbox and go as mostly either mailbox full, non existing email or unreachable server. - The mail servers are even more creative at using different formats to describe the same error as the smtp error messages (that at least have a standard).

[Piotr Szotkowski]

SMTP code 451 is normally returned by most servers detecting which mail is sent into queue and tried again later.

Ok, so the problem is that CiviMail treats 4xx SMTP codes as bounces, while they’re transient errors.

I filed CRM-6718 to track this; I’ll reschedule it to an earlier version if we can come up with an easy solution.

For example: would it make sense to treat 4xx SMTP codes as successes, and assume a ‘real’ bounce will result if the error stops being transient (and will then be handled by EmailProcessor)?

[Piotr Szotkowski]

does it mean that the cronjob is waiting on each smtp connection to the each remote server to get the answer or that these error messages comes back as bounces or... ?

I assume it depends on the local mailserver. You talk SMTP to it, it’s its decision whether to answer you based on hope (here’s 2xx: got the message, will try to deliver sometime later) or ping the addressee upfront and return the code after some hesitation, when it’s a bit more sure whether it’ll deliver the message or not.

If you need performance, you can always write directly to the mailserver’s output queue and assume you’ll deal with any bounces when you get them via email…

[bcobin]

For example: would it make sense to treat 4xx SMTP codes as successes, and assume a ‘real’ bounce will result if the error stops being transient (and will then be handled by EmailProcessor)?

Heavens, yes! I'd say that would be a rather elegant solution - certainly better than waiting for 3.4!

Sorry to mix topics here:

Ah, you mean the script from here? (I was talking about the new action avaliable as a search results task from inside CiviCRM’s UI.)

Where would that be? I can't seem to find it... thanks!

[xavier]

If you need performance, you can always write directly to the mailserver’s output queue and assume you’ll deal with any bounces when you get them via email…

More or less what mail() does (depending on what mail server, bla, but in general, you just inject a new message in a queue, and let the mail server handles it asynchroneously.

X+

[Piotr Szotkowski]

For example: would it make sense to treat 4xx SMTP codes as successes, and assume a ‘real’ bounce will result if the error stops being transient (and will then be handled by EmailProcessor)?

Heavens, yes! I'd say that would be a rather elegant solution - certainly better than waiting for 3.4!

Ok, updated the issue accordingly.

Ah, you mean the script from here? (I was talking about the new action avaliable as a search results task from inside CiviCRM’s UI.)

Where would that be? I can't seem to find it... thanks!

Search for contacts → choose Unhold Emails from the actions drop-down.

[obiuquido144]

Just to clarify for someone who might be new to the thread:
Piotr is talking about a new (3.2+ ?) feature that allows to batch-remove the ON HOLD statuses directly from Civi's UI (in the actions dropdown on the search results page). This previously had to be done by a DB admin using a SQL query in phpmyadmin, or through some custom script that the operator would call.
When the ON HOLD flags are removed, you can then use aaaMailBounceResend to perform the actual 2nd mailing to the undelivered addresses - that's what bcobin is talking about.

The ability to remove the ON HOLD from the UI is very important because (after the initial aaaMailBounceResend setup) the resending can now be done by 'the regular operator guy/girl' instead of by the 'geek/superuser'.



BTW, in our own installation, we additionally modified templates\CRM\Mailing\Page\Event.tpl by adding a link to the aaaMailBounceResend search page to the bottom every mailing report detail page.

(
If you're interested, we added this just above the "Back to Report" link:

Code: [Select]

<br/>
        <div class="">
            <a href="{crmURL p="civicrm/contact/search/custom" q="csid=16&reset=1"}" >&raquo; {ts}Resend to the bounced emails using the aaaMailBounceResend custom search{/ts} - mailing ID is <b>{$mailing_id}</b></a>
        </div>
<br/>
(note the csid might not be 16 in your case)
)
If someone is in the unfortunate situation of needing to do a lot of resending using my custom search, having this link might speed things up a little.
But our experience is that once we resolved our smtp issues by fine-tuning the batch size and the cron call interval, we haven't ever used aaaMailBounceResend again, so the link on the report page is not needed anymore; it now acts more as an indication that resending is possible (...should someone new be operating Civi).

[bcobin]

Thanks for the post and the addition, obiuquido144 - this is helpful.

But it's also important to note that, per earlier posts in this thread, it's probably not a good idea just to resend the emails insofar as (at least in my case), the emails actually ended up being delivered after all. The problem currently, as I understand it, is that the delayed deliveries aren't being reported back to Civi, so the accounts stay on hold, meaning that bounced addresses won't receive subsequent mailings until/unless their status is changed.

Effectively, this means that bounce reports are not terribly useful until this behavior is changed.

I think Piotr's suggestion to treat 4xx SMTP codes (transient errors) as successes is right on the money; addresses should only be put on hold in the case of a confirmed failure, which will happen eventually if the address is bad.

Worst case here is that you end up resending to a bad address, which doesn't really hurt anyone, as opposed to having your mailing list gradually get whittled down. (I have one client whose list of 19,000+ eventually was reduced to about 4,000 before we figured this out.)

I should add here that taking addresses off hold doesn't seem to affect the "user opt out," so it doesn't look like you'd be bothering people who have unsubscribed.

Piotr- please correct me if I'm wrong about anything here - thanks everyone for your help in getting this resolved!

[Piotr Szotkowski]

Piotr- please correct me if I'm wrong about anything here

It feels like your reasoning is quite right.

I attached an initial approach to fixing this to the issue; I still feel a bit queasy with such an unsophisticated/blunt approach, but it might be a good tentative fix for your case.

Some additional qualms come from my reading of RFC 3463 with regards to the 451 code:

4.XXX.XXX Persistent Transient Failure

A persistent transient failure is one in which the message as sent is valid, but persistence of some temporary condition has caused abandonment or delay of attempts to send the message. If this code accompanies a delivery failure report, sending in the future may be successful.

[…]

X.5.1 Invalid command

A mail transaction protocol command was issued which was either out of sequence or unsupported. This is useful only as a permanent error.

I don’t really understand what the RFC means about the 451 combination – is it transient or permanent?

(I still can see that Exim (ab)uses this code in a particular situation and so it might be worthwhile to handle it anyway just for it.)

[bcobin]

Thanks for the mod, Piotr - I made the code change over the weekend on the sites of the two clients I have who are actively using CiviMail. I'll be monitoring the results and will keep you advised.

Thank you SO much for your work on this, Piotr - you're a lifesaver!