CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Overriding VERP - not working ?
Pages: [1] 2

Author Topic: Overriding VERP - not working ?  (Read 4853 times)

Denver Dave

  • Ask me questions
  • ****
  • Posts: 471
  • Karma: 9
Overriding VERP - not working ?
February 22, 2010, 12:00:47 pm
Overriding VERP - not working?

I'm checking the box to Override VERP in step 3 of CiviMail, yet the return address indicated in the reject log still says:
<myemailusername>+b.57.86.09111b0c7f406d48@<mydomain>

I thought that checking the box would allow the email address to be used as the return address?   Am I not understanding how the override VERP is supposed to work?

Thanks.  Dave
« Last Edit: February 22, 2010, 01:13:25 pm by Denver Dave »

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Overriding VERP - not working ?
February 24, 2010, 03:51:17 am
It seems that it's that way if you don't override the VERP. Doesn't seem logical IMO.

Could someone confirm that:

Without overriding the VERP:
1) The envelope is the VERP one
2) The reply to is VERP

With the VERP
1) The envelope is the VERP
2) The reply to is the from


Suggestion

Without overriding the VERP:
1) The envelope is the VERP one
2) The reply to is the from

With the VERP
1) The envelope is the from
2) The reply to is the from
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Piotr Szotkowski

  • Moderator
  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Overriding VERP - not working ?
February 24, 2010, 05:01:43 am
Quote from: Denver Dave on February 22, 2010, 12:00:47 pm
Overriding VERP - not working?

I'm checking the box to Override VERP in step 3 of CiviMail, yet the return address indicated in the reject log still says:
<myemailusername>+b.57.86.09111b0c7f406d48@<mydomain>

I thought that checking the box would allow the email address to be used as the return address?   Am I not understanding how the override VERP is supposed to work?

Ok, let me clarify this a bit. :)

There are two VERP-like addresses that we use in this scenario – one is the bounce address (which lets CiviMail track hard and soft bounces), the other is the reply address.

When CiviMail sends a mailing, it sets the SMTP envelope’s FROM address to the bounce VERP address – so that if there’s a hard bounce on the SMTP level, CiviMail knows who was bouncing. The SMTP envelope’s FROM address is also put in the email’s Return-Path header, and this is the address the soft bounces go to (when the SMTP transaction succeeded, but some server down the road had problems – like the mailbox being over quota or the target server being temporarily offline).

We use another VERP-like address to track bounces – this is the address that the Reply-To header is set to, and this lets CiviMail track (and forward, if so checked on the first mailing step) replies to a given mailing. The ‘Override VERP?’ checkbox is only for this address – if it’s checked, Reply-To will be set to the From email address and CiviMail won’t be able to track replies. It has nothing to do with the bounce VERP address, the SMTP envelope’s FROM address nor the Return-Path header.

Your server logs tell about the Return-Path header, not the Reply-To header that ‘Override VERP?’ concerns.
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

Piotr Szotkowski

  • Moderator
  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Overriding VERP - not working ?
February 24, 2010, 05:04:14 am
Quote from: xavier on February 24, 2010, 03:51:17 am
Without overriding the VERP:
1) The envelope is the VERP one
2) The reply to is VERP

The SMTP envelope FROM and Return-Path is the bounce VERP one, the Reply-To is the reply VERP one.

Quote
With the VERP
1) The envelope is the VERP
2) The reply to is the from

The SMTP envelope FROM and the Return-Path is the bounce VERP one, the Reply-To is the From.

Quote
Without overriding the VERP:
1) The envelope is the VERP one
2) The reply to is the from

This doesn’t sound like not overriding the (reply) VERP, it sounds like overriding the reply VERP…
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Overriding VERP - not working ?
February 24, 2010, 06:04:19 am
Ouch too many options. Let me try to take it from another angle.

In the doc it says that you don't have to setup a return channel, but as the return path and envelope  are always set, it means that the bounce are going to r.xxx@fixme.org (the default mailbox) ?

The message on the help is
"If this box is checked, the automatically generated VERP address in the Reply-To will be overriden by the From email address specified above. Consult with your site administrator before using this feature as it may affect handling of bounced emails and replies."

But checking it has no impact on the bounced emails, as civimail  will always use a VERP and go to the default mailbox, right ?

I think it would be useful to have an option to use civimail without return channel that works, ie.
* if the mailbox is fixme, put the from in the envelope and return-path.
* Override is checked by default (actually can be hidden)

Otherwise, I'm confused, in my postfix log I see the sender email address, not the VERP. Am I missing something ?



-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Piotr Szotkowski

  • Moderator
  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Overriding VERP - not working ?
February 24, 2010, 07:47:55 am
Quote from: xavier on February 24, 2010, 06:04:19 am
In the doc it says that you don't have to setup a return channel, but as the return path and envelope  are always set, it means that the bounce are going to r.xxx@fixme.org (the default mailbox) ?

Right. The reasoning was that if you don’t want to process the bounces, you don’t care about them (but I can understand the case where you want to process them by hand).

Quote
The message on the help is
"If this box is checked, the automatically generated VERP address in the Reply-To will be overriden by the From email address specified above. Consult with your site administrator before using this feature as it may affect handling of bounced emails and replies."

But checking it has no impact on the bounced emails, as civimail  will always use a VERP and go to the default mailbox, right ?

Right, the message is misleading. Fixed in r26394. (Technically, I can foresee a SMTP server that behaves differently based on Reply-To, but I doubt it’s a common setup.)

Quote
I think it would be useful to have an option to use civimail without return channel that works, ie.
* if the mailbox is fixme, put the from in the envelope and return-path.
* Override is checked by default (actually can be hidden)

As per IRC chat, I’ll think this through and propose something that will hopefully address your case and make sense for the current users (and, ideally, not introduce a situation in which mailing X works and a subsequent mailing Y does not for mysterious reasons [because the From changed]).
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: Overriding VERP - not working ?
July 08, 2010, 04:31:00 am
just ran into this issue, and agree with Denver Dave that the existing behavior is misleading (and incomplete).

we need a way to allow people to completely disable any VERP behaviors -- whether on the reply or return path. in one case I have, the organization hosts email through exchange and doesn't make use of any automated message processing actions. it appears that the VERP-created return-path may be causing a large number of emails to be flagged as spam on the receiving end, in addition to causing bounces to be lost (as they don't land in a real mailbox).
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Overriding VERP - not working ?
July 08, 2010, 06:21:06 am
Hi,

1) You need VERP to properly handle bounces
I've added a header that contains the same information than on the VERP, I quickly looked at the bounces and some don't have this information.

2) You can have the bounce email hosted elsewhere than in xchange. Sucks ? yes, but good workaround: create a yourorg.bounce@gmail, go on the mailbox, add filters so bounced emails aren't identified as bounces, and that's it, it works.

As for 1) if someone can come with the code to use the header instead of the envelope, it might work good enough in most of the cases (that's what we did on PHPList) and non VERP might be an option. But that will still won't be as good as VERP.

X+



-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: Overriding VERP - not working ?
July 08, 2010, 08:05:13 am
hey X+
I know we had this conversation elsewhere in the forums recently, but I think we need to have the ability to completely shut down VERP if desired. Bottom line is that some orgs don't want to handle bounces this way, and the current config forces the issue, with the only option to hack code if you want to disable.
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Overriding VERP - not working ?
July 08, 2010, 09:38:04 am

i dont think i agree with this (i did not agree on the previous thread either)

allowing folks to use civimail as a potential spamming engine and not follow "mass email" best practices does not seem the right thing to do

if an org cant handle the civimail requirements internally, use something like civismtp

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Overriding VERP - not working ?
July 08, 2010, 09:56:20 am
Quote from: lcdweb on July 08, 2010, 08:05:13 am
hey X+
Bottom line is that some orgs don't want to handle bounces this way.

How do they want to handle it ?
1) The don't want to handle it. That's not good for their karma, and a sure way to get the server flagged as a spammer.

Just don't do it.

2) By using something else than VERP. The only other method I know is to put an extra X-verp type=xxx of header, and when you receive the bounce, you check for that header, so you can match the sender.

This sometimes works, but quite a few proxy/mail servers delete these extra headers when they send a bounce.

You can try reading the bounces, you'll see that quite a few don't have the needed header info, so you don't know who bounced.
Try on one of your bounce mailboxes, write a program that read the emails and tell us what % have the header.

3) Any other method I've tried sucks even more.

-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: Overriding VERP - not working ?
July 08, 2010, 10:02:10 am
how would overriding verp expose the org as a potential spammer? (i'm not familiar enough with how verp works)

so a very simple use case is a small organization with a small list of contacts (let's say < 500) who simply wants all bounces returned to their primary mailbox for manual processing. on average they only get 1-5 bounces, because their size is small and they know their constituents pretty well. the workflow is better to receive the bounces via email rather than have it processed in civimail, as they follow up with any bounces to try to identify the correct email.

why would they be at risk of being regarded as a spammer? if that's a factor, i definitely want to understand why.
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

Piotr Szotkowski

  • Moderator
  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Overriding VERP - not working ?
July 09, 2010, 04:33:15 am
Quote from: lcdweb on July 08, 2010, 10:02:10 am
so a very simple use case is a small organization with a small list of contacts (let's say < 500) who simply wants all bounces returned to their primary mailbox for manual processing.

If their mailbox is ngo@example.org then make the bounces use the ngo+ localpart; this way the bounces will land in their inbox (most servers deliver ngo+b.1.2.…@example.org-bound mail to ngo@example.org by default, the rest should be configurable to do so; if not, they can use Gmail as a proxy for this).
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Overriding VERP - not working ?
July 09, 2010, 06:22:00 am
Hi,

Didn't think about handling manually the bounces in the list.

As for piotr's suggestion, works fine in almost all the mail servers but one. Hint: it's a product of a company that start with Micro and finishes soft ;)

X+

-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: Overriding VERP - not working ?
July 09, 2010, 06:44:23 am
...which naturally is what they are using (the client that prompted this discussion).

i'm still curious -- if verp is completely disabled and bounces are processed manually, how does that risk exposing the site as a perceived spammer?
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

Pages: [1] 2
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Overriding VERP - not working ?

This forum was archived on 2017-11-26.