CiviMail no longer sending mails after 3.1.3 -> 3.2.3 upgrade

[grahamgilchrist]

Hi guys,
Can anyone help me?

I recently upgraded from 3.1.3 to 3.2.3. I have been experiencing problems with some emails from civiMail not being delivered. The only addresses receiving mails were those on the same domain as the sender (i.e. internal to our organisation). Everyone outside was failing.

After debugging through our live system, which was the only place the problem appeared, I discovered that it was all down the the '-fsender@domain.com' flag being appended to the sendmail command in packages/mail/mailphp. As the PHP docs (http://php.net/manual/en/function.mail.php) say, unless your php user is in the trusted senders group, many mail servers will reject an envelope sender set this way. I have no root access to the server to add or check if the user is trusted, and do not us the VERP/reply checking features, so do not want this flag.
Wondering why it worked on 3.1.3, I discovered this was changed in v3.1.4 (http://issues.civicrm.org/jira/browse/CRM-5946).

Looking at the message code for each message, The flag is being set as '-fb.268.23912.6d5028a4d303ea61@' and also the return path header is set as: 'Return-Path: b.268.23912.6d5028a4d303ea61@'. This happens both when using the mail() and sendmail options in the email config (/civicrm/admin/setting/smtp&reset=1)
I do not have an incoming email server set-up and all the track reply options are disabled in each mailing, so can someone explain why these VERP fields are still appearing in my mails, and how to get rid of them?

Is there any way I can disable this flag, and turn off the VERP features?
Alternatively it would be really useful to have a patch to make this flag optional?

At the moment I have hacked packages/mail/mail.php to remove the fifth parameter when sending, but I don't want to have to maintain this hacked version through upgrades, so any suggestions as to config changes or patches to get this to work for us would be greatly appreciated.

Cheers

Graham

[xavier]

You really really should handle the bounces. That's a sure way of getting blacklisted not to do it.

As for the sendmail, replace with mail() and it should go through.

I'd strongly suggest you to go through that chapter and configure your install properly.
http://en.flossmanuals.net/CiviCRM/SystemConfiguration


X+

[grahamgilchrist]

Thanks for the reply, I really appreciate the help on this.

You really really should handle the bounces. That's a sure way of getting blacklisted not to do it.

....
I'd strongly suggest you to go through that chapter and configure your install properly.
http://en.flossmanuals.net/CiviCRM/SystemConfiguration

We have been through the docs before when setting up for 3.1.  You are right of course, and I am trying to figure a way to do this, the problem being we are stuck on servers provided by our institutional IT and bureaucracy/lack of resources (I guess similar to many non-profits) mean I can't get server settings changed or get a bounce handling email address, or even add an SPF DNS entry. It is a good fast server, but we are just not allowed to change the config settings...
I am going to investigate using Gmail to handle the bounces though...

As for the sendmail, replace with mail() and it should go through.

Anyway, regardless of how we are handling bounces, our specific problem is that the -f flag which is used to set the envelope sender (and is also set when using mail() rather than sendmail) stops all external services receiving our email. I can only presume this is because the web server PHP user is not in the sendmail 'trusted users' group as described in the PHP docs (http://php.net/manual/en/function.mail.php).
In 3.1.3 this flag did not appear to be appended to the mail (or was supposed to have been, but a bug prevented it), and everything worked fine. In 3.2, it is added for every mail sending type, and we can no longer send mails outside our own email domain

Again, we have no control over this (and I imagine neither do many others even on quality managed hosting), so I was hoping there might be a possibility of adding this as an option in the civiMail settings?

Otherwise, I think my only option here is to maintain a forked version of packages/mail/mail.php with these settings removed?

Unless anyone has any other suggestions for stuff I've missed?

[FatherShawn]

I think far and away the best option is to use SMTP.  Will your institution give you an email address to send mail from?  You can then just setup civi as an smtp client.

You can definitely use gmail for the bounce handling.  They support "plussed" emails.

[grahamgilchrist]

Hi,

Cheers for the suggestion. That had occurred to me yesterday too, although I haven't had time to test it yet. I found out our server system runs postfix instead of sendmail, although there is supposed to be a compatibility layer which supports the -f flag, something doesn't work somewhere....

Hopefully, like you suggest, SMTP should remove the problem, as there is no local program to run and all the VERP headers can be passed in a standard way?

[xavier]

Cheers for the suggestion. That had occurred to me yesterday too, although I haven't had time to test it yet. I found out our server system runs postfix instead of sendmail, although there is supposed to be a compatibility layer which supports the -f flag, something doesn't work somewhere....

Hopefully, like you suggest, SMTP should remove the problem, as there is no local program to run and all the VERP headers can be passed in a standard way?

Can confirm that the -f works with postfix... and his "standard", that's your security settings that is a bit non standard   If you get a shell access, you might run the cron as a user that is in the list of postfix... or simply get rid of this limitation ? kind of standard to be able to send emails as someone else than www-data@yourdomain

Anyway, let us know if the smtp workaround works around

X+

[grahamgilchrist]

SMTP does indeed work so thanks for the suggestions everyone

After some discussion with the server admins, I've managed to get the -f flag working through postfix now too. It seemed to be a combination of this and something to do with the email server config since I noticed that PHP was always reporting the emails being sent fine. I think the -f flag was operating ok, but the e-mail server was not interpreting the envelope sender correctly.

A wierd side issue I noticed while testing was that if your VERP/civiMail account address is not well formed, then the email is passed to the server fine, but is not then relayed. I.e. if you make a mistake by having a mis-formed envelope return address without the domain or user parts, or miss an @, the receiving email servers seem to reject the message. Worth making sure this is right before sending things out. That might help someone with a similar issue.

Anyhoo, something along the way was broken and is now fixed. I also got bounce handling set up right using Gmail which was pretty easy and would recommend this to anyone else who needs it.

To use Gmail as a civiMail account (/civicrm/admin/mailSettings), the settings are:

Server: pop.gmail.com
Username: gmailusername@gmail.com
Password: gmail password
Localpart: gmailusername+
Email Domain: gmail.com   
Return-Path: leave this blank
Protocol: POP3
Source: Leave blank
Use SSL: Ticked

Thanks everyone!

[xavier]

Hey, nice recap, thanks.

Pro tip: you will have to add some filters on your gmail so bounces don't end up in the spam folder. After the first few mailings, go to this gmail account, check what is (wrongly) in the spam folder, add a filter rule that whitelist it, repeat and rince.

If someone does that and can share the filters generated, would be great

X+

[grahamgilchrist]

If someone does that and can share the filters generated, would be great

Actually I just did this this morning

I set up three filters, all matching the from field:

• Matches: from:postmaster
  Do this: Never send it to Spam
• Matches: from:(Mail Delivery Subsystem)
  Do this: Never send it to Spam
• Matches: from:(MAILER-DAEMON)
  Do this: Never send it to Spam

Those have caught everything so far...

[FatherShawn]

I match on civicrm OR the user-id on the gmail address as in GMAIL-USER-PART@gmail.com

Matches: (civicrm OR GMAIL-USER-PART)
Do this: Never send it to Spam

No trouble with bounce processing via IMAP at all with this setup.