CiviCRM Community Forums (archive)

Hi.

I am caring for a server running a Drupal 6 / CiviCRM 3.1 and CiviMail installation. This install sends mailings to 4,000-6,000 contacts/week, and this has been growing steadily lately. We need to have more statistics on bounces, hence our interest in the return channel configuration.

This civicrm install appears to have been in use since 1.x releases. At that time it is clear that there was an attempt to use the amavisd and postfix for the return channel, as I can see the evolution of the commented out regex in the postfix virtual file. It also appears that that system never was fully configured, as civi has never reported any bounces from mailings on this server. It does however, place addresses on hold. Currently there are over 21,000 "unprocessed" bounce messages in the /var/spool/mail file  (mbox) for the civimail user account. I can authenticate on this account and fetch all of this mail via Thunderbird from the (old) dovecot 1.0.13 IMAP server. I can also create single depth folders in this IMAP account with Thunderbird.

When attempting to process this mailbox via dovecot-IMAP, with both CiviMailProcessor.php and/or EmailProcessor.php I receive the same php error message that indicates the script cannot create the necessary folders in the mailbox. This makes sense as they are more than a single folder deep.  This appears to be a limitation of the combination of dovecot version and the mbox mail storage format.  Here is the error:

PHP Fatal error:  Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. The IMAP server could not create mailbox 'INBOX/CiviMail/ignored': A0006 NO Mailbox doesn't allow inferior mailboxes.' in /var/www/html/sites/all/modules/civicrm/packages/ezc/Mail/src/transports/imap/imap_transport.php:774
maillog excerpt with verbose logging from dovecot:

May 10 11:56:52 www dovecot: imap-login: Login: user=<civimail>, method=PLAIN, rip=::ffff:127.0.0.1, lip=::ffff:127.0.0.1, secured
May 10 11:56:52 www dovecot: IMAP(civimail): Effective uid=507, gid=507, home=/home/civimail
May 10 11:56:52 www dovecot: IMAP(civimail): mbox: data=/home/civimail/mail/:INBOX=/var/spool/mail/civimail
May 10 11:56:52 www dovecot: IMAP(civimail): mbox: root=/home/civimail/mail, index=/home/civimail/mail, inbox=/var/spool/mail/civimail
May 10 11:56:52 www dovecot: IMAP(civimail): Disconnected: Logged out
Even if I could have those scripts create those folders and process all that mail, I will have to do it nice'd from the command line.

With the volume of mail we are sending, I am hesitant to try and push it all through one of our google apps accounts, and have heard mixed reports on the ability to do that with google apps accounts (as distinct from gmail accounts) due to header rewriting on the part of google.

Other options include re-configuring the server's mail system to use courier-IMAP and sendmail instead of dovecot-IMAP and postfix (not so fun , or building a more recent version of dovecot that will support nested folders (which I have to confirm is even possible)... I could convert the mbox to a Maildir and try to use the Maildir processing system...

It seems like the IMAP / php route is the preferred way of handling the return channel..

Any ideas? Thanks for sharing.. 

I have transitioned to Maildir format on this server and can login via IMAP  Dovecot (Thunderbird 3.0 and via CiviMailProcessor.php, and EmailProcessor.php) I can see the (now) 34,000 messages in this mailbox.

when running CiviMailProcessor.php I receive the following and only the following:

connecting to localhost, authenticating as civimail and selecting INBOX mailboxes found: Trash, INBOX, INBOX.CiviMail.ignored, INBOX.CiviMail.processed
the page stops loading.
Dovecot shows a disconnection immediately:

Nov  9 15:58:29 www dovecot: imap-login: Login: user=<civimail>, method=PLAIN, rip=::ffff:127.0.0.1, lip=::ffff:127.0.0.1, TLS
Nov  9 15:58:29 www dovecot: IMAP(civimail): Effective uid=507, gid=507, home=/home/civimail
Nov  9 15:58:29 www dovecot: IMAP(civimail): maildir: data=/home/civimail/Maildir
Nov  9 15:58:29 www dovecot: IMAP(civimail): maildir: root=/home/civimail/Maildir, index=/home/civimail/Maildir, control=, inbox=
Nov  9 15:58:29 www dovecot: IMAP(civimail): Disconnected: Logged out

no related php errors in the apache error_log.

As a interesting and somewhat unrelated side note;

With the modded Amavisd pre content filter, the regex in the postfix virtual file (etc/postfix/virtual)
/(b|c|e|o|r|u)\.(\d+)\.(\d+)\.([0-9a-f]{16})@www\.example\.org$/ civimail@www.example.org

only works as a pcre regex for me, and the documentation for the amavisd return channel tells you to use this regex as a
regexp:/etc/postfix/vitrual

virtual_alias_maps = regexp:/etc/postfix/virtual <<--doesn't work
virtual_alias_maps = pcre:/etc/postfix/virtual <<--does work

http://wiki.civicrm.org/confluence/display/CRMDOC31/AMaViS

Pretty sure you don't want to go the patched Amavisd way.

http://forum.civicrm.org/index.php/topic,16430.0.html

Unless very good reason, stick to the config we suggest in the book.

X+

So instead of trying to make this old system work, I scrapped it and used a google apps e-mail address (with the localpart: mailaccountusername+ ) and IMAP for the php-based return channel. so far so good.  we'll see soon if I have any problems.

thx for the nudge X

 Everything is working quite well now!  I am using postfix SMTP for the outgoing mailings so I don't have to send that magnitude of mailings via Google. I already have an SPF record in DNS set up for that mailserver, so google and other domains know that the sender is valid, and the mails have a spf pass in the header when delivered. I am processing the mail via the wget in cron , at a rate of 50 mails every 10 minutes. I logged in a few times and pulled bounced mails categorized as spam with the "Not spam" button on the Google apps email web interface, and it seems to be doing quite well at figuring out the classification without any filters.

thanks again for the tips xavier

Maybe I should add that I created a folder in my bounce mail account that I am trying to use. So instead of using the default INBOX I am using (or trying to use) a folder called CiviCRM_INBOX.