CiviMail setup notes on 2.0

[Chris Burgess]

Set up CiviMail on 2.0 tonight, thought I'd contribute a few notes on it ... {{ EDITED }}

All in all, it went a lot more easily than I expected - took me a few hours to set up and test, and no major obstacles appeared. Initial read of the docs had me surprised to think I might need to be reconfiguring an entire mailserver just to send emails, but it turned out that wasn't necessary. We use Exim and I'd be keen to help provide documentation for CiviMail setup with that mailserver, once I understand AMaViS's role in the scheme of things.

Methodology

1. Amavis + Postfix wasn't really an option for us, as we have a fairly extensive existing setup using Exim4 + a couple of virtual domains + Drupal integration. However the imap2soap.pl script seems to function fine so far if installed as per the docs. I got the impression that it might not handle heavier mailing loads - or is it just less well tested?

I'd love to read about the reasons behind using a custom built version of AMaViS - is there a post on it somewhere?

Issues Encountered

2. In ?q=civicrm/admin/setting/url the docs for CiviCRM Resource URL say "Absolute or Relative URL". However, if this is a relative URL (which I prefer, so I can switch test servers without having to update config settings in DB) then the generated clicktrack URLs in the email will be of the form http://path/to/file

Eg, our CiviCRM Resource URL is set to /drupal/sites/all/modules/civicrm/, so the clicktrack URL in the email becomes <a href="/drupal/sites/all/modules/civicrm/extern/url.php?u=2&qid=15"> which doesn't work in the recipient's email (unless they HAPPEN to be reading it via webmail on the same machine!)

I'll file this as an issue against 2.0 - I think that CiviCRM should correctly handle relative resource URLs by calculating the correct absolute URL in this case. (Alternative interpretation: there is a bug in the inline documentation, which should say that relative CiviCRM Resource URLs are OK, except with CiviMail.)

3. The HTML editor mostly worked, but I couldn't paste a URL into the little URL pastey drop-down thing. FFox 2.0 / OSX. Will test more on this before filing a report. It showed only an X where the URL should go, and accepted input only into the "link text" field, but not the URL field.

Docs

4. The alternate config docs say "Note the Perl modules required in the script and make sure they are installed on the computer you will be running the script on.*" While the actual list of modules is listed once or twice on the same page, I found the docs here a bit oblique, and wasn't sure about the exact list of modules they wanted.

These turned out to be: libsoap-lite-perl, libmail-imapclient-perl, libmail-verp-perl (under Debian). Or, in CPAN terms, SOAP::Lite, Mail::ImapClient, Mail::VERP - is that right?

Foolishness, and a Suggestion

5. I confused myself by trying to specify two email addresses in the test facility. Turned out you have to send them to single addresses using that box. This was entirely my own stupidity, but it would be cool if that test field could handle multiple emails being entered into it (so I can test x@hotmail.com,x@gmail.com,x@mac.com).

Other

6. One contact had an email address in his profile, but didn't appear in the mailing results until I edited and saved him. (Probably just didn't have is_primary set - working with dodgy test data set.)

7. Another question: The alternate config docs say that there is a VERP address of the format, group:subscribe.domain-id.group-id@yourCiviMail-mail-domain

The others all commence with a keyword then a period; optout.XXX, unsubscribe.XXX, bounce.XXX ... Does this other VERP address really have a colon? Or is it a typo in the docs? Where in CiviCRM can I see group subscribe by email functionality in action?

[Piotr Szotkowski]

We use Exim and I'd be keen to help provide documentation for CiviMail setup with that mailserver, once I understand AMaViS's role in the scheme of things.

That’d be super-cool. AMaViS’s role is to handle emails incoming to the CiviMail install – basically, the default setup is to make Postfix route all email traffic through AMaViS, and then make AMaViS cherry-pick the CiviMail-relevant emails and push them through CiviCRM’s SOAP interface.

However the imap2soap.pl script seems to function fine so far if installed as per the docs. I got the impression that it might not handle heavier mailing loads - or is it just less well tested?

This is the reason why we went with AMaViS in the first place – imap2soap is convenient in that it does not require root access to the box, but we consider AMaViS to be better performance-wise. Also, if I get the idea behind imap2soap right, it’s cron-based, while AMaViS is more or less ‘live’ in that it passes the incoming emails to CiviCRM ’s SOAP interface as soon as they come.

Issues Encountered

Thanks for catching these. Please file them in our issue tracker.

4. The alternate config docs say "Note the Perl modules required in the script and make sure they are installed on the computer you will be running the script on.*" While the actual list of modules is listed once or twice on the same page, I found the docs here a bit oblique, and wasn't sure about the exact list of modules they wanted.

These turned out to be: libsoap-lite-perl, libmail-imapclient-perl, libmail-verp-perl (under Debian). Or, in CPAN terms, SOAP::Lite, Mail::ImapClient, Mail::VERP - is that right?

We don’t support imap2soap (as it’s a community contribution), but it seems the list is about right – to get our AMaViS working under Debian, you need to satisfy the amavisd-new package’s dependencies and, additionally, install libsoap-lite-perl and libmail-verp-perl, so adding libmail-imapclient-perl to this for imap2soap makese sense.

5. I confused myself by trying to specify two email addresses in the test facility. Turned out you have to send them to single addresses using that box. This was entirely my own stupidity, but it would be cool if that test field could handle multiple emails being entered into it (so I can test x@hotmail.com,x@gmail.com,x@mac.com).

Please file an issue for this.

7. Another question: The alternate config docs say that there is a VERP address of the format, group:subscribe.domain-id.group-id@yourCiviMail-mail-domain

This is a typo, I fixed the docs. Thanks!

Where in CiviCRM can I see group subscribe by email functionality in action?

You mean, once somebody subscribed? They should be simply added to the relevant group.

[Chris Burgess]

Thanks for your detailed responses, Piotr, and for fixing up the docs.

I can't duplicate the Dojo richtext editor bug on demo (it's sitting at "welcome to your new drupal site") so I will leave that one until I can reproduce it. I have confirmed that both myself and another Mac / FFox user can't use the widget to insert a URL, but I was able to successfully enter a URL using Safari.

I have filed the relative URL issue with CiviMail click tracking; it's http://issues.civicrm.org/jira/browse/CRM-2994

If AMaViS is basically just piping addresses matching certain formats into calls to a SOAP handler, then Exim + a wrapper script should be fine. You're right in that imap2soap.pl does introduce a (configurable) delay before bounces etc are handled.

The reason I couldn't find self-serve group subscriptions was that I hadn't yet turned on the permissions setting "access CiviMail subscribe/unsubscribe pages" in Drupal. I had a look on demo and saw the sign-up screen, and that put me on the right path

[Piotr Szotkowski]

If AMaViS is basically just piping addresses matching certain formats into calls to a SOAP handler, then Exim + a wrapper script should be fine.

Yeah. You can diff our version(s) of AMaViS against either the upstream or Debian version (I can’t remember at the moment which one we forked) and see the changes.

You're right in that imap2soap.pl does introduce a (configurable) delay before bounces etc are handled.

Another advantage of imap2soap is that it can operate remotely from another machine, which makes it possible to install CiviMail in restricted hosting configurations (without any incoming email handling other than catch-all, for example).