Integrating civimail with Amazon SES

[xavier]

amazon has launched a mass mailing solution

http://aws.amazon.com/ses

Haven't tried, but their api seems to be simple to use after a 2 min analysis, and given the amount of not so solicited emails they send, they got their boxes tuned and whitelisted enough.

Might be something to consider if working on integration with mailchimp or the others.

Some ideas to improve the native functions of civimail:

- When you create a sender email, it sents an email to verify you can read from the email you are sending from

This might save us a lot of support time to have something like that plugged on the mailboxes and from email addresses: you enable them by clicking on a link.

And be sure to die/warning clearly when trying to send an email that the system  isn't properly configured if no mailbox has been enabled.

X+

[jtbayly]

I'd *very* much like to see Amazon SES incorporated, for the record. Is anybody out there working or planning to work on this?

[pcoughlin]

Would anyone be interested in sharing the cost of getting SES integrated into CiviCRM.  I would be able to do the work, just need a small group to help defray some of the costs with the programmers.

I have used the SES in a stand alone test and it should be reasonably straight forward to implement it.

Post something here or contact me through my profile.

Thanks ... Paul

[drupleg]

Hi, I've been using SES SMTP with CiviCRM for a few weeks now.  I have found it best, if you have server-level root access, to configure your postfix server for use with SES/SMTP.  There is a very informative webinar which covers this topic in detail step-by-step.

AWS SES SMTP Webinar (EC2+SMTP instructions start at 17:10 - 21:00)

And Duo Consulting has documented the steps they've found to work with SES/SMTP here:
Using Amazon’s SES with Postfix as a smarthost forwarder

The major concern with using SES/SMTP directly (with a username/password auth from an application) is that you'll encounter issues and bounces with the e-mail per second limit.  Currently, at the lowest level SES I've been using, the limit is 5 e-mails per second/10,000 per day.  If your application (CiviCRM or other) sends a barrage of 100 e-mails per second - SES SMTP will bounce those that exceed the threshold of 5 e-mails per second.  This limit is OK for use with your personal Outlook e-mail client (which they cover in the first part of the webinar).  But it is for this reason that it's best in my option to configure postfix as a relay for SES SMTP.  This way postfix is a buffer between your application and SES, so postfix will spool, delay, and retry sending the e-mails, and send them at a reasonable rate over time that will comply with the 5 per second limit.  Otherwise, if not using the postfix relay method, it is the application's responsibility to throttle back the e-mail per second send rate. 

While it would be awesome to just plug-in the SMTP credentials in CiviCRM directly, as that would be hugely beneficial for those on shared servers (without root access to postfix), I'm not a coder and don't know what it would take to have CiviCRM handle TLS and e-mail per second limits?

For now this postfix relay method is working very well for me - and my AWS bill has been less than any shared host, but I've been a hardcore server admin since Red Hat 4.x days (15+ years) so this level of configuration doesn't scare me .  I've been running a non-profit website on an EC2 instance for the past 18 months.  I was paying for an outside SMTP relay service because cloud/EC2 e-mail was unreliable and often blacklisted.  Now with SES/SMTP I am running this site on a FREE EC2 instance, with a 10,000 e-mail per day limit for this non-profit of about 1,000 members.  And it's only costing about $2 per month, and that's just because I have a few gigs of EBS volumes stored as backup. AWS+EC+SES/SMTP=WIN!

-- Tony

[xavier]

Interesting,

Have you tried to use postfix to deliver directly? Do you see a difference on the delivery rate/load on the server?

X+

[Dennis Gray]

We have been using CiviSMTP for several years and are generally happy with the service but it's always a good thing to look at alternatives. Has there been any progress on this?

One driving factor for us is that we are about to shift our production environment to EC2 and there may be some benefits to switching from CiviSMTP. Like I said, CiviSMTP has been good for us and service would outweigh cost in making a decision to move.

[xavier]

Not to my knowledge

[loriguidos]

Integrating AWS SES and CiviCRM has been a big disappointment.   We use Google Apps, but because of the 500 email limit we were hoping to integrate AWS SES with CiviCRM to save money.

We were using VPS.Net service with CiviCRM and setting up email took some time but nothing like integrating with SES which has taken us weeks and we still are not able to send mail.

Of course now we are trying to get a mailing out about our upcoming event and we are stuck...  I would recommend that you not try to get SES to work with CiviCRM when you are under an event deadline!

We have set up an Ubuntu server to implement Postfix are seeing mail being completed using the SES console (so we know we are doing something right), but now are working to set up Crontabs to meet Amazon's 5 per second rule instead of trying to figure out how to create a wrapper...

Neither of us setting up Drupal or AWS are employed as programmers or consultants -- simply two volunteers trying to set up a non profit to build a disability community.

It seems like there should be more complete instructions integrate SES into Civimail.

This document was some what helpful evanhoffman.com/evan/2011/04/28/integrating-amazon-simple-email-service-with-postfix-for-smarthost-relaying/

Webinar also helpful youtube.com/watch?v=1jEmL0UYQ5M&feature=youtu.be

But it would be soooooo helpful if someone at CiviCRM could make this process easier for us as many of us are small non profits --

There must be something that could document how to integrate Civicrm and Amazon SES?   Even better would be a module that might make this process easier?

Best,

Lori

[xavier]

Hi,

Could you start documenting what you've done and where you are in the wiki? You seem to be one of the first that is trying the integration with SES and as you know, civi is a community effort and relies on the pioneers documenting what they do so others can follow more easily.

I am not sure what's the issue, but if it's limiting the rate, you should try to do it at the postfix level (ie. civi pushes as normal speed the emails and postfix does the buffering)

This script allows to do that for what I've seen
http://postfwd.org/

Let us know how it goes.

(disclaimer, haven't tried)

[Donald Lobo]

Might want to use CiviSMTP: http://civismtp.com/drupal/

It is quite cheap and they provide excellent service. It definitely helps the ecosystem and the community to buy and support "local" services

lobo

[mrkschan]

Hi there, I'm also trying to use Amazon SES SMTP to support CiviMail. Unfortunately, I cannot make them work together right now. (My CiviCRM version is 4.0.4).

When I try to send a test email via CiviMail, the email is having two mail-from headers: (i) b.62.83.7a390e1c8cda57ea @ xxx.com and (ii) test @ xxx.com. I have already SES verified the second email address (test @ xxx.com) but I am not able to SES verify the first email address since that address is generated by CiviMail for bounce/reply tracking.

Did anyone encounter the same issue?

[xavier]

Hi,

CiviMail uses a different email for each mail sent to be able to track the bounces.

http://cr.yp.to/proto/verp.txt

If you put a prefix on your mailbox configuration (test+) you might be lucky and get it working.

Otherwise, it will need changes on civimail to work with SES.

X+

[mrkschan]

I do believe there must be some work to make CiviCRM work with Amazon SES (see my suggested patch several months ago http://forum.civicrm.org/index.php?topic=21583.0). For instance, some email headers (e.g. 'Precedence') are not supported by Amazon SES.

BTW, you mentioned putting a prefix to my config. Do you mean the "Localpart" settings of "Mail Accounts"?

[mrkschan]

An update. After providing the Localpart settings to CiviMail, Amazon SES accepts the VERP email address.

My configuration is having the following...

(i) Amazon SES verified email address - members@example.com
(ii) Amazon SES verified emaill address - civimail@example.com
(iii) Applied my patch at http://forum.civicrm.org/index.php?topic=21583.0
(iv) CiviMail Mail Accounts applied Localpart "civimail+"
(v) Postfix + SES setup by following http://blog.duoconsulting.com/2012/01/30/using-amazons-ses-with-postfix-as-a-smarthost-forwarder/
(vi) stunnel4 setup by following http://www.stefanolocati.it/blog/?p=737

And now, I can send a email from members@example.com, and its VERP email address (the envelope MAIL FROM header) is civimail+b.63.84.09b96f7c436bf3b7@example.com. It seems that Amazon SES loves the "civimail+" prefix.

[nicolas]

I have been using ElasticEmail with a number of CiviCRM installs and have been very satisfied: instant setup with the SMTP configuration, excellent deliverability, real-time stats on their dashboard, and very reasonable prices. I especially like that their stats can be overall or per sender address, and that they have a complete log of all emails sent  - so you can always check if that particular email really was sent. They also throttle the emails on their side, so you can submit 50,000 emails at once with SMTP and they will send them over whatever period of time is necessary to not hurt your reputation.

I had been looking at SES for the unbeatable pricing, but will stick to ElasticEmail for now because of the above features.