Author Topic: Integrating civimail with Amazon SES  (Read 10997 times)

Offline dotsam

  • I’m new here
  • *
  • Posts: 10
  • Karma: 0
  • CiviCRM version: 3.x, 4.x
  • CMS version: Drupal 6.x, 7.x
  • MySQL version: Various
  • PHP version: Various
Re: Integrating civimail with Amazon SES
« Reply #15 on: May 27, 2012, 10:01:16 am »
For reference, SES now supports domain verification and VERP, which makes the process of setting up with Civi quite easy.

I use Postfix to submit mail via STARTTLS to the Amazon SES SMTP. I didn't have to use stunnel, I just followed the process outlined in the first part of the integration documentation.

I can't get it to work using the SMTP settings for Outbound Mail in Civi; I just use sendmail instead (which is actually postfix). I haven't throttled postfix yet to respect Amazon's sending rate limits, but I think that's reasonably easy.
« Last Edit: May 27, 2012, 10:05:44 am by dotsam »

Offline drupleg

  • I post occasionally
  • **
  • Posts: 39
  • Karma: 2
    • Drupal.org Profile
  • CiviCRM version: 4.1
  • CMS version: Drupal 7
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #16 on: August 29, 2012, 05:01:31 pm »
Hi, just a follow up since my last post, as I received an inquiry from a potential client asking me about SES integration.  In short - I've given up on AWS SES+Postfix, I had nothing but trouble with implementing it.  It would work for a few hundred e-mails, then go belly-up when I sent out 2,000+ emails, with most of those I assume, going in to an email black hole. The queuing/per minute limit of SES was killing my messages and deliverability, and very frustrating to work with.

I have used the newer Amazon SES/SMTP which was much easier to configure CiviCRM with SMTP credentials, avoiding the Postfix integration altogether. Still the reporting was minimal and I just didn't like AWS SES/SMTP, it was not very user friendly in my opinion.

Then along came Mandrill by MailChimp a few months ago - http://blog.mailchimp.com/public-beta-of-mandrill-smtp-email-service/. This was a godsend for me as a freelance developer/one-man webshop, and has made my life easier. The biggest reason is that, though Mandrill and SendGrid have initial sending limits of 200 message per hour for example - both of these services handle the spooling/queue for you! So CiviCRM can blast out 2,000 emails in one shot, then Mandrill will send those messages out over how ever many hours, 200 each hour, until they are all delivered. That alone is worth moving away from SES to Mandrill or SendGrid.

I've migrated 7 of my clients to Mandrill, and couldn't be happier with the service.  I also used SendGrid for a few months too - they also have excellent SMTP deliverability and statistics, as well they're very affordable - less then $1 per month for most of my clients sending a few thousand e-mails per month.

Bottom line - if you want the easiest setup via SMTP for CiviCRM go with Mandrill or SendGrid. If you send less than 12,000 e-mails per month, Mandrill is free. Personally I'm not planning on using Amazon SES again after using these other services that offer so many features for such a low cost.  I've also moved all of my hosting off of AWS and on to HPCloud.

Lastly - I did not sign-up directly with Mandrill, I found it best to sign-up for Mandrill through MailChimp - first creating a free MailChimp account, then activating the Mandrill service for the MailChimp account through the Account > Extras > Integrations menu.  Hope this helps someone in the future.

--Tony

Offline Michael McAndrew

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1274
  • Karma: 55
    • Third Sector Design
  • CiviCRM version: various
  • CMS version: Nearly always Drupal
  • MySQL version: 5.5
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #17 on: August 30, 2012, 09:03:55 am »
Hey there,

I looked at using SES but was under the impression that you can only send from a limited number of specified email addresses which made it not appropriate for our use case since we host a number of sites for different clients and they all want to be able to send from their domains.  Can anyone tell me if I am mistaken on that?
Service providers: Grow your business, build your reputation and support CiviCRM. Become a partner today

Offline drupleg

  • I post occasionally
  • **
  • Posts: 39
  • Karma: 2
    • Drupal.org Profile
  • CiviCRM version: 4.1
  • CMS version: Drupal 7
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #18 on: August 30, 2012, 01:21:44 pm »
I can't find anything in the AWS SES documentation that places any limits on From: addresses. However if each client is hosting different sites on different domains - my personal preference would be to set them up each with their own AWS account, I wouldn't operate several clients under just 1 AWS/SES account, though you could if you wanted to just to get started experimenting.

Offline Michael McAndrew

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1274
  • Karma: 55
    • Third Sector Design
  • CiviCRM version: various
  • CMS version: Nearly always Drupal
  • MySQL version: 5.5
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #19 on: August 31, 2012, 10:41:47 am »
Hey there,

http://aws.amazon.com/ses/

Step 2 on the above page says 'Verify sending address(es) or domain(s).'

I think that must have changed since i last looked at it.  I don't remember seeing the option to verify a domain ~6 months ago, tho maybe i missed it.

Thanks for clarifying :)
Service providers: Grow your business, build your reputation and support CiviCRM. Become a partner today

Offline drupleg

  • I post occasionally
  • **
  • Posts: 39
  • Karma: 2
    • Drupal.org Profile
  • CiviCRM version: 4.1
  • CMS version: Drupal 7
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #20 on: September 04, 2012, 09:48:29 am »
Yes, you can now verify multiple email addresses from different domains for use with SES. I just prefer to keep each client's domain on separate accounts for logistical reasons.

Offline drupleg

  • I post occasionally
  • **
  • Posts: 39
  • Karma: 2
    • Drupal.org Profile
  • CiviCRM version: 4.1
  • CMS version: Drupal 7
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #21 on: September 04, 2012, 09:50:47 am »
I'm just tossing another reason out there to use Mandrill - There's a new CiviCRM extension for use with Mandrill transactional e-mails that integrates nicely with CiviCRM, allows bounce tracking through the CiviMail dashboard. I just installed it today, have a look.

http://civicrm.org/extensions/mandrill-transactional-emails

Offline drupleg

  • I post occasionally
  • **
  • Posts: 39
  • Karma: 2
    • Drupal.org Profile
  • CiviCRM version: 4.1
  • CMS version: Drupal 7
  • PHP version: 5.3
Re: Integrating civimail with Amazon SES
« Reply #22 on: September 04, 2012, 04:07:53 pm »
Another update, that if you are using Amazon SES - CiviCRM 4.2 does now support throttling: http://issues.civicrm.org/jira/browse/CRM-9698

Offline Sean_Cleary

  • I’m new here
  • *
  • Posts: 3
  • Karma: 0
  • CiviCRM version: 4.3.2
  • CMS version: Drupal 7.22
  • MySQL version: 5.5.28
  • PHP version: 5.4
Re: Integrating civimail with Amazon SES
« Reply #23 on: May 07, 2013, 01:16:12 pm »
G'day all! Bumping this topic to ask what would be required to make this a Make It Happen project: Amazon SES Transactional Email

Michael McA I have just read your blog-post about the current Make It Happen process. My reading of that process is that it is about enhancing or adding new features for CiviCRM release 4.4. wiki dot civicrm dot org/confluence/display/CRM/Make+it+Happen

So the Civi integration with AWS SES would appear to be a separate Make it Happen - given that might be backwardly compatible with existing deployments of earlier versions of CiviCRM rather than focussed on future releases?? (How far back may be at question given the relevance for the SES task of the throttling features implemented in Civi 4.2)

Given recent advancements in SES what do earlier posters who have explored options for Civi integration with SES see as the path forward technically? Is there a need for a CiviMail AWS SES extension/module? (In the same way as there is now an extension/module for Mandrill: www dot civicrm dot org/extensions/mandrill-transactional-emails )

So to unblock this log-jam what direction does the solution lie in?

I'm not a coder - but a novice CiviCRM volunteer implementer working for a Sydney-based not-for-profit in a Drupal 7 & Civi 4.3 environment. As such I'd be happy to throw USD$100 of my own funds into the pool and to also do some work to see if there are any others with a similar interest to commit some $'s to make this happen. So can we start the discussion to determine what the solution might look like technically and thus work out how much funding would be needed?

Michael Mc what would be the process for this to become a stand-alone Make It Happen project? (Or would it be incorporated into the current 4.4 effort?)

[NB Everyone posting here who has searched for solutions would be very aware that there are other paid providers available for transactional email solutions - but given the specifically stated topic of this thread my sense of protocol is that further flagging of these here as alternate solutions to Amazon SES is off-topic - and possibly spamming!]
« Last Edit: May 07, 2013, 01:19:24 pm by Sean_Cleary »

Offline xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Integrating civimail with Amazon SES
« Reply #24 on: May 07, 2013, 01:22:37 pm »
Hi,

mandrill is for transactional emails, ie. small volume, that's a different aim than amazon SES/sendgrid/civismtp...

I think each of these providers re-invented a different way to handle bounces, and possibly sending too (because smtp isn't good enough it seems), so it probably makes sense a separate extensions.

Don't think they have been a lot of traction to actually make this extension happen, let's see if your push is what is needed to get a wider (financial) support....

X+

Offline Sean_Cleary

  • I’m new here
  • *
  • Posts: 3
  • Karma: 0
  • CiviCRM version: 4.3.2
  • CMS version: Drupal 7.22
  • MySQL version: 5.5.28
  • PHP version: 5.4
Re: Integrating civimail with Amazon SES
« Reply #25 on: May 07, 2013, 01:33:50 pm »
Hi Xavier.

Quote
Amazon Simple Email Service (Amazon SES) is a highly scalable and cost-effective bulk and transactional email-sending service for businesses and developers.
So are you suggesting that whereas SES can do transactional - that facility is not required for most CiviCRM purposes - and would therefore add an unnecessary layer of complexity (and cost?) to what is needed?

Interesting to discover that MailChimp's pre-Mandrill transactional service ('STS) was itself using Amazon SES:
http:// blog dot mailchimp dot com/tag/transactional-email/
« Last Edit: May 07, 2013, 01:52:10 pm by Sean_Cleary »

Offline xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Integrating civimail with Amazon SES
« Reply #26 on: May 07, 2013, 10:32:32 pm »
Quote
So are you suggesting that whereas SES can do transactional - that facility is not required for most CiviCRM purposes - and would therefore add an unnecessary layer of complexity (and cost?) to what is needed?

I'm afraid you are over-reading me, I was simply not aware Amazon SES was presenting itself as both transactional and bulk ;)

The transactional emails sent from Civi aren't automatically managed for bounces, ie. if you register to an event with an invalid email, it won't automatically flag it as such in civi (nor the participant status updated). Having a better tracking of that would be useful and is missing.

As for the opening rate of mass mailing, I found it roughly similar to external solutions after properly configured and the IP address "approved"... but properly configure it (SPF, DKIM, reverse DNS...) is probably more work than most sites are willing to put.

So both type of emails might benefit having an external solution, those like sendgrid and amazon that offers to handle both sounds like an interesting option.

X+

Offline Sean_Cleary

  • I’m new here
  • *
  • Posts: 3
  • Karma: 0
  • CiviCRM version: 4.3.2
  • CMS version: Drupal 7.22
  • MySQL version: 5.5.28
  • PHP version: 5.4
Re: Integrating civimail with Amazon SES
« Reply #27 on: May 12, 2013, 06:06:47 pm »
Thanks Xavier - so is it particularly good news to discover that Amazon AWS SES can also do transactional email?

Still happy to hear suggestions from anyone as to what form a project such as this might take? - and what it would look like...?

- what will be the steps needed to achieve this functionality with Civi?

- is this a whole of CiviCRM component  - or specific only to use of CiviMail?

- how achievable might this be (i.e. how big a project is this? USD $1000 ?? or USD $50,000 ?

- is an extension the way to go?? or...??

Offline JonGold

  • Ask me questions
  • ****
  • Posts: 638
  • Karma: 81
    • Palante Technology
  • CiviCRM version: 4.1 to the latest
  • CMS version: Drupal 6-7, Wordpress 4.0+
  • PHP version: PHP 5.3-5.5
Re: Integrating civimail with Amazon SES
« Reply #28 on: February 24, 2016, 08:46:16 pm »
This topic is old, but it's still the top link for "Amazon SES CiviCRM" in Google, so:
I have some pretty decent documentation here:
https://hq.palantetech.coop/projects/commons/wiki/Amazon_SES_for_CiviCRM

This conversation has been continued on Stack Exchange here:
http://civicrm.stackexchange.com/questions/694/does-civimail-smtp-work-with-amazon-aws-ses
Sign up to StackExchange and get free expert CiviCRM advice: https://civicrm.org/blogs/colemanw/get-exclusive-access-free-expert-help