Processing Bounces With PHP

[Denver Dave]

I've been looking into the idea of processing bounced emails with PHP since I seem to be allergic to PERL.  So far we have:

(1) Setup a catch all pop3 address mailbox and we are using this mailbox as the reply mailbox for CiviMail.  We are getting lots, but an expected number of bounces in the mailbox with some other mail (total of 1,400 from a dozen or so mailings of 200 to 1,600 contacts).

(2) Using a somewhat simplified version of the following php script:
http://www.phpit.net/article/read-email-php-pop3/
we are able to read the emails in the pop3 email account setup for replies and bounces.  I think there are many other scripts that allow php to read pop3 emails, the first one that I tried worked.

(3) With a little trial and error, the [2] item in the email array seems to be the bounced error and I can list out the message with the following added code using the script mentioned above:

for ($i = 1; $i <= $msg_list["count_mails"]; $i++) {
        $email = $pop3->get_mail($i);

        if ($email == false) {
                echo $pop3->error;
                continue;
        }

      if (substr($email[2], 0, 20) == 'Envelope-to: bounce.')
      echo $email[2] . '<br>';
}
exit;

I'm not quite sure what I'm doing, but it seems like with a little work we should be able to produce a list of email addresses that are bouncing and perhaps for those that are bouncing more than x times and maybe even update the contact record to place the email address on hold.

I take it that there may be other bounce error messages that are not caught by the mailbox and may not be available to the php programs?  Otherwise, why use PEARL rather than PHP?  A few clues on this?

If any others are interested in processing bounced emails with a php script rather than imap2soap, please let me know. 

[Piotr Szotkowski]

Thanks a lot, Dave, for looking into it. In fact, we (CiviCRM) are quite interested in this.

Some remarks before I’ll try to wrap my mind around this further:

We (AMaViS being the supported solution) and others (imap2soap being the community-driven one) use Perl because it’s available on the servers almost by default, is way faster than PHP and is known to more server administrators than PHP.

We support AMaViS as this is the proven solution for handling mass mailings; we doubt a PHP-based solution can even get close to Perl-based daemon running in the background all the time.

That said, we’re willing to address the needs of people who can’t run (in particular – a modifield version of) AMaViS, so I’m very interested if the PHP solution works for you in the end.

Basically, all you have to do is to pass the bounce emails into our SOAP interface. You don’t have to scan for the email addresses, you don’t have to judge the bounce reasons, you don’t have to count the number of bounces and put emails on hold – this is all done by CiviCRM. (Take a look at the civicrm_mailing_bounce_type and civicrm_mailing_bounce_pattern tables – having said the above, we’re very interested in real-life use cases and any additions to these tables’ contents.)

If you figure out how to take emails from your inbox and pass them to CiviCRM’s SOAP interface (in the proper form), my bet would be you’re basically done with your solution.

[Denver Dave]

In the post above, I discussed how I was printing out the bounce information lines from the pop3 emails.  The format is a little strange, so I wrote the following function to reconstruct the email addresses and then print them out on a webpage:

function get_bounce($str1)
{

   $dash = strpos($str1, '-') + 1;
   $equal = strpos($str1, '=') +1;
   $at = strpos($str1, '@');
   
   $email_bounced = substr($str1, $dash, $equal - $dash - 1) .
   '@' .
   substr($str1, $equal, $at - $equal);

   return $email_bounced;
}

Then I print them out:

for ($i = 1; $i <= $msg_list["count_mails"]; $i++)
//for ($i = 1; $i <= 10; $i++)
{
        $email = $pop3->get_mail($i);

        if ($email == false) {
                echo $pop3->error;
                continue;
        }
      if (substr($email[2], 0, 20) == 'Envelope-to: bounce.')
      {
         echo get_bounce(substr($email[2],20));
              echo '<br>';
      }
}

The above prints out the reconstructed bounced email addresses, one per line.  This takes a while to run and results in 1440 bounces, but its not as bad as it seems since many repeat 6 or more times.   I then copy and paste the list to an Excel file and sort based on the email address.

Then manually, I placed a few email addresses "on hold" in the database.  After a couple of these, I looked into importing the email address and the "on hold" status of 1.  However, the "on hold" field does not seem to be one of the allowed import fields.

In table civicrm_email there is a field email, on_hold, hold_date and reset_date which seem to properly reflect my manual updates.

In advanced search, there does not seem to be a search field for on_hold, however, in Search Builder, there is an on_hold field and I can select the on_hold records for review there.  May not be perfect, since only searching the primary email, but good enough for us.   I can probably create a group or use the print button to produce a list for review.

I'm getting pretty close to being able to have the program flag an email as on hold, perhaps also marking the hold_date or there may be a function that I can call.  Also time to look at the the previous suggestion by Piotr:

Basically, all you have to do is to pass the bounce emails into our SOAP interface. You don’t have to scan for the email addresses, you don’t have to judge the bounce reasons, you don’t have to count the number of bounces and put emails on hold – this is all done by CiviCRM. (Take a look at the civicrm_mailing_bounce_type and civicrm_mailing_bounce_pattern tables – having said the above, we’re very interested in real-life use cases and any additions to these tables’ contents.)

If you figure out how to take emails from your inbox and pass them to CiviCRM’s SOAP interface (in the proper form), my bet would be you’re basically done with your solution.

I'm not saying this approach is optimal, but with 3,000 contact records, we may be able to keep a better handle on the bounces.

I would love to share ideas with anyone experimenting along this line.

Thank you.

[Dennis Gray]

I have a need which probably requires a new topic but I'll post it here since it is relevant to the thread going on  here and people reading this thread probably already understand this.

Simply, how can I do a batch update to take the over 2000 "on hold" members off hold. I tried using a profile but my skills with CiviCRM are pretty limited right now. I'm going for some training in Melbourne next month so hope to be smarter by then. In the meantime, we need to take these people off hold as soon as possible.

Thanks for your help.

[Donald Lobo]

I'm assuming you have a good reason and permission from the 2000 contacts to take them "off hold"

if you have the contact ids, a simple sql statement will do the needful. Rob Thorne should be able to help you out with this pretty easily and quickly. Not sure if you can do this via the UI but it will be at least 20 form submits, so doing it via SQL might be a safer bet (and avoid user errors and potentially allow u to backtrack!)

lobo

[Dennis Gray]

I thought on hold meant the email bounced and was put on hold by the system. If a user unsubscribes, does that also put them on hold?

I can probably write the SQL myself after I study the data model a bit more. We have a test replica to do that.

[Donald Lobo]

the on_hold is a column in the civicrm_email table.

When a user unsubs, their status in the group is switched to "Removed". on_hold is not used to manage unsubscribes

lobo
 

[john ackers]

I, too, am interested in a PHP version of IMAP2SOAP.

I am trying to track down a bug similar to http://www.nabble.com/Error-with-imap2soap.pl-td16870914s15986.html in the existing Perl script and it is slightly annoying because I have to look at the server software at the same time which is written in PHP.   

Piotr Szotkowski said 'use Perl because it’s available on the servers almost by default, is way faster than PHP and is known to more server administrators than PHP.'  What do you mean by faster, less CPU time per message or less processing (elapsed) time per message?  Given that the imap2soap.pl script reads a message from an IMAP mailbox on a possibly remote server, it is going to be very slow in terms of elapsed time, irrespective of programming language.

 

[Piotr Szotkowski]

Piotr Szotkowski said 'use Perl because it’s available on the servers almost by default, is way faster than PHP and is known to more server administrators than PHP.'  What do you mean by faster, less CPU time per message or less processing (elapsed) time per message?

In our default approach (i.e., with AMaViS), the daemon runs in background, so there’s no overhead for starting the scripting engine for every email; also, AMaViS was designed to handle large amounts of email as fast as possible.

Given that the imap2soap.pl script reads a message from an IMAP mailbox on a possibly remote server, it is going to be very slow in terms of elapsed time, irrespective of programming language.

That’s true, and we’d prefer to work with PHP rather than Perl as well; it’s just that we believe a PHP version of this script might not be a viable replacement for the AMaViS solution in 100% of the cases (i.e., in all the cases when imap2soap isn’t).

[Denver Dave]

At the level we are operating at (2,400) emails out in a big run, a manually initiated batch process or a process kicked off by a cron job to read the reply email box and identify bounces is not likely to consume a significant amount of resources.  Unlike mailing out which we are doing at 50 emails every 10 minutes, processing bounces could be done once a day, once a week or on demand.

We have every bounce we've ever had currently in our reply inbox, and it does take a while to process all of them with PHP where we are reading every line of every returned email, but PHP does handle it.  When we get rid of most of the old bounces and only process current bounces, we shouldn't have more than a few items to process with each bounce process run.  Also, in theory, we should not have to read a few lines from the bounce email, not the whole email.

I confess I'm curious about this whole process.  Even with my php coding level, it was not hard to read the reply mailbox and list off the bounces with php. (certainly easier that figuring out which script runs other processes in CiviCRM).  At this point, I haven't done anything with the bounces other than list them off.  My plans are to do a report list combining the email address of the bounce, date of the email with our custom field that identifies the state chapter that the record belongs to.  Then each chapter will investigate their own bounces.   

Bounces mostly represent bad email addresses, so we need to try to contact the person anyway and try to get a good or current email address.

= = = = =

Other advantages of processing return email box with php, in addition to bounces: 

All CiviMail returned and reply email seems to be going to the same inbox, regardless of who sent it.  We have 5 different chapters, sometimes with more than one person in each chapter doing the mailings, so the replies are all mixed up together.  Perhaps if we read the inbox with php, we could split the replies by who sent the mailing and make the reply information available to the right person.

[clearlytechnical]

Here is my two cents.

I just went through the dependency pain and got soap::lite to install. But still had issues with imap2soap. After that I had to install php-soap an that took care of it.

After all of this wonderful time spent on this small piece of a much bigger project, I thought of this idea.

Why not run the imap2soap as a free service or donation based service.

1. User comes to http://civiGate.civicrm.org (or whatever).
2. User registers.
3. User puts in the 8 fields or whatever that is in the user's conf file (imap2soap.conf)
4. User now has the whole problem resolved and can focus on installing and setting up CiviCRM instead of trying to kludge her way through definitely intermediate level tasks for any admin.

I might be missing something. But form what I understand, the .pl script doesn't care where it is located. It talks to IMAP and it talks to the soap.php file. So why not just make this a service and rapidly decrease the installation time.

I just don't get it. I know someone would have to put up the server. But we are talking about a basic php form and the 2 scripts. I will be more than happy to put up the resources if anyone wants to help. I just think it would make a great tool for rapid adoption of this wonderful platform.

To be clear, I am not talking about an SMTP service. Just a server dedicated to running the imap2soap.pl script for users. In the time it takes a user to sign up for the forums and post a problem about this issue, they could have the service up and running. It might also provide some interesting statistics to help the CiviCRM community.

I am not as wise and worldly as the posters here on the board. Amazingly brilliant the lot of you. Rather I am looking for a simple solution to a rather complex (by my capabilities anyway) problem.

I welcome feedback and apologize in advance if this is a ridiculous notion.

Better yet, as I write this post, why not just add it as a checkbox option under the global setting for the smtp setup ;-)

I will provide the server and get the first script to run if anyone wants to pitch in and help. I'm a kludger at best, not a programmer. I can get this working, but it would be VERY slick if it were built into the CiviCRM administration panel.

[Donald Lobo]

There is a service called CiviSMTP which takes over all smtp and inbounce processing services for CiviMail / CiviCRM. You can find more info here:

http://civismtp.uas.coop/drupal/
http://www.google.com/search?q=civismtp+site%3Acivicrm.org&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a

lobo

[clearlytechnical]

Donald,

I was very specific to say that I was aware of that and that I was not suggestioning a full SMTP service. I am aware of the full service SMTP service you mentioned (again, as was stated in my original post).

My point is to provide a VERY BASIC and simple to use imap2soap service. It would only handle the imap processing and send to the soap client on Civic (soap.php).

Within the CiviCRM there could be a simple form to fill out (it would already know where your soap.php file is and know your domain name) which would include your imap account information.

This would DRAMATICALLY reduce the headaches the average admin goes through to wrestle with dependency issues as well as limitations due to shared/virtual hosting, etc.

I would require some help, but I am willing to put up the resources to provide the service.

Please read post before replying. Thank you ;-)

[Donald Lobo]

yes, i did read the post before replying

however not sure if u've setup civismtp, but it is fairly trivial to set up and does BOTH inbound and outbound processing for a pretty low price (note that we have no incentive to plug that service). typically shared hosts also have issues with high volumes of outbound mail also and a more complete service like CiviSMTP makes more sense, IMO. Running a 24x7 service is a pretty intensive operation and a bit more involved than having an available resource

lobo

[ronald]

I have installed imap2soap on my server and the two issues I have run into were
1. perl dependency hell
2. a problem with symbolic links being resolved where I'd rather they weren't
Once I had those sorted out I simply followed the instructions in the docs and it all seems to work.

Issue 2 has been discussed elsewhere on this board, should not be a problem for most people, and solved in 2.2.

As for issue 1, look at the topic of this thread, "Processing Bounces With PHP", and the discussion that actually started it.
Which is about replacing the perl script with a PHP-based solution. That should get rid of dependency hell.
Everything else being equal (i.e. same functionality and good documentation) that should just work, no?

Personally, I would much prefer that approach over having to configure some external service to process my bounces.
Which may in fact not be as trivial to setup as it seems on first glance.

So if we're talking about avoiding perl dependencies, wouldn't it make more sense to rather avoid perl and put this work towards a working PHP-implementation, i.e. getting it to work seamlessly, than for setting up an external service, and then implementing all the changes in order to be able to use that, to achieve the same ends?

Just my 3 Cents, anyway, and I can't write PHP, either.

Ronald