CiviCRM Community Forums (archive)

I've been looking into the idea of processing bounced emails with PHP since I seem to be allergic to PERL.  So far we have:

(1) Setup a catch all pop3 address mailbox and we are using this mailbox as the reply mailbox for CiviMail.  We are getting lots, but an expected number of bounces in the mailbox with some other mail (total of 1,400 from a dozen or so mailings of 200 to 1,600 contacts).

(2) Using a somewhat simplified version of the following php script:
http://www.phpit.net/article/read-email-php-pop3/
we are able to read the emails in the pop3 email account setup for replies and bounces.  I think there are many other scripts that allow php to read pop3 emails, the first one that I tried worked.

(3) With a little trial and error, the [2] item in the email array seems to be the bounced error and I can list out the message with the following added code using the script mentioned above:

for ($i = 1; $i <= $msg_list["count_mails"]; $i++) {
        $email = $pop3->get_mail($i);

        if ($email == false) {
                echo $pop3->error;
                continue;
        }

      if (substr($email[2], 0, 20) == 'Envelope-to: bounce.')
      echo $email[2] . '<br>';
}
exit;

I'm not quite sure what I'm doing, but it seems like with a little work we should be able to produce a list of email addresses that are bouncing and perhaps for those that are bouncing more than x times and maybe even update the contact record to place the email address on hold.

I take it that there may be other bounce error messages that are not caught by the mailbox and may not be available to the php programs?  Otherwise, why use PEARL rather than PHP?  A few clues on this?

If any others are interested in processing bounced emails with a php script rather than imap2soap, please let me know. 

In the post above, I discussed how I was printing out the bounce information lines from the pop3 emails.  The format is a little strange, so I wrote the following function to reconstruct the email addresses and then print them out on a webpage:

function get_bounce($str1)
{

   $dash = strpos($str1, '-') + 1;
   $equal = strpos($str1, '=') +1;
   $at = strpos($str1, '@');
   
   $email_bounced = substr($str1, $dash, $equal - $dash - 1) .
   '@' .
   substr($str1, $equal, $at - $equal);

   return $email_bounced;
}

Then I print them out:

for ($i = 1; $i <= $msg_list["count_mails"]; $i++)
//for ($i = 1; $i <= 10; $i++)
{
        $email = $pop3->get_mail($i);

        if ($email == false) {
                echo $pop3->error;
                continue;
        }
      if (substr($email[2], 0, 20) == 'Envelope-to: bounce.')
      {
         echo get_bounce(substr($email[2],20));
              echo '<br>';
      }
}

The above prints out the reconstructed bounced email addresses, one per line.  This takes a while to run and results in 1440 bounces, but its not as bad as it seems since many repeat 6 or more times.   I then copy and paste the list to an Excel file and sort based on the email address.

Then manually, I placed a few email addresses "on hold" in the database.  After a couple of these, I looked into importing the email address and the "on hold" status of 1.  However, the "on hold" field does not seem to be one of the allowed import fields.

In table civicrm_email there is a field email, on_hold, hold_date and reset_date which seem to properly reflect my manual updates.

In advanced search, there does not seem to be a search field for on_hold, however, in Search Builder, there is an on_hold field and I can select the on_hold records for review there.  May not be perfect, since only searching the primary email, but good enough for us.   I can probably create a group or use the print button to produce a list for review.

I'm getting pretty close to being able to have the program flag an email as on hold, perhaps also marking the hold_date or there may be a function that I can call.  Also time to look at the the previous suggestion by Piotr:

Basically, all you have to do is to pass the bounce emails into our SOAP interface. You don’t have to scan for the email addresses, you don’t have to judge the bounce reasons, you don’t have to count the number of bounces and put emails on hold – this is all done by CiviCRM. (Take a look at the civicrm_mailing_bounce_type and civicrm_mailing_bounce_pattern tables – having said the above, we’re very interested in real-life use cases and any additions to these tables’ contents.)

If you figure out how to take emails from your inbox and pass them to CiviCRM’s SOAP interface (in the proper form), my bet would be you’re basically done with your solution.

I'm not saying this approach is optimal, but with 3,000 contact records, we may be able to keep a better handle on the bounces.

I would love to share ideas with anyone experimenting along this line.

Thank you.

I'm assuming you have a good reason and permission from the 2000 contacts to take them "off hold"

if you have the contact ids, a simple sql statement will do the needful. Rob Thorne should be able to help you out with this pretty easily and quickly. Not sure if you can do this via the UI but it will be at least 20 form submits, so doing it via SQL might be a safer bet (and avoid user errors and potentially allow u to backtrack!)

lobo

the on_hold is a column in the civicrm_email table.

When a user unsubs, their status in the group is switched to "Removed". on_hold is not used to manage unsubscribes

lobo
 

Piotr Szotkowski said 'use Perl because it’s available on the servers almost by default, is way faster than PHP and is known to more server administrators than PHP.'  What do you mean by faster, less CPU time per message or less processing (elapsed) time per message?

In our default approach (i.e., with AMaViS), the daemon runs in background, so there’s no overhead for starting the scripting engine for every email; also, AMaViS was designed to handle large amounts of email as fast as possible.

Given that the imap2soap.pl script reads a message from an IMAP mailbox on a possibly remote server, it is going to be very slow in terms of elapsed time, irrespective of programming language.

That’s true, and we’d prefer to work with PHP rather than Perl as well; it’s just that we believe a PHP version of this script might not be a viable replacement for the AMaViS solution in 100% of the cases (i.e., in all the cases when imap2soap isn’t).

Here is my two cents.

I just went through the dependency pain and got soap::lite to install. But still had issues with imap2soap. After that I had to install php-soap an that took care of it.

After all of this wonderful time spent on this small piece of a much bigger project, I thought of this idea.

Why not run the imap2soap as a free service or donation based service.

1. User comes to http://civiGate.civicrm.org (or whatever).
2. User registers.
3. User puts in the 8 fields or whatever that is in the user's conf file (imap2soap.conf)
4. User now has the whole problem resolved and can focus on installing and setting up CiviCRM instead of trying to kludge her way through definitely intermediate level tasks for any admin.

I might be missing something. But form what I understand, the .pl script doesn't care where it is located. It talks to IMAP and it talks to the soap.php file. So why not just make this a service and rapidly decrease the installation time.

I just don't get it. I know someone would have to put up the server. But we are talking about a basic php form and the 2 scripts. I will be more than happy to put up the resources if anyone wants to help. I just think it would make a great tool for rapid adoption of this wonderful platform.

To be clear, I am not talking about an SMTP service. Just a server dedicated to running the imap2soap.pl script for users. In the time it takes a user to sign up for the forums and post a problem about this issue, they could have the service up and running. It might also provide some interesting statistics to help the CiviCRM community.

I am not as wise and worldly as the posters here on the board. Amazingly brilliant the lot of you. Rather I am looking for a simple solution to a rather complex (by my capabilities anyway) problem.

I welcome feedback and apologize in advance if this is a ridiculous notion.

Better yet, as I write this post, why not just add it as a checkbox option under the global setting for the smtp setup ;-)

I will provide the server and get the first script to run if anyone wants to pitch in and help. I'm a kludger at best, not a programmer. I can get this working, but it would be VERY slick if it were built into the CiviCRM administration panel.

Donald,

I was very specific to say that I was aware of that and that I was not suggestioning a full SMTP service. I am aware of the full service SMTP service you mentioned (again, as was stated in my original post).

My point is to provide a VERY BASIC and simple to use imap2soap service. It would only handle the imap processing and send to the soap client on Civic (soap.php).

Within the CiviCRM there could be a simple form to fill out (it would already know where your soap.php file is and know your domain name) which would include your imap account information.

This would DRAMATICALLY reduce the headaches the average admin goes through to wrestle with dependency issues as well as limitations due to shared/virtual hosting, etc.

I would require some help, but I am willing to put up the resources to provide the service.

Please read post before replying. Thank you ;-)

I have installed imap2soap on my server and the two issues I have run into were
1. perl dependency hell
2. a problem with symbolic links being resolved where I'd rather they weren't
Once I had those sorted out I simply followed the instructions in the docs and it all seems to work.

Issue 2 has been discussed elsewhere on this board, should not be a problem for most people, and solved in 2.2.

As for issue 1, look at the topic of this thread, "Processing Bounces With PHP", and the discussion that actually started it.
Which is about replacing the perl script with a PHP-based solution. That should get rid of dependency hell.
Everything else being equal (i.e. same functionality and good documentation) that should just work, no?

Personally, I would much prefer that approach over having to configure some external service to process my bounces.
Which may in fact not be as trivial to setup as it seems on first glance.

So if we're talking about avoiding perl dependencies, wouldn't it make more sense to rather avoid perl and put this work towards a working PHP-implementation, i.e. getting it to work seamlessly, than for setting up an external service, and then implementing all the changes in order to be able to use that, to achieve the same ends?

Just my 3 Cents, anyway, and I can't write PHP, either.

Ronald