PHP-based solution for the return channel

[Piotr Szotkowski]

As some of you know, we’re coming up with a PHP-based solution for handling CiviMail’s return channel (which will work as imap2soap does, but should be much easier to setup and also work with POP3 and Maildir email sources). The relevant issue is CRM-3616.

This will be a part of CiviCRM 2.2, but for those who would like to alpha/beta-test an unsupported version on their 2.1 installations, the files can be fetched from our repository:

• bin/CiviMailProcessor.php
• CRM/Mailing/MailStore.php
• all the files from CRM/Mailing/MailStore

Eventually, for 2.2, these files will retrieve the configuration from the database, but for your purpose please edit the CRM/Mailing/MailStore.php’s getStore() method.

Do feel free to ask any questions (and post any comments!) in this thread.

Note: This is by far not finished, but it did work for me at least once for all of IMAP/POP3/Maildir.

Edit: As I want to divert this for trunk, I committed a 2.1-compatible version to the v2.1 branch. The links above are updated to point to it.

[clearlytechnical]

First, thank you very much.

Second, I am going to give this the college try. My first question would be:

How do we initiate this? If I copy the files to the appropriate directories, then what next?

My assumption is that CivicMailProcessor.php is a modified version of an existing file and is being called somehow?

Not certain how to fire the event to cause this to pull from imap. Do I run a cron job?

Also, do I make a directory called MailStore and put those files you provided in there? (eg POP3, etc.)

Ok, off to work I go.

UPDATE:

Is there a Debugging tool for this? I can't tell if anything is happening.

I tried sending an email with verp info, but nothing seems to have processed.

[clearlytechnical]

Ok, I put all the files in the appropriate directories that mirror that of the branch.

I edited the method to reflect the server/user/pass as per your instructions.

Then I tried to go to the bin directory and just fire up the CiviMailProcessor.php and I received this message (after getting past safe mode issues and rights issues).

------ERROR BEGIN-------

ERROR: You need to send a valid user name and password to execute this file

------ERROR END-------

[clearlytechnical]

Ok, regarding the ERROR I was getting, please see this link:

http://forum.civicrm.org/index.php?topic=4239.0

I am going to try to call the CiviMailProcessor.php using curl and append the Joomla username/password file that I had originally setup for imap2soap.pl

I will report back with my findings.

UPDATE:

Curl gave me an error about username and password, but from within a browser it does not give me an error.


UPDATE:

If you use Curl, make sure you put the entire URL string in Quotes e.g.

curl "http://domain.com/blahblah?username=whatever"

If you don't put the quotes in, it will throw an error. This is why it worked via the browser, but not curl. Please be aware that I am an end user, so while many of you know the proper syntax for curl, others who are just administrators, etc. may find this information helpful, so please forgive the abundant posting ;-)

[clearlytechnical]

Ok, here is an update. (I am running Joomla 1.5.7)

I tried to call the CiviMailProcessor.php from my browser with the following (please note I changed the domain/username/password to protect client).

http://www.DOMAIN.com/administrator/components/com_civicrm/civicrm/bin/CiviMailProcessor.php?name=civicrm&pass=secretsquirrel

This time I did not get an error.

I will now check my mailbox, clean it up of all files. Then generate some verp (or whatever they are called) type emails to my imap/catchall and see if they get processed and report back.

Sorry for all the posts, just trying to help.

UPDATE:

Nothing seems to be happening. Will pick it up later. It would be helpful if I had a procedure to do or a way to debug this. Thank you again for your contribution Piotr.

UPDATE:
Ok, here is what I did to test.

1. Installed the scripts and set rights to appropriate directories as per the branch.

2. setup a test contact and put in a sample of all tokens

3. Sent a test email. then selected to reply to the token starting with o to "opt out"

4. after it arrived in my imap account, i then ran the script using Curl as per the string provided herein this thread.

5. went back to check the test contact to see if opt out was checked, it was not :-(

So I am stuck now and will wait to hear back from you. Let me know if you have any questions. Thank you.

[Piotr Szotkowski]

Hey, thanks a lot for testing! In the meantime, I created a wiki page on configuration, but it seems you managed to figure it all out.

It seems I need to put in some debug info in the scripts; I’ll do that right away and ping you back here.

In the meantime, please check whether this is not a problem with accessing the IMAP server with encryption (the default behaviour of CiviMail Processor). Also, make sure you have the domain set properly in CiviCRM (but if the emails are not being moved to CiviMail.processed and CiviMail.ignored folders – which they aren’t, if I get you right – then it’s a problem on an earlier stage).

[Piotr Szotkowski]

It seems I need to put in some debug info in the scripts; I’ll do that right away and ping you back here.

I just did in the recent commit to the v2.1 branch. Please try the files now (you might want to replace all of them and just re-edit CRM/Mailing/MailStore.php to add your settings back).

[clearlytechnical]

Piotr,

Thank you for the prompt reply and follow up. Nice wiki!

I will do as you suggest and report back in this message with "UPDATE"s.

Also, is there an actually test of some kind or procedure you recommend to test this? Or was what I did in the previous post sufficient?


Thanks!

UPDATE:
I see in the WIKI you made, that you are using WGET for the cron job. Could you brief me on why wget verses CURL?

When I used WGET, it just downloaded the file and saved it. (whatever that page would have been or whatever.)

[Piotr Szotkowski]

Also, is there an actually test of some kind or procedure you recommend to test this? Or was what I did in the previous post sufficient?

It was sufficient; hopefully the debug messages I added (controlled with the $_debug variable in CRM/Mailing/MailStore.php for 2.1) will shine some light on where the problems lie.

I see in the WIKI you made, that you are using WGET for the cron job. Could you brief me on why wget verses CURL?

I added the counterpart curl command now. Basically, in some systems the curl is not installed by default, and my experience is wget is ‘always there’.

If the command is working, then you can of course discard the results of the command (so, make cron not mail you the results of curl or use wget’s --delete-after parameter).

[clearlytechnical]

Hey, thanks a lot for testing! In the meantime, I created a wiki page on configuration, but it seems you managed to figure it all out.

It seems I need to put in some debug info in the scripts; I’ll do that right away and ping you back here.

In the meantime, please check whether this is not a problem with accessing the IMAP server with encryption (the default behaviour of CiviMail Processor). Also, make sure you have the domain set properly in CiviCRM (but if the emails are not being moved to CiviMail.processed and CiviMail.ignored folders – which they aren’t, if I get you right – then it’s a problem on an earlier stage).

I use IMAP with encryption by default, but I will double check this and also the domain. It might be a good idea to provide a sample of info for the perfect MailStore.php configuration. So in the wiki, we might provide a section:

1. IMAP info

2. Joomla/Drupal mail account with advanced access info

3. exactly where to put the info in the mailstore.php file which correlates to 1 and 2 above

This will help reduce syntax errors maybe and help newbies who don't understand what they are doing and just need more of a direct how to. I would fall under this category, lol.

UPDATE:

Also, and this lends to my claim about being a noob as stated above, lol. But why do we need a cron job? Isn't there a way to "push" or initiate the script based on an event trigger?

Can't we run this as a service and if new mail arrives then fire off the script?

Another thought that comes to mind is for those who use RBL's or SPAMASSASSIN. Couldn't we fire the script by having it launch when SA is called? I know we can do granular SA for each domain and even just for a single account on that domain. This way, when we check it with SA, we can use a filter to immediately throw it to the script for processing?

Again, I am sure this has all been thought of an there are reasons. I just want to share and learn and either know why it can't, shouldn't, or is being done.


Thanks,

[Piotr Szotkowski]

I use IMAP with encryption by default, but I will double check this and also the domain. It might be a good idea to provide a sample of info for the perfect MailStore.php configuration.

Have you seen the last version of CRM/Mailing/MailStore.php (the one with debugging)? I added explicit variables for server/username/password…

So in the wiki, we might provide a section:

1. IMAP info

2. Joomla/Drupal mail account with advanced access info

I’m not sure what these ‘Joomla/Drupal mail account’ are – can you elaborate?

3. exactly where to put the info in the MailStore.php file which correlates to 1 and 2 above

I added some info in the wiki, but do feel free to edit as appropriate.

This will help reduce syntax errors maybe and help newbies who don't understand what they are doing and just need more of a direct how to.

Well, technically, CiviMail Processor is a ‘preview’ in CiviCRM 2.1 (for the advanced users who are not afraid to edit PHP files). Fortunately, CiviCRM 2.2 will have a nice GUI to edit these settings.

But why do we need a cron job? Isn't there a way to "push" or initiate the script based on an event trigger?

There isn’t, without hooking somehow to the mail server (or without the mail server allowing this explicitly, which we can’t assume).

Another thought that comes to mind is for those who use RBL's or SPAMASSASSIN. Couldn't we fire the script by having it launch when SA is called? I know we can do granular SA for each domain and even just for a single account on that domain. This way, when we check it with SA, we can use a filter to immediately throw it to the script for processing?

This is exactly what our patched AMaViS version does – checks for viruses (if so configured) and parses for CiviMail-related stuff the moment the mail arrives. But installing AMaViS is definitely non-trivial (that’s what caused the need for imap2soap, and later CiviMail Processor).

[nocabt]

Hi, this is great, thank you for working on it.  When I attempt to access the CiviMailProcessor.php file for testing, I get the error I've pasted below-

connecting to server, authenticating as username and selecting Inbox
Fatal error: Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. Failed to connect to the server: server:993.' in /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/packages/ezc/Mail/src/transports/transport_connection.php:96 Stack trace: #0 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/packages/ezc/Mail/src/transports/imap/imap_transport.php(212): ezcMailTransportConnection->__construct('server', 993, Object(ezcMailImapTransportOptions)) #1 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/CRM/Mailing/MailStore/Imap.php(57): ezcMailImapTransport->__construct('server', NULL, Array) #2 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/CRM/Mailing/MailStore.php(60): CRM_Mailing_MailStore_Imap->__construct('server', 'username', 'password', true, 'Inbox') #3 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/bin/CiviMailProcessor.php(48): CRM_Mailing_MailSto in /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/packages/ezc/Mail/src/transports/transport_connection.php on line 96

Thanks,

Thomas

[Piotr Szotkowski]

Fatal error: Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. Failed to connect to the server: server:993.'

This means you either didn’t mean to use IMAP or didn’t edit IMAP’s options – both things are to be edited in CRM/Mailing/MailStore.php (the server:993 above means you’re trying to connect with server named ‘server’ using SSL).

[nocabt]

Duh, I had edited the MailStore.php file but uploaded it to the wrong dir, so it was using the default, unedited one.  Now, I get this response:

connecting to imap.gmail.com, authenticating as user@domain.org and selecting Inbox mailboxes found: INBOX, [Gmail]/All Mail, [Gmail]/Drafts, [Gmail]/Sent Mail, [Gmail]/Spam, [Gmail]/Starred, [Gmail]/Trash

Which I assume is correct.  Thanks again.

[nocabt]

One more question, will this solution work  on a CiviCRM 2.0.x install?

Thanks,

Thomas