Author Topic: PHP-based solution for the return channel (Read 36106 times)

Piotr Szotkowski · October 16, 2008, 03:52:24 pm

As some of you know, we’re coming up with a PHP-based solution for handling CiviMail’s return channel (which will work as imap2soap does, but should be much easier to setup and also work with POP3 and Maildir email sources). The relevant issue is CRM-3616.

This will be a part of CiviCRM 2.2, but for those who would like to alpha/beta-test an unsupported version on their 2.1 installations, the files can be fetched from our repository:

bin/CiviMailProcessor.php
CRM/Mailing/MailStore.php
all the files from CRM/Mailing/MailStore

Eventually, for 2.2, these files will retrieve the configuration from the database, but for your purpose please edit the CRM/Mailing/MailStore.php’s getStore() method.

Do feel free to ask any questions (and post any comments!) in this thread.

Note: This is by far not finished, but it did work for me at least once for all of IMAP/POP3/Maildir.

Edit: As I want to divert this for trunk, I committed a 2.1-compatible version to the v2.1 branch. The links above are updated to point to it.

clearlytechnical · October 22, 2008, 04:45:37 pm

First, thank you very much.

Second, I am going to give this the college try. My first question would be:

How do we initiate this? If I copy the files to the appropriate directories, then what next?

My assumption is that CivicMailProcessor.php is a modified version of an existing file and is being called somehow?

Not certain how to fire the event to cause this to pull from imap. Do I run a cron job?

Also, do I make a directory called MailStore and put those files you provided in there? (eg POP3, etc.)

Ok, off to work I go.

UPDATE:

Is there a Debugging tool for this? I can't tell if anything is happening.

I tried sending an email with verp info, but nothing seems to have processed.

clearlytechnical · October 22, 2008, 05:01:30 pm

Ok, I put all the files in the appropriate directories that mirror that of the branch.

I edited the method to reflect the server/user/pass as per your instructions.

Then I tried to go to the bin directory and just fire up the CiviMailProcessor.php and I received this message (after getting past safe mode issues and rights issues).

------ERROR BEGIN-------

ERROR: You need to send a valid user name and password to execute this file

------ERROR END-------

clearlytechnical · October 22, 2008, 05:49:44 pm

Ok, regarding the ERROR I was getting, please see this link:

http://forum.civicrm.org/index.php?topic=4239.0

I am going to try to call the CiviMailProcessor.php using curl and append the Joomla username/password file that I had originally setup for imap2soap.pl

I will report back with my findings.

UPDATE:

Curl gave me an error about username and password, but from within a browser it does not give me an error.

UPDATE:

If you use Curl, make sure you put the entire URL string in Quotes e.g.

curl "http://domain.com/blahblah?username=whatever"

If you don't put the quotes in, it will throw an error. This is why it worked via the browser, but not curl. Please be aware that I am an end user, so while many of you know the proper syntax for curl, others who are just administrators, etc. may find this information helpful, so please forgive the abundant posting ;-)

clearlytechnical · October 22, 2008, 05:55:51 pm

Ok, here is an update. (I am running Joomla 1.5.7)

I tried to call the CiviMailProcessor.php from my browser with the following (please note I changed the domain/username/password to protect client).

http://www.DOMAIN.com/administrator/components/com_civicrm/civicrm/bin/CiviMailProcessor.php?name=civicrm&pass=secretsquirrel

This time I did not get an error.

I will now check my mailbox, clean it up of all files. Then generate some verp (or whatever they are called) type emails to my imap/catchall and see if they get processed and report back.

Sorry for all the posts, just trying to help.

UPDATE:

Nothing seems to be happening. Will pick it up later. It would be helpful if I had a procedure to do or a way to debug this. Thank you again for your contribution Piotr.

UPDATE:
Ok, here is what I did to test.

1. Installed the scripts and set rights to appropriate directories as per the branch.

2. setup a test contact and put in a sample of all tokens

3. Sent a test email. then selected to reply to the token starting with o to "opt out"

4. after it arrived in my imap account, i then ran the script using Curl as per the string provided herein this thread.

5. went back to check the test contact to see if opt out was checked, it was not :-(

So I am stuck now and will wait to hear back from you. Let me know if you have any questions. Thank you.

Piotr Szotkowski · October 23, 2008, 05:26:11 pm

Hey, thanks a lot for testing! In the meantime, I created a wiki page on configuration, but it seems you managed to figure it all out.

It seems I need to put in some debug info in the scripts; I’ll do that right away and ping you back here.

In the meantime, please check whether this is not a problem with accessing the IMAP server with encryption (the default behaviour of CiviMail Processor). Also, make sure you have the domain set properly in CiviCRM (but if the emails are not being moved to CiviMail.processed and CiviMail.ignored folders – which they aren’t, if I get you right – then it’s a problem on an earlier stage).

Piotr Szotkowski · October 23, 2008, 09:05:02 pm

Quote from: Piotr Szotkowski on October 23, 2008, 05:26:11 pm

It seems I need to put in some debug info in the scripts; I’ll do that right away and ping you back here.

I just did in the recent commit to the v2.1 branch. Please try the files now (you might want to replace all of them and just re-edit CRM/Mailing/MailStore.php to add your settings back).

clearlytechnical · October 24, 2008, 07:14:33 am

Piotr,

Thank you for the prompt reply and follow up. Nice wiki!

I will do as you suggest and report back in this message with "UPDATE"s.

Also, is there an actually test of some kind or procedure you recommend to test this? Or was what I did in the previous post sufficient?

Thanks!

UPDATE:
I see in the WIKI you made, that you are using WGET for the cron job. Could you brief me on why wget verses CURL?

When I used WGET, it just downloaded the file and saved it. (whatever that page would have been or whatever.)

Piotr Szotkowski · October 24, 2008, 09:21:28 am

Quote from: clearlytechnical on October 24, 2008, 07:14:33 am

Also, is there an actually test of some kind or procedure you recommend to test this? Or was what I did in the previous post sufficient?

It was sufficient; hopefully the debug messages I added (controlled with the $_debug variable in CRM/Mailing/MailStore.php for 2.1) will shine some light on where the problems lie.

Quote from: clearlytechnical on October 24, 2008, 07:14:33 am

I see in the WIKI you made, that you are using WGET for the cron job. Could you brief me on why wget verses CURL?

I added the counterpart curl command now.

Basically, in some systems the curl is not installed by default, and my experience is wget is ‘always there’.

If the command is working, then you can of course discard the results of the command (so, make cron not mail you the results of curl or use wget’s --delete-after parameter).

clearlytechnical · October 25, 2008, 09:48:06 am

Quote from: Piotr Szotkowski on October 23, 2008, 05:26:11 pm

Hey, thanks a lot for testing! In the meantime, I created a wiki page on configuration, but it seems you managed to figure it all out.

It seems I need to put in some debug info in the scripts; I’ll do that right away and ping you back here.

In the meantime, please check whether this is not a problem with accessing the IMAP server with encryption (the default behaviour of CiviMail Processor). Also, make sure you have the domain set properly in CiviCRM (but if the emails are not being moved to CiviMail.processed and CiviMail.ignored folders – which they aren’t, if I get you right – then it’s a problem on an earlier stage).

I use IMAP with encryption by default, but I will double check this and also the domain. It might be a good idea to provide a sample of info for the perfect MailStore.php configuration. So in the wiki, we might provide a section:

1. IMAP info

2. Joomla/Drupal mail account with advanced access info

3. exactly where to put the info in the mailstore.php file which correlates to 1 and 2 above

This will help reduce syntax errors maybe and help newbies who don't understand what they are doing and just need more of a direct how to. I would fall under this category, lol.

UPDATE:

Also, and this lends to my claim about being a noob as stated above, lol. But why do we need a cron job? Isn't there a way to "push" or initiate the script based on an event trigger?

Can't we run this as a service and if new mail arrives then fire off the script?

Another thought that comes to mind is for those who use RBL's or SPAMASSASSIN. Couldn't we fire the script by having it launch when SA is called? I know we can do granular SA for each domain and even just for a single account on that domain. This way, when we check it with SA, we can use a filter to immediately throw it to the script for processing?

Again, I am sure this has all been thought of an there are reasons. I just want to share and learn and either know why it can't, shouldn't, or is being done.

Thanks,

Piotr Szotkowski · October 27, 2008, 11:48:33 am

Quote from: clearlytechnical on October 25, 2008, 09:48:06 am

I use IMAP with encryption by default, but I will double check this and also the domain. It might be a good idea to provide a sample of info for the perfect MailStore.php configuration.

Have you seen the last version of CRM/Mailing/MailStore.php (the one with debugging)? I added explicit variables for server/username/password…

Quote from: clearlytechnical on October 25, 2008, 09:48:06 am

So in the wiki, we might provide a section:

1. IMAP info

2. Joomla/Drupal mail account with advanced access info

I’m not sure what these ‘Joomla/Drupal mail account’ are – can you elaborate?

Quote from: clearlytechnical on October 25, 2008, 09:48:06 am

3. exactly where to put the info in the MailStore.php file which correlates to 1 and 2 above

I added some info in the wiki, but do feel free to edit as appropriate.

Quote from: clearlytechnical on October 25, 2008, 09:48:06 am

This will help reduce syntax errors maybe and help newbies who don't understand what they are doing and just need more of a direct how to.

Well, technically, CiviMail Processor is a ‘preview’ in CiviCRM 2.1 (for the advanced users who are not afraid to edit PHP files). Fortunately, CiviCRM 2.2 will have a nice GUI to edit these settings.

Quote from: clearlytechnical on October 25, 2008, 09:48:06 am

But why do we need a cron job? Isn't there a way to "push" or initiate the script based on an event trigger?

There isn’t, without hooking somehow to the mail server (or without the mail server allowing this explicitly, which we can’t assume).

Quote from: clearlytechnical on October 25, 2008, 09:48:06 am

Another thought that comes to mind is for those who use RBL's or SPAMASSASSIN. Couldn't we fire the script by having it launch when SA is called? I know we can do granular SA for each domain and even just for a single account on that domain. This way, when we check it with SA, we can use a filter to immediately throw it to the script for processing?

This is exactly what our patched AMaViS version does – checks for viruses (if so configured) and parses for CiviMail-related stuff the moment the mail arrives. But installing AMaViS is definitely non-trivial (that’s what caused the need for imap2soap, and later CiviMail Processor).

nocabt · October 29, 2008, 07:29:27 pm

Hi, this is great, thank you for working on it. When I attempt to access the CiviMailProcessor.php file for testing, I get the error I've pasted below-

connecting to server, authenticating as username and selecting Inbox
Fatal error: Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. Failed to connect to the server: server:993.' in /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/packages/ezc/Mail/src/transports/transport_connection.php:96 Stack trace: #0 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/packages/ezc/Mail/src/transports/imap/imap_transport.php(212): ezcMailTransportConnection->__construct('server', 993, Object(ezcMailImapTransportOptions)) #1 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/CRM/Mailing/MailStore/Imap.php(57): ezcMailImapTransport->__construct('server', NULL, Array) #2 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/CRM/Mailing/MailStore.php(60): CRM_Mailing_MailStore_Imap->__construct('server', 'username', 'password', true, 'Inbox') #3 /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/bin/CiviMailProcessor.php(48): CRM_Mailing_MailSto in /home/texasgsa/public_html/administrator/components/com_civicrm/civicrm/packages/ezc/Mail/src/transports/transport_connection.php on line 96

Thanks,

Thomas

Piotr Szotkowski · October 30, 2008, 07:43:54 am

Quote from: nocabt on October 29, 2008, 07:29:27 pm

Fatal error: Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. Failed to connect to the server: server:993.'

This means you either didn’t mean to use IMAP or didn’t edit IMAP’s options – both things are to be edited in CRM/Mailing/MailStore.php (the server:993 above means you’re trying to connect with server named ‘server’ using SSL).

nocabt · October 30, 2008, 08:41:02 am

Duh, I had edited the MailStore.php file but uploaded it to the wrong dir, so it was using the default, unedited one. Now, I get this response:

connecting to imap.gmail.com, authenticating as user@domain.org and selecting Inbox mailboxes found: INBOX, [Gmail]/All Mail, [Gmail]/Drafts, [Gmail]/Sent Mail, [Gmail]/Spam, [Gmail]/Starred, [Gmail]/Trash

Which I assume is correct. Thanks again.

nocabt · October 30, 2008, 08:42:55 am

One more question, will this solution work on a CiviCRM 2.0.x install?

Thanks,

Thomas

CiviCRM Community Forums (archive)

News:

Author Topic: PHP-based solution for the return channel (Read 36106 times)

Piotr Szotkowski

PHP-based solution for the return channel

clearlytechnical

Re: PHP-based solution for the return channel

clearlytechnical

Re: PHP-based solution for the return channel

clearlytechnical

Re: PHP-based solution for the return channel

clearlytechnical

Re: PHP-based solution for the return channel

Piotr Szotkowski

Re: PHP-based solution for the return channel

Piotr Szotkowski

Re: PHP-based solution for the return channel

clearlytechnical

Re: PHP-based solution for the return channel

Piotr Szotkowski

Re: PHP-based solution for the return channel

clearlytechnical

Re: PHP-based solution for the return channel

Piotr Szotkowski

Re: PHP-based solution for the return channel

nocabt

Re: PHP-based solution for the return channel

Piotr Szotkowski

Re: PHP-based solution for the return channel

nocabt

Re: PHP-based solution for the return channel

nocabt

Re: PHP-based solution for the return channel