Mail Processor not doing anything?

[mac]

I've set up CiviMail and I can send emails. The emails arrive, the opening gets tracked properly.

One question first:
Can somebody explain to me what the trick with the "Note that to catch bounces, you must use a catchall account unless you use addresses of the form civimailuser+b.123.456.abcdef0123456789@xxx.org." is?

Is this supposed to mean that if you configure your box properly, you don't need a catchall and the emails will go to the box civimailuser@xxx.org? If yes, can you do this for POP/IMAP too?


Now the trouble:
I don't seem to be able to track any replies and stuff and I think this is because the Mail Processor is actually not doing anything when I run it.

Two questions:
1) How do I know that the processor ran at all?
2) How do I know it actually did something and whether it run into problems?

If I run the script as described in the manual once with correct and once with wrong parameters, I don't seem to be able to see any difference in the browser output ... ?

I'm pretty sure it's a stupid small thing that I'm doing wrong but I just can't figure out what this is about.

Thanks in advance.

PS: I have tried to execute the CiviMailprocessor.php directly through putty (I'm not sure whether this was a good idea?) and the outcome is the following:

$ php CiviMailprocessor.php?name=name&pass=pass&key=key
[1] 21640
[2] 21641

Status: 404
Content-type: text/html

No input file specified.

[Piotr Szotkowski]

I've set up CiviMail and I can send emails. The emails arrive, the opening gets tracked properly.

One question first:
Can somebody explain to me what the trick with the "Note that to catch bounces, you must use a catchall account unless you use addresses of the form civimailuser+b.123.456.abcdef0123456789@xxx.org." is?

Is this supposed to mean that if you configure your box properly, you don't need a catchall and the emails will go to the box civimailuser@xxx.org? If yes, can you do this for POP/IMAP too?

Yes, if you configure your SMTP (receiving) server properly (Postfix, for example, has this enabled by default; GMail accounts as well) this is exactly how it’s supposed to work. Note that this does not affect POP/IMAP in any way; you simply poll the civimailuser@xxx.org account as usual.

Now the trouble:
I don't seem to be able to track any replies and stuff and I think this is because the Mail Processor is actually not doing anything when I run it.

Two questions:
1) How do I know that the processor ran at all?
2) How do I know it actually did something and whether it run into problems?

If I run the script as described in the manual once with correct and once with wrong parameters, I don't seem to be able to see any difference in the browser output ... ?

First, CiviMail Processor uses the CRM_Mailing_MailStore class and the debug output is turned on by default, so running it from the command line should output the info on what’s happening.

Second, I’m not sure what you mean by parameters – you adjust the account settings to poll in Administer → CiviMail → Mail Accounts, do you mean that place?

PS: I have tried to execute the CiviMailprocessor.php directly through putty (I'm not sure whether this was a good idea?) and the outcome is the following:

$ php CiviMailprocessor.php?name=name&pass=pass&key=key
[1] 21640
[2] 21641

Status: 404
Content-type: text/html

No input file specified.

On the command line the & means ‘run another process’, hence you actually try to run three processes above (the first ones are ‘21640’ and ‘21641’). Try quoting the script + params:

$ php 'CiviMailprocessor.php?name=name&pass=pass&key=key'

[mac]

First: Piotr, dzieki za szybka odpowiedz

Yes, if you configure your SMTP (receiving) server properly (Postfix, for example, has this enabled by default; GMail accounts as well) this is exactly how it’s supposed to work. Note that this does not affect POP/IMAP in any way; you simply poll the civimailuser@xxx.org account as usual.

I think we'll stick to POP/IMAP. I was just wondering if there was an option how to omit setting up a catch all on the box.

First, CiviMail Processor uses the CRM_Mailing_MailStore class and the debug output is turned on by default, so running it from the command line should output the info on what’s happening.

well ... I tried to run it from the command line, but this doesn't seem to help me in terms of knowing what I'm doing wrong (see below).

Second, I’m not sure what you mean by parameters – you adjust the account settings to poll in Administer → CiviMail → Mail Accounts, do you mean that place?

By "parameters" I meant the php parameters for the mail processor = drupal username, passwd and the key. Whether I provide the wrong or the right passwd for my account, the outcome is the same = nothing.

On the command line the & means ‘run another process’, hence you actually try to run three processes above (the first ones are ‘21640’ and ‘21641’). Try quoting the script + params:

$ php 'CiviMailprocessor.php?name=name&pass=pass&key=key'

thanks, I tried to do that. The outcome is basically the same though:

$ php 'CiviMailprocessor.php?name=name&pass=pass&key=key'
Status: 404
Content-type: text/html

No input file specified.

Can you please tell me (some tutorial/forum links?) how to learn why nothing is happening? In particular, where do I find the logs that you mentioned previously?

Just to be sure: Does it matter whether I run the CiviMailprocessor.php from the bin directory directly, or whether I run it from the source directory and specify the path to the bin directory?

EDIT: I have now tried to set up a cron job on within our hosting environment on goDaddy as described in the wiki:

wget 'http://mysite.com/civicrm/bin/civimail.cronjob.php?name=<name>&pass=<pass>&key=<key>'

The outcome is the following:

--12:30:02--  http://mysite.com/civicrm/bin/CiviMailProcessor.php?name=<name>&pass=<pass>&key=<key>
Resolving mysite.com... my.ip
Connecting to mysite.com|my.ip|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
12:25:04 ERROR 403: Forbidden.

And the Mail Processor:

wget 'http://mysite.com/civicrm/bin/CiviMailProcessor.php?name=<name>&pass=<pass>&key=<key>'

with the outcome

--12:25:01--  http://mysite.com/civicrm/bin/CiviMailProcessor.php?name=<name>&pass=<pass>&key=<key>
Resolving mysite.com... my.ip
Connecting to mysite.com|my.ip|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
12:25:04 ERROR 403: Forbidden.

What could I be doing wrong? And how should I check it?

I have one idea: What if the key was not recognized by CiviCRM? After I have set the key in the civicrm.settings.php file, do I have to do something special so that the new key is recognized by CiviCRM? I tried to run the upgrade script, but maybe that wasn't enough?

[Piotr Szotkowski]

First: Piotr, dzieki za szybka odpowiedz

I think we'll stick to POP/IMAP. I was just wondering if there was an option how to omit setting up a catch all on the box.

Sure you’ll stick to POP/IMAP, I was just pointing out that this has nothing to do with the protocol(s)/server(s) used to access the emails (POP/IMAP), but rather with the SMTP server, which receives the emails and routes to the proper account. Again: having the localpart support (i.e., account+any-garbage@example.org being routed to the account@example.org account) is enabled by default in Postfix, works in GMail (who might be using Postfix themselves, of course, but we don’t know) and I assume might be enabled by default in most SMTP servers, so as well yours.

thanks, I tried to do that. The outcome is basically the same though:

$ php 'CiviMailprocessor.php?name=name&pass=pass&key=key'
Status: 404
Content-type: text/html

No input file specified.

Yes, because I’m an idiot, sorry. The proper way to run PHP scripts from the commandline in this case is

$ php CiviMailprocessor.php name=name pass=pass key=key

Unfortunately, this does not seem to work for me (under Ubuntu). I’ll try to figure out why, but maybe it’ll work for you in the meantime.

Can you please tell me (some tutorial/forum links?) how to learn why nothing is happening? In particular, where do I find the logs that you mentioned previously?

The above should either work or tell you that the provided credentials are wrong.

Just to be sure: Does it matter whether I run the CiviMailprocessor.php from the bin directory directly, or whether I run it from the source directory and specify the path to the bin directory?

I think it needs to be ran from the bin dir, as it tries to include ../civicrm.config.php explicitly.

wget 'http://mysite.com/civicrm/bin/civimail.cronjob.php?name=<name>&pass=<pass>&key=<key>'

wget 'http://mysite.com/civicrm/bin/CiviMailProcessor.php?name=<name>&pass=<pass>&key=<key>'

The 403 HTTP errors mean the access is denied. Can you try visiting these URLs with your browser? (Note that the problem might be either due to webserver config or due to name/pass/key problems.)

I have one idea: What if the key was not recognized by CiviCRM? After I have set the key in the civicrm.settings.php file, do I have to do something special so that the new key is recognized by CiviCRM? I tried to run the upgrade script, but maybe that wasn't enough?

No, the key should ‘just work’. The only thing I can think off the top of my head is some special chars escaping (a + in the pass or key will be treated as space after going through the web server, for example, and needs to be escaped to %2B), so it’s best if you can test with alphanumeric user/pass/key for starters.

[Piotr Szotkowski]

The proper way to run PHP scripts from the commandline in this case is

$ php CiviMailprocessor.php name=name pass=pass key=key

Unfortunately, this does not seem to work for me (under Ubuntu). I’ll try to figure out why, but maybe it’ll work for you in the meantime.

Ok, so after a bit of more digging I might’ve remember wrong, and the php command does not, by itself, put these params in $_REQUEST. Your best bet for now is to use wget or curl to simulate a browser request; in the future, we might end up implementing something like this to parse $_SERVER['argv'] contents into $_REQUEST.

[mac]

Sure you’ll stick to POP/IMAP, I was just pointing out that this has nothing to do with the protocol(s)/server(s) used to access the emails (POP/IMAP), but rather with the SMTP server, which receives the emails and routes to the proper account. Again: having the localpart support (i.e., account+any-garbage@example.org being routed to the account@example.org account) is enabled by default in Postfix, works in GMail (who might be using Postfix themselves, of course, but we don’t know) and I assume might be enabled by default in most SMTP servers, so as well yours.

Our provider does not support the localpart option so we'll have to live with the catch all (and maybe some additional filters).

The above should either work or tell you that the provided credentials are wrong.

I'm 99.9% sure that I'm not making a mistake there. How can I figure out though whether CiviCRM thinks that they are not right?

The 403 HTTP errors mean the access is denied. Can you try visiting these URLs with your browser? (Note that the problem might be either due to webserver config or due to name/pass/key problems.)

The access to the directory should be granted properly. If I run the url in the browser, I come to the CiviCRM dashboard page without any additional notice/information displayed. It's basically the same page as if I would go to the dashboard page directly. That's why I was asking initially about how to figure out whether it's even doing anything.

The difference though between running it in the browser vs. running it with wget is that when I run it in the browser, there's not even an error in the Drupal status log. Whereas if I run the URL through wget it ends up in the log as "access denied".

No, the key should ‘just work’. The only thing I can think off the top of my head is some special chars escaping (a + in the pass or key will be treated as space after going through the web server, for example, and needs to be escaped to %2B), so it’s best if you can test with alphanumeric user/pass/key for starters.

All parameters (name, passwd, and key) are only alphanumeric, so I don't think there's anything that should be escaped.

Any ideas how I should go from here?

EDIT: Can it be that I'm executing the wrong URL?

If my dashboard's URL is "http://mysite.com/civicrm/dashboard?reset=1", is the MailProcessor URL supposed to be "http://mysite.com/civicrm/bin/CiviMailProcessor.php?name=<name>&pass=<pass>&key=<key>", or am I maybe screwing up the path/URL to the MailProcessor?

EDIT II: I checked my theory, and I was right. The path was wrong. The path has to be:
http://mysite.com/sites/all/modules/civicrm/bin/CiviMailProcessor.php?name=name&pass=pass&key=key

Now I'm getting the following response:

$ wget 'http://mysite.com/sites/all/modules/civicrm/bin/CiviMailProcessor.php?name=name&pass=pass&key=key'
--15:21:07--  http://mysite.com/sites/all/modules/civicrm/bin/CiviMailProcessor.php?name=name&pass=pass&key=key
Resolving mysite.com... myip
Connecting to mysite.com|myip|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `CiviMailProcessor.php?name=name&pass=pass&key=key.1'

    [ <=>                                                                                                                 ] 1,414       --.-K/s   in 0s

15:21:07 (70.4 MB/s) - `CiviMailProcessor.php?name=name&pass=pass&key=key.1' saved [1414]

If I put the same URL into the browser now, I get the following response:

connecting to mymailserver.com, authenticating as user@mysite.com and selecting INBOX
mailboxes found: INBOX, Bulk Mail, Drafts, Email_Templates, Send_Later, Sent Items, Trash

Fatal error: Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. The IMAP server could not create mailbox 'INBOX/CiviMail/ignored': A0006 NO [ALERT] Folders cannot be created in a system folder.' in /sites/all/modules/civicrm/packages/ezc/Mail/src/transports/imap/imap_transport.php:746

I found this thread about a similar problem, and I'm trying to work things out now ...

EDIT III: I figured it out! The problem had actually nothing to do with the fact that the path 'INBOX/CiviMail/ignored' could not be found. I don't quite understand where this message is coming from. Cause when I created the path (case sensitive), the problem was still there.

So I went to Administer CiviCRM -> CiviMail -> Mail Accounts. In the settings of the account I'm using, the Source was empty. I didn't change it initially, cause it says that

Folder to poll from when using IMAP (will default to Inbox when empty), path to poll from when using Maildir, etc..

But the fact is, the it was pulling from "INBOX". So I put in "Inbox" for the source. When I ran the processor another time, everything worked out! The path "Inbox/CiviMail/ignored" was created without any trouble, and so was "Inbox/CiviMail/processed". The emails were pulled and analyzed and everything seems to work.

I'm actually waiting for the results of the scripts running through a cron job, but I assume this will work out.

Thanks for the help Piotr.

PS: Maybe you want to make the wiki/manual a little bit more precise on this topic ...

[Piotr Szotkowski]

connecting to mymailserver.com, authenticating as user@mysite.com and selecting INBOX
mailboxes found: INBOX, Bulk Mail, Drafts, Email_Templates, Send_Later, Sent Items, Trash

Fatal error: Uncaught exception 'ezcMailTransportException' with message 'An error occured while sending or receiving mail. The IMAP server could not create mailbox 'INBOX/CiviMail/ignored': A0006 NO [ALERT] Folders cannot be created in a system folder.' in /sites/all/modules/civicrm/packages/ezc/Mail/src/transports/imap/imap_transport.php:746

Note that this error is about the IMAP server, and the failing operation is CiviMail Processor’s attempt to create an IMAP folder of INBOX/CiviMail/ignored.

EDIT III: I figured it out!

Yay!

In the settings of the account I'm using, the Source was empty. I didn't change it initially, cause it says that

Folder to poll from when using IMAP (will default to Inbox when empty), path to poll from when using Maildir, etc..

But the fact is, the it was pulling from "INBOX".

I vaguely remember that we changed the default from Inbox to INBOX; IMAP servers should be case-insensitive about folder names, but I vaguely remember there was one that insisted on the name being INBOX. I guess your does the opposite.

Thanks for the bug info, CiviCRM 2.2.9 and 3.0.beta2 will have the info on default fixed.

PS: Maybe you want to make the wiki/manual a little bit more precise on this topic ...

Note that it’s a wiki, and as such it’s user-editable. We definitely should provide better docs, but quite often it’s the users who can phrase the bummer-solvers the best.