CiviCRM Community Forums (archive)

A while ago I posted scripts for cron jobs to send and receive emails. This post supersedes that with some improvements (especially the use of EmailProcessor.php which replaces CiviMailProcessor.php in 3.4).

The value-add with these scripts is the error checking. They only notify the administrator if something has gone wrong (they take advantage of the fact that under Linux, cron only sends an email of it's activity if the script it runs produces a non-empty result), so one doesn't get bombarded with stuff. (I run these every 5 minutes or so.)

The scripts were written for Drupal 6.x running in a Linux environment. They assume you are sending email using SMTP and receiving it via IMAP. They support receiving both CiviMail bounces etc, and Email-to-Activity mailboxes. They should also work for Joomla! 1.5.x (after changing ThePath in the configuration file).

You should understand what it is you're automating before you start. Read http://wiki.civicrm.org/confluence/display/CRMDOC/Command-line+Script+Configuration. Don't start with the scripts until you can run the civimail.cronjob.php and EmailProcessor.php jobs by hand from your browser.

This is what the scripts do ...

• Use wget to run the CiviCRM script and get the results
• Analyse the results and produce a non-empty output if there is a problem
• Clean up after themselves
• (The cron daemon will send an email if the output is non-empty. You need to have the cron job run by a user whose email you will receive.)

All the script files should be placed in the same directory and the configuration script is called 'configuration.sh'. If you change either of these details you will need to edit how the configuration script is sourced by the other scripts.

The first script is the configuration script. Replace the "..." with the values you used to run the scripts manually. Replace the Temp Directory to suit your environment and change the path for Joomla!.
#!/bin/bash
#
# Configuration parameters used by the CiviCRM scripts
#

TmpDirectory=/tmp

TheCommand=wget
TheHost=https://...
ThePath=/sites/all/modules/civicrm/bin
TheUser=...
ThePass=...
TheKey=...
Next is the script for sending CiviMails. As the wget command returns an empty file when all is well, this script is simpler than the receiving script that follows it. If wget fails, or a non-empty result is returned, this information is output to be sent to the administrator.
#!/bin/bash
#
# This script will send CiviMail emails.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviMail send wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

SendUrl="${TheHost}${ThePath}/civimail.cronjob.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log

$TheCommand "${SendUrl}" --output-document=$Document --output-file=$Log
Result=$?

if [[ $Result != 0 ]]
then
        echo Failed to connect to CiviCRM
        echo ' '
        echo ---- Return-code returned by wget ----
        echo $Result
        echo ' '
        echo ---- Contents of wget log ----
        cat $Log
elif [[ -s $Document ]]
then
        echo CiviCRM returned a non-empty result
        echo ' '
        echo ---- Response from CiviCRM ----
        cat $Document
        echo ' '
        echo ---- Contents of wget log ----
        cat $Log
elif [[ ! -e $Document ]]
then
        echo CiviCRM did not return a result
        echo ' '
        echo ---- Contents of wget log ----
        cat $Log
fi

rm -f $Document $Log
Next is the script for receiving CiviMails. As wget returns a non-empty result, this needs to be parsed to detect any errors. This is done with an awk script.
#!/bin/bash
#
# This script will receive CiviMail emails.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviMail receive wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

ReceiveUrl="${TheHost}${ThePath}/EmailProcessor.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log
StatusLog=${FilePattern}.status.log

$TheCommand "${ReceiveUrl}" --output-document=$Document --output-file=$Log
Result=$?

if [[ $Result != 0 ]]
then
        echo Failed to connect to CiviCRM
        echo ' '
        echo ---- Return-code returned by wget ----
        echo $Result
        echo ' '
        echo ---- Contents of wget log ----
        cat $Log
elif [[ -s $Document ]]
then
        awk -f $Directory/mail-receive-status.awk $Document > $StatusLog
        if [[ -s $StatusLog ]]
        then
                cat $StatusLog
                echo --------------------
                cat $Document
        fi
elif [[ -e $Document ]]
then
        echo CiviCRM returned an empty result
        echo ' '
        echo ---- Contents of wget log ----
        cat $Log
else
        echo CiviCRM did not return a result
        echo ' '
        echo ---- Contents of wget log ----
        cat $Log
fi

rm -f $Document $Log $StatusLog
Lastly is the awk script for analysing the result of the receiving script. The receiving script given above expects this to be in a file called 'mail-receive-status.awk' in the same directory as itself.
#!/usr/bin/awk
#
# Awk script to interpret the results of a CiviMail receipt
#

# Here is a sample file that should be validated by this script ...
#
# REFERENCE : CONTENTS
# ..........:............
# 1         : connecting to example.com, authenticating as xxxx@example.com and selecting INBOX
# 2         : mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Pro
cessed replies, INBOX.Processed bounces, INBOX
# 3{1}      : fetching 50 messages
# 3{1}.1[1] : retrieving message 2464
# 3{1}.1[2] : retrieving message 2465
# 3{1}.2[1] : setting 2464 as seen and moving it to the processed mailbox
# 3{1}.2[2] : setting 2465 as seen and moving it to the processed mailbox
# 4         : got to the end of the mailbox
#
# To understand the reference column ...
#   * the numbers indicate sequence,
#   * '.' indicates a component of a larger structure,
#   * '[]' indicates repetition one or more times, and
#   * '{}' indicates repetition zero or more times
#

# This script produces an empty file if the input file is valid.
# Otherwise it produces a non-empty file explaining the problem
#

# Specific problems that are targeted ...
#   * Missing line-reference 1, 2, or 4
#   * Mail sent to the 'ignored' mailbox on line-reference 3[].2[]
#   * Invalid content on any line
#

# Program states, and state transitions ...
#
# STATE : AFTER SEEING ...      : VALID NEXT STATES : VALID PREVIOUS STATES
# ......:.......................:...................:......................
# 1     : BEGIN                 : 2                 : -
# 2     : connecting to         : 3                 : 1, 99
# 3     : mailboxes found       : 4, 99             : 2
# 4     : fetching messages     : 5                 : 3, 6
# 5     : retrieving message    : 5, 6, 7           : 4, 5
# 6     : setting as seen       : 4, 6, 99          : 5, 6, 7
# 7     : Processed as Activity : 6                 : 5, 6
# 99    : got to end            : 100, 2            : 3, 6
# 100   : END                   : -                 : 99

BEGIN   {
                state = 1;
                processed = 0;
        }

/^connecting to [a-z.-]+, authenticating as [a-z.@-]+ and selecting [A-Za-z]+$/ {
                if (state != 1 && state != 99)
                        print "Line", NR, "-", "Unexpected 'connecting' message";
                state = 2;
                processed++;
        }

/^mailboxes found: .+$/ {
                if (state != 2)
                        print "Line", NR, "-", "Unexpected 'mailboxes found' message";
                state = 3;
                processed++;
        }

/^fetching [0-9]+ messages$/ {
                if (state != 3 && state != 6)
                        print "Line", NR, "-", "Unexpected 'fetching' message";
                state = 4;
                processed++;
        }

/^retrieving message [0-9]+$/ {
                if (state != 4 && state != 5)
                        print "Line", NR, "-", "Unexpected 'retrieving' message";
                state = 5;
                processed++;
        }

/^setting [0-9]+ as seen and moving it to the processed mailbox$/ {
                if (state != 5 && state != 6 && state != 7)
                        print "Line", NR, "-", "Unexpected 'setting' message";
                state = 6;
                processed++;
        }

/^setting [0-9]+ as seen and moving it to the ignored mailbox$/ {
                if (state != 5 && state != 6 && state != 7)
                        print "Line", NR, "-", "Unexpected 'setting' message";
                state = 6;
                print "Line", NR, "-", "Message ignored by EmailProcessor";
                processed++;
        }

/^Processed as Activity: .+$/ {
                if (state != 5 && state != 6)
                        print "Line", NR, "-", "Unexpected 'processed as activity' message";
                state = 7;
                processed++;
        }

/^got to the end of the mailbox$/ {
                if (state != 3 && state != 6)
                        print "Line", NR, "-", "Unexpected 'got to end' message";
                state = 99;
                processed++;
        }

/^$/    {       # Ignore blank lines
                processed++;
        }

/^.+$/  {       # Report unexpected non-blank lines
                if (NR != processed)
                {
                        print "Line", NR, "-", "Unexpected line:", $0
                        processed++;
                }
        }

END     {
                if (state != 99)
                        print "After line", NR, "-", "Unexpected end-of-file. Expected 'got to end of mailbox' message"
        }
Ken

(If you find these scripts helpful then please applaud. If you improve or extend them, please reply with the details. One extension I think might be helpful is to invoke the scripts from the CLI. But as the scripts aren't running that often, I find that the overhead is OK.)

Xavier,

When I get around to it ...

Hello Ken,

It seems that the compiler does not understand the commands in mail-receive-status.awk

First, I had to comment out all the blank lines.

Second, I have put the first command on one line like: BEGIN { state = 1; processed = 0; }

Now I get the error:
awk: /***/***/***/joomla/administrator/components/com_civicrm/civicrm/bin/Mailing/mail-receive-status-new.awk:50: BEGIN { state = 1; processed = 0; }
awk: /***/***/***/joomla/administrator/components/com_civicrm/civicrm/bin/Mailing/mail-receive-status-new.awk:50:                                    ^ invalid char '
' in expression

I have read that adding a backslash at every line that is continued on the next line might do the job, but this also results in an error: backslash not last character on line

Any idea?

Thanks,
Paul

 

Hello Ken,

I am on a linux server.
What is strange is that I can execute awk from the command line, but not when calling a file out of a shell script.
The hosting responsible even told me they do not support it 

I am looking into a solution. Awk is part of linux so I do not get why it should not work.
If you might know about an alternative for awk, that could help me. I''l report back when I have found a solution.

Best regards,
Paul

Hello Ken,

It's running 

In the beginning, I used a PHP editor, next I moved to WinVi and now I am using nano (http://www.nano-editor.org/).
It's a shitty program to use, but it's not putting unexpected chars where they should not have to be.

I had to put all awk command on a single line which brings your mail-receive-status.awk to 16 lines of coding 
But, as I said already, it's working.

I now encounter the following problem.
As specified by the manual, I created two mail boxes:
- return@***.be which handles the bounces and non-addressed mails
- archief@***.be which handles the auto-filing (which works well).

You can imagine, EmailProcessor handles both mailboxes, which gives me the following result:
Line 2 - Unexpected 'got to the end' message
Line 4 - Unexpected 'got to the end' message
--------------------
connecting to mail.****.be and authenticating as return@****.be
got to the end of the mailbox
connecting to mail.****.be and authenticating as archief@****.be
got to the end of the mailbox

First question: in mail-receive-status.awk you expect line 2 to be
"mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Pro"

Am I missing something in the settings, as apparently, EmailProcessor does not find any mailboxes. Do I have to create them somewhere?

Second question: I suppose I have to rewrite your awk in order to adapt it to the use of several mailboxes. Or am I doing something wrong?

Thanks,
Paul