CiviCRM Community Forums (archive)

I've updated these scripts. See http://forum.civicrm.org/index.php/topic,18816 for details. CiviMailProcessor.php will be superseded by EmailProcessor.php in 3.4, and the mail receiving script doesn't handle the case when emails are being converted into activities.

These are the cron jobs I use for sending and receiving CiviMails. The scripts work for 2.2+ as they use the CiviMailProcessor.php script. The scripts were written for Joomla! 1.5.x running in a Linux environment.

The value-add with these scripts is the error checking. They only notify the administrator if something has gone wrong (they take advantage of the fact that under Linux, cron only sends an email of it's activity if the script it runs produces a non-empty result), so one doesn't get bombarded with stuff. (I run these every 5 minutes or so.)

This is what they do ...

• Use wget to run the CiviCRM script and get the results
• Analyse the results and produce a non-empty output if there is a problem
• Clean up after themselves
• (The cron daemon will send an email if the output is non-empty. You need to have the cron job run by a user whose email you will receive.)

All the scripts are in the same directory and the configuration script is called 'configuration.sh'. If you change either of these details you will need to edit how the configuration script is sourced by the other scripts.

The first script is the configuration script. The variable assignments with the empty RHS will need to be set to suit your environment.
#!/bin/bash
#
# Configuration parameters used by the CiviCRM scripts
#

TmpDirectory=               # Directory for temporary files (need rw access)

TheHost=                    # eg, http://google.com
ThePath=/administrator/components/com_civicrm/civicrm/bin
TheUser=                    # Supply credentials for the Joomla! user
ThePass=                    # under whom the scripts are executed,
TheKey=                     # plus the CiviCRM key

Next is the script for sending CiviMails. As the wget command returns an empty file when all is well, this script is simpler than the receiving script that follows it. If wget fails, or a non-empty result is returned, this information is output to be sent to the administrator.
#!/bin/bash
#
# This script will send CiviMail emails.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviMail send wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

SendUrl="${TheHost}${ThePath}/civimail.cronjob.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log

wget "${SendUrl}" --output-document=$Document --output-file=$Log
Result=$?

if [[ $Result != 0 ]]
then
echo Failed to connect to CiviCRM
echo ' '
echo ---- Return-code returned by wget ----
echo $Result
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ -s $Document ]]
then
echo CiviCRM returned a non-empty result
echo ' '
echo ---- Response from CiviCRM ----
cat $Document
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ ! -e $Document ]]
then
echo CiviCRM did not return a result
echo ' '
echo ---- Contents of wget log ----
cat $Log
fi

rm -f $Document $Log

Next is the script for receiving CiviMails. As wget returns a non-empty result, this needs to be parsed to detect any errors. This is done with an awk script.
#!/bin/bash
#
# This script will receive CiviMail emails.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviMail receive wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

ReceiveUrl="${TheHost}${ThePath}/CiviMailProcessor.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log
StatusLog=${FilePattern}.status.log

wget "${ReceiveUrl}" --output-document=$Document --output-file=$Log
Result=$?

if [[ $Result != 0 ]]
then
echo Failed to connect to CiviCRM
echo ' '
echo ---- Return-code returned by wget ----
echo $Result
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ -s $Document ]]
then
awk -f $Directory/mail-receive-status.awk $Document > $StatusLog
if [[ -s $StatusLog ]]
then
cat $StatusLog
echo --------------------
cat $Document
fi
elif [[ -e $Document ]]
then
echo CiviCRM returned an empty result
echo ' '
echo ---- Contents of wget log ----
cat $Log
else
echo CiviCRM did not return a result
echo ' '
echo ---- Contents of wget log ----
cat $Log
fi

rm -f $Document $Log $StatusLog

Lastly is the awk script for analysing the result of the receiving script. The receiving script given above expects this to be in a file called 'mail-receive-status.awk' in the same directory as itself.
#!/usr/bin/awk
#
# Awk script to interpret the results of a CiviMail receipt
#

# Here is a sample file that should be validated by this script ...
#
# REFERENCE : CONTENTS
# ..........:............
# 1         : connecting to yyy.zzz, authenticating as xxx@yyy.zzz and selecting INBOX
# 2         : mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Processed replies, INBOX.Processed bounces, INBOX
# 3{1}      : fetching 50 messages
# 3{1}.1[1] : retrieving message 2464
# 3{1}.1[2] : retrieving message 2465
# 3{1}.2[1] : setting 2464 as seen and moving it to the processed mailbox
# 3{1}.2[2] : setting 2465 as seen and moving it to the processed mailbox
# 4         : got to the end of the mailbox
#
# To understand the reference column ...
#   * the numbers indicate sequence,
#   * '.' indicates a component of a larger structure,
#   * '[]' indicates repetition one or more times, and
#   * '{}' indicates repetition zero or more times
#

# This script produces an empty file if the input file is valid.
# Otherwise it produces a non-empty file explaining the problem
#

# Specific problems that are targeted ...
#   * Missing line-reference 1, 2, or 4
#   * Mail sent to the 'ignored' mailbox on line-reference 3[].2[]
#   * Invalid content on any line
#

# Program states, and state transitions ...
#
# STATE : AFTER SEEING ...   : VALID NEXT STATES : VALID PREVIOUS STATES
# ......:....................:...................:......................
# 1     : BEGIN              : 2                 : -
# 2     : connecting to      : 3                 : 1
# 3     : mailboxes found    : 4, 7              : 2
# 4     : fetching messages  : 5                 : 3, 6
# 5     : retrieving message : 5, 6              : 4, 5
# 6     : setting as seen    : 4, 6, 7           : 5, 6
# 7     : got to end         : 8                 : 3, 6
# 8     : END                : -                 : 7

BEGIN {
state = 1;
processed = 0;
}

/^connecting to [a-z.-]+, authenticating as [a-z.@-]+ and selecting [A-Za-z]+$/ {
if (state != 1)
print "Line", NR, "-", "Unexpected 'connecting' message";
state = 2;
processed++;
}

/^mailboxes found: .+$/ {
if (state != 2)
print "Line", NR, "-", "Unexpected 'mailboxes found' message";
state = 3;
processed++;
}

/^fetching [0-9]+ messages$/ {
if (state != 3 && state != 6)
print "Line", NR, "-", "Unexpected 'fetching' message";
state = 4;
processed++;
}

/^retrieving message [0-9]+$/ {
if (state != 4 && state != 5)
print "Line", NR, "-", "Unexpected 'retrieving' message";
state = 5;
processed++;
}

/^setting [0-9]+ as seen and moving it to the processed mailbox$/ {
if (state != 5 && state != 6)
print "Line", NR, "-", "Unexpected 'setting' message";
state = 6;
processed++;
}

/^setting [0-9]+ as seen and moving it to the ignored mailbox$/ {
if (state != 5 && state != 6)
print "Line", NR, "-", "Unexpected 'setting' message";
state = 6;
print "Line", NR, "-", "Message ignored by CiviMail";
processed++;
}

/^got to the end of the mailbox$/ {
if (state != 3 && state != 6)
print "Line", NR, "-", "Unexpected 'got to end' message";
state = 7;
processed++;
}

/^$/ { # Ignore blank lines
processed++;
}

/^.+$/ { # Report unexpected non-blank lines
if (NR != processed)
{
print "Line", NR, "-", "Unexpected line:", $0
processed++;
}
}

END {
if (state != 7)
print "After line", NR, "-", "Unexpected end-of-file. Expected 'got to end of mailbox' message"
}

Ken

(If you find these scripts helpful then please applaud. If you improve or extend them, please reply with the details. One extension I think might be helpful is for a PERL programmer to rewrite them in PERL to avoid the bash and awk overheads. But as the scripts aren't running that often, the overhead is OK.)

Thank you for this, Ken, but I don't know how to implement these scripts.  Would you supply more detail for the Linux novice?

I are a engineer, so in this case, I would take the little n.  Thanks, sir.  I'll let you know how this works for me.

Aaron,

Yes, storing passwords in files is insecure. But this is no less secure than the CiviCRM and Joomla practice of putting database passwords in configuration files (I don't know whether Drupal does that).

Unless the folks at CiviCRM change their implementation and take the password out of the URL, you will need to either not use cron (do it manually) or secure your configuration.

Here are some tips ...

• Place the scripts in a directory that cannot be accessed by Apache (so the world can't see it)
• Make the file containing the password readable only by the user executing the cron script
• Have the cron job executed by a user who is unable to log into your machine (increasing the effectiveness of the previous point)
• Set your Apache access logs to not be readable except by the Apache user, as the URL is stored therein and it contains all those juicy credentials
• Have a good password policy for both your server and your CMS
• Run the scripts from the machine hosting your CiviCRM instance, so stuff is not on the net
• Execute the scripts with the https scheme and not http

Ken

What about running these scripts from a phpcli instead of wget them from a bash ?

FYI, here is a "wrapper" on bin/cli.php

Sure,

As I see it, the main benefit of using php-cli insead of wget are :

• Better control of the priorities (eg you can run the cron at a lower priority than the interactive processes)
• Get rid of the webserver overhead
• Handles better the memory limit/timout
• And let the web resources free for web visitors instead of blocking one process/thread on back office stuff

kmarkley,

The output you have is odd!

Can I suggest executing the script by hand? (That is, cut-and-paste each statement into a shell window, and test the conditions by inspection: for instance, using "ls -l" rather than "-s" to test id a file exists and has non-zero size.)

Perhaps that might show up an error. (For example, error output being written to stderr might be being ignored by the script.)

Ken

kmarkley,

Weird! It's almost like your awk script is simply "{print $0}" which sends input to output.

Is the script being specified in the "-f" parameter? (eg, replace "awk -f" with "cat" )
Try "diff" between your awk script and the one on this forum.
Try "which awk" to see it's picking up something sensible. (On Ubuntu 9.10 I get the response "/usr/bin/awk".)

Ken