Useful cron jobs: sending and receiving CiviMails

[ken]

I've updated these scripts. See http://forum.civicrm.org/index.php/topic,18816 for details. CiviMailProcessor.php will be superseded by EmailProcessor.php in 3.4, and the mail receiving script doesn't handle the case when emails are being converted into activities.

These are the cron jobs I use for sending and receiving CiviMails. The scripts work for 2.2+ as they use the CiviMailProcessor.php script. The scripts were written for Joomla! 1.5.x running in a Linux environment.

The value-add with these scripts is the error checking. They only notify the administrator if something has gone wrong (they take advantage of the fact that under Linux, cron only sends an email of it's activity if the script it runs produces a non-empty result), so one doesn't get bombarded with stuff. (I run these every 5 minutes or so.)

This is what they do ...

• Use wget to run the CiviCRM script and get the results
• Analyse the results and produce a non-empty output if there is a problem
• Clean up after themselves
• (The cron daemon will send an email if the output is non-empty. You need to have the cron job run by a user whose email you will receive.)

All the scripts are in the same directory and the configuration script is called 'configuration.sh'. If you change either of these details you will need to edit how the configuration script is sourced by the other scripts.

The first script is the configuration script. The variable assignments with the empty RHS will need to be set to suit your environment.

Code: [Select]

#!/bin/bash
#
# Configuration parameters used by the CiviCRM scripts
#

TmpDirectory=               # Directory for temporary files (need rw access)

TheHost=                    # eg, http://google.com
ThePath=/administrator/components/com_civicrm/civicrm/bin
TheUser=                    # Supply credentials for the Joomla! user
ThePass=                    # under whom the scripts are executed,
TheKey=                     # plus the CiviCRM key

Next is the script for sending CiviMails. As the wget command returns an empty file when all is well, this script is simpler than the receiving script that follows it. If wget fails, or a non-empty result is returned, this information is output to be sent to the administrator.

Code: [Select]

#!/bin/bash
#
# This script will send CiviMail emails.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviMail send wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

SendUrl="${TheHost}${ThePath}/civimail.cronjob.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log

wget "${SendUrl}" --output-document=$Document --output-file=$Log
Result=$?

if [[ $Result != 0 ]]
then
echo Failed to connect to CiviCRM
echo ' '
echo ---- Return-code returned by wget ----
echo $Result
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ -s $Document ]]
then
echo CiviCRM returned a non-empty result
echo ' '
echo ---- Response from CiviCRM ----
cat $Document
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ ! -e $Document ]]
then
echo CiviCRM did not return a result
echo ' '
echo ---- Contents of wget log ----
cat $Log
fi

rm -f $Document $Log

Next is the script for receiving CiviMails. As wget returns a non-empty result, this needs to be parsed to detect any errors. This is done with an awk script.

Code: [Select]

#!/bin/bash
#
# This script will receive CiviMail emails.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviMail receive wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

ReceiveUrl="${TheHost}${ThePath}/CiviMailProcessor.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log
StatusLog=${FilePattern}.status.log

wget "${ReceiveUrl}" --output-document=$Document --output-file=$Log
Result=$?

if [[ $Result != 0 ]]
then
echo Failed to connect to CiviCRM
echo ' '
echo ---- Return-code returned by wget ----
echo $Result
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ -s $Document ]]
then
awk -f $Directory/mail-receive-status.awk $Document > $StatusLog
if [[ -s $StatusLog ]]
then
cat $StatusLog
echo --------------------
cat $Document
fi
elif [[ -e $Document ]]
then
echo CiviCRM returned an empty result
echo ' '
echo ---- Contents of wget log ----
cat $Log
else
echo CiviCRM did not return a result
echo ' '
echo ---- Contents of wget log ----
cat $Log
fi

rm -f $Document $Log $StatusLog

Lastly is the awk script for analysing the result of the receiving script. The receiving script given above expects this to be in a file called 'mail-receive-status.awk' in the same directory as itself.

Code: [Select]

#!/usr/bin/awk
#
# Awk script to interpret the results of a CiviMail receipt
#

# Here is a sample file that should be validated by this script ...
#
# REFERENCE : CONTENTS
# ..........:............
# 1         : connecting to yyy.zzz, authenticating as xxx@yyy.zzz and selecting INBOX
# 2         : mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Processed replies, INBOX.Processed bounces, INBOX
# 3{1}      : fetching 50 messages
# 3{1}.1[1] : retrieving message 2464
# 3{1}.1[2] : retrieving message 2465
# 3{1}.2[1] : setting 2464 as seen and moving it to the processed mailbox
# 3{1}.2[2] : setting 2465 as seen and moving it to the processed mailbox
# 4         : got to the end of the mailbox
#
# To understand the reference column ...
#   * the numbers indicate sequence,
#   * '.' indicates a component of a larger structure,
#   * '[]' indicates repetition one or more times, and
#   * '{}' indicates repetition zero or more times
#

# This script produces an empty file if the input file is valid.
# Otherwise it produces a non-empty file explaining the problem
#

# Specific problems that are targeted ...
#   * Missing line-reference 1, 2, or 4
#   * Mail sent to the 'ignored' mailbox on line-reference 3[].2[]
#   * Invalid content on any line
#

# Program states, and state transitions ...
#
# STATE : AFTER SEEING ...   : VALID NEXT STATES : VALID PREVIOUS STATES
# ......:....................:...................:......................
# 1     : BEGIN              : 2                 : -
# 2     : connecting to      : 3                 : 1
# 3     : mailboxes found    : 4, 7              : 2
# 4     : fetching messages  : 5                 : 3, 6
# 5     : retrieving message : 5, 6              : 4, 5
# 6     : setting as seen    : 4, 6, 7           : 5, 6
# 7     : got to end         : 8                 : 3, 6
# 8     : END                : -                 : 7

BEGIN {
state = 1;
processed = 0;
}

/^connecting to [a-z.-]+, authenticating as [a-z.@-]+ and selecting [A-Za-z]+$/ {
if (state != 1)
print "Line", NR, "-", "Unexpected 'connecting' message";
state = 2;
processed++;
}

/^mailboxes found: .+$/ {
if (state != 2)
print "Line", NR, "-", "Unexpected 'mailboxes found' message";
state = 3;
processed++;
}

/^fetching [0-9]+ messages$/ {
if (state != 3 && state != 6)
print "Line", NR, "-", "Unexpected 'fetching' message";
state = 4;
processed++;
}

/^retrieving message [0-9]+$/ {
if (state != 4 && state != 5)
print "Line", NR, "-", "Unexpected 'retrieving' message";
state = 5;
processed++;
}

/^setting [0-9]+ as seen and moving it to the processed mailbox$/ {
if (state != 5 && state != 6)
print "Line", NR, "-", "Unexpected 'setting' message";
state = 6;
processed++;
}

/^setting [0-9]+ as seen and moving it to the ignored mailbox$/ {
if (state != 5 && state != 6)
print "Line", NR, "-", "Unexpected 'setting' message";
state = 6;
print "Line", NR, "-", "Message ignored by CiviMail";
processed++;
}

/^got to the end of the mailbox$/ {
if (state != 3 && state != 6)
print "Line", NR, "-", "Unexpected 'got to end' message";
state = 7;
processed++;
}

/^$/ { # Ignore blank lines
processed++;
}

/^.+$/ { # Report unexpected non-blank lines
if (NR != processed)
{
print "Line", NR, "-", "Unexpected line:", $0
processed++;
}
}

END {
if (state != 7)
print "After line", NR, "-", "Unexpected end-of-file. Expected 'got to end of mailbox' message"
}

Ken

(If you find these scripts helpful then please applaud. If you improve or extend them, please reply with the details. One extension I think might be helpful is for a PERL programmer to rewrite them in PERL to avoid the bash and awk overheads. But as the scripts aren't running that often, the overhead is OK.)

[petbos157]

Ken,

I will applaud. It work like a charm. Even on a drupal-systeem

Peter

[NullSmurf]

Thank you for this, Ken, but I don't know how to implement these scripts.  Would you supply more detail for the Linux novice?

[ken]

Dear Null (or is that Smurf?),

Are you a little-n novice or a big-N novice? I'm not sure if this description is too basic or too high-level, but ...

A. You should understand what it is you're automating. Read http://wiki.civicrm.org/confluence/display/CRMDOC/Command-line+Script+Configuration and get to the point that you can invoke civimail.cronjob.php and CiviMailProcessor.php from your browser. Remember the details for later (C.2).

B. From a command line, try the following commands to see if all the necessary bits are there (most Linux distro's install the manual or 'man' pages when you have the command installed) ...

man cron
man bash
man awk
man wget

I suggest you browse these 'man' pages so you know (basically) what these guys do

C. If that's OK, then ...

1. Create a directory where the scripts can live
2. Create configuration.sh with the contents indicated (you will need to supply the details after the '=' signs: this is the same details you used when running the script manually)
3. Create mail-send.sh and mail-receive.sh and mail-receive-status.awk in the same way
4. Make the *.sh scripts executable
5. Run the scripts by hand  so you know they work
6. Automate them using cron (if you need help doing that, do what I did: type "ubuntu howto cron" to find out how to use cron for my distro)

Hope that helps!

Ken

[NullSmurf]

I are a engineer, so in this case, I would take the little n.  Thanks, sir.  I'll let you know how this works for me.

[amb922]

This is probably obvious to everyone except me, but how does one put the password in the first .sh file without creating a security risk?

Thanks,

Aaron

[ken]

Aaron,

Yes, storing passwords in files is insecure. But this is no less secure than the CiviCRM and Joomla practice of putting database passwords in configuration files (I don't know whether Drupal does that).

Unless the folks at CiviCRM change their implementation and take the password out of the URL, you will need to either not use cron (do it manually) or secure your configuration.

Here are some tips ...

• Place the scripts in a directory that cannot be accessed by Apache (so the world can't see it)
• Make the file containing the password readable only by the user executing the cron script
• Have the cron job executed by a user who is unable to log into your machine (increasing the effectiveness of the previous point)
• Set your Apache access logs to not be readable except by the Apache user, as the URL is stored therein and it contains all those juicy credentials
• Have a good password policy for both your server and your CMS
• Run the scripts from the machine hosting your CiviCRM instance, so stuff is not on the net
• Execute the scripts with the https scheme and not http

Ken

[amb922]

Thanks!  I got it working in a private directory.

[xavier]

What about running these scripts from a phpcli instead of wget them from a bash ?

FYI, here is a "wrapper" on bin/cli.php

[ken]

Xavier,

My approach has been to follow the documented instructions, and add a layer to filter the output so I am notified when anything 'interesting' happens, and not notified otherwise.

I don't have enough PHP in me to desire to make those changes.

An alternate approach would be to adjust the existing PHP to have a 'silent' mode (https://...&silent=1) which forces the script to only produce output if "human attention" is needed

• An error has occurred
• CiviMail is ignoring an incoming message

Otherwise, no output is produced.

This would help whether the code was invoked from bash/wget or phpcli.

Ken

[xavier]

Sure,

As I see it, the main benefit of using php-cli insead of wget are :

• Better control of the priorities (eg you can run the cron at a lower priority than the interactive processes)
• Get rid of the webserver overhead
• Handles better the memory limit/timout
• And let the web resources free for web visitors instead of blocking one process/thread on back office stuff

[kmarkley]

Thank you for these, Ken.  Very helpful.

But I'm having a little trouble with mail-receive.sh.  Every time I run it, I get the following output:

Code: [Select]

connecting to smtp.gmail.com, authenticating as email@example.com and selecting INBOX
mailboxes found: Follow up, INBOX, INBOX/CiviMail/ignored, INBOX/CiviMail/processed, Misc, Priority, [Gmail]/All Mail, [Gmail]/Drafts, [Gmail]/Sent Mail, [Gmail]/Spam, [Gmail]/Starred, [Gmail]/Trash
got to the end of the mailbox
--------------------
connecting to smtp.gmail.com, authenticating as email@example.com and selecting INBOX
mailboxes found: Follow up, INBOX, INBOX/CiviMail/ignored, INBOX/CiviMail/processed, Misc, Priority, [Gmail]/All Mail, [Gmail]/Drafts, [Gmail]/Sent Mail, [Gmail]/Spam, [Gmail]/Starred, [Gmail]/Trash
got to the end of the mailbox
I'm not a programmer, but I can follow the logic and tell it is coming from this part of the script:

Code: [Select]

elif [[ -s $Document ]]
then
awk -f $Directory/mail-receive-status.awk $Document > $StatusLog
if [[ -s $StatusLog ]]
then
cat $StatusLog
echo --------------------
cat $Document
fi
The weird thing is I am not seeing any of the

print "Line", NR, "-", "Unexpected message";

stuff from mail-receive-status.awk.  I don't see any PRINT lines in the file that are not in this format.  (Plus earlier, I was having other problems and I did get the expected output from mail-receive-status.awk, so I know it is working.)

I tried commenting out

rm -f $Document $Log $StatusLog

and the resulting files (.status.log and .document) exactly match the output as expected.

Can anyone point me in the right direction?

[ken]

kmarkley,

The output you have is odd!

Can I suggest executing the script by hand? (That is, cut-and-paste each statement into a shell window, and test the conditions by inspection: for instance, using "ls -l" rather than "-s" to test id a file exists and has non-zero size.)

Perhaps that might show up an error. (For example, error output being written to stderr might be being ignored by the script.)

Ken

[kmarkley]

The problem is that awk is passing the input straight through to the output.  I took the example data from the top of the file and made a mail-receive-test.txt file thus:

Code: [Select]

connecting to yyy.zzz, authenticating as xxx@yyy.zzz and selecting INBOX
mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Processed replies, INBOX.Processed bounces, INBOX
fetching 50 messages
retrieving message 2464
retrieving message 2465
setting 2464 as seen and moving it to the processed mailbox
setting 2465 as seen and moving it to the processed mailbox
got to the end of the mailbox
Then from the command line, I ran:

Code: [Select]

awk -f mail-receive-status.awk mail-receive-test.txtand the output exactly matched the input file. 

Next I added a nonsense line to the test file:

Code: [Select]

connecting to yyy.zzz, authenticating as xxx@yyy.zzz and selecting INBOX
mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Processed replies, INBOX.Processed bounces, INBOX
fetching 50 messages
gobledy-goop, it makes no sense
retrieving message 2464
retrieving message 2465
setting 2464 as seen and moving it to the processed mailbox
setting 2465 as seen and moving it to the processed mailbox
got to the end of the mailbox
and ran the command line again.  This time the output was:

Code: [Select]

connecting to yyy.zzz, authenticating as xxx@yyy.zzz and selecting INBOX
mailboxes found: INBOX.CiviMail.processed, INBOX.CiviMail.ignored, INBOX.Processed subscriptions, INBOX.Trash, INBOX.Sent, INBOX.Drafts, INBOX.Junk, INBOX.Processed replies, INBOX.Processed bounces, INBOX
fetching 50 messages
gobledy-goop, it makes no sense
Line 4 - Unexpected line: gobledy-goop, it makes no sense
retrieving message 2464
retrieving message 2465
setting 2464 as seen and moving it to the processed mailbox
setting 2465 as seen and moving it to the processed mailbox
got to the end of the mailbox
I've been scouring the awk man page for some kind of setting to change this behavior, but I haven't figured it out yet.

[ken]

kmarkley,

Weird! It's almost like your awk script is simply "{print $0}" which sends input to output.

Is the script being specified in the "-f" parameter? (eg, replace "awk -f" with "cat" )
Try "diff" between your awk script and the one on this forum.
Try "which awk" to see it's picking up something sensible. (On Ubuntu 9.10 I get the response "/usr/bin/awk".)

Ken