CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Post-installation Setup and Configuration (Moderator: Dave Greenberg) »
  • Useful cron jobs: sending and receiving CiviMails
Pages: 1 [2] 3 4

Author Topic: Useful cron jobs: sending and receiving CiviMails  (Read 10683 times)

kmarkley

  • I post frequently
  • ***
  • Posts: 178
  • Karma: 14
  • CiviCRM version: 4.4.3
  • CMS version: Drupal 7.24
  • MySQL version: 5.1.56
  • PHP version: 5.3.27
Re: Useful cron jobs: sending and receiving CiviMails
December 18, 2009, 08:35:25 am
I found the problem, and I'm a little embarrassed.  When I first copy-pasted the awk script, I missed a couple characters at the very beginning.  So instead of

#!/usr/bin/awk

I had

/bin/awk

Despite all my subsequent staring at the file while trying to figure out what was going on, I did not notice the change until I ran diff as you suggested.  Thanks

kmarkley

  • I post frequently
  • ***
  • Posts: 178
  • Karma: 14
  • CiviCRM version: 4.4.3
  • CMS version: Drupal 7.24
  • MySQL version: 5.1.56
  • PHP version: 5.3.27
Re: Useful cron jobs: sending and receiving CiviMails
January 31, 2010, 10:36:26 am
Extending this method to ParticipantProcessor.php

This has a pretty low threshold for what is 'interesting' enough to report on.  Basically if any participant statuses are changed the script will generate output.  Also uses the very low-tech method of just comparing the output from ParticipantProcessor.php to a reference file.  But it works for me.

participant-update.sh
Code: [Select]
#!/bin/bash
#
# This script will update participant statuses.
# It does this by invoking the appropriate CiviCRM PHP script.
# If anything unusual happens, a mail is sent to the system administrator
#

Program="CiviEvent update wrapper"

Directory=`dirname $0`
. $Directory/configuration.sh

SendUrl="${TheHost}${ThePath}/ParticipantProcessor.php?name=${TheUser}&pass=${ThePass}&key=${TheKey}"
FilePattern=$TmpDirectory/`basename $0`.$$
Document=${FilePattern}.document
Log=${FilePattern}.log
StatusLog=${FilePattern}.status.log

wget "${SendUrl}" --output-document=$Document --output-file=$Log
Result=$?

sed -i s/"<br \/>"/\\n/g $Document

if [[ $Result != 0 ]]
then
echo Failed to connect to CiviCRM
echo ' '
echo ---- Return-code returned by wget ----
echo $Result
echo ' '
echo ---- Contents of wget log ----
cat $Log
elif [[ -s $Document ]]
then
diff -w $Directory/participant-update-null-result.txt $Document > $StatusLog
if [[ -s $StatusLog ]]
then
echo CiviCRM updated participants
echo ' '
echo -----------------------------
echo ' '
cat $Document
fi
elif [[ -e $Document ]]
then
echo CiviCRM returned an empty result
echo ' '
echo ---- Contents of wget log ----
cat $Log
else
echo CiviCRM did not return a result
echo ' '
echo ---- Contents of wget log ----
cat $Log
fi

rm -f $Document $Log $StatusLog

participant-update-null-result.txt
Code: [Select]
Updating..

Number of Expired registration(s) = 0
Number of registration(s) require approval =  0
Number of registration changed to Pending from waitlist = 0


Participant records updated. (Done)

ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
February 01, 2010, 12:34:37 am
Thanks, kmarkley!

(BTW, low-tech is often good-tech!)

Alan.Guggenheim

  • Guest
Re: Useful cron jobs: sending and receiving CiviMails
February 14, 2010, 09:35:59 pm
Ken,

I would love to use your scripts. They seem very interesting.
I am running into sevral issues, though (Drupal 6.15, PHP 5.2.11 on Centos)
When I run the script manually, it gives me several error messages:
it looks like it errors on evry blank line, and it does not resolve properly the SendURL variable: It gets only the key
the configuration.sh file is OK, but mail-send.sh does not connect to CiviCRM.

ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
February 16, 2010, 02:38:22 am
Alan,

I'd like to help, but I'm going to need to see the exact error messages (sanitised of course so you don't reveal your passwords to the whole web)

  • I don't know much about Drupal, but perhaps the URL is different (I wrote the scripts for the Joomla environment)
  • Run the scripts with the verbose flag on (set the first line to #!/bin/bash -v)
  • Try executing the script by hand, one command at a time. That might reveal what the error is.

Ken
« Last Edit: February 16, 2010, 02:44:33 am by ken »

Alan.Guggenheim

  • Guest
Re: Useful cron jobs: sending and receiving CiviMails
March 02, 2010, 12:09:54 pm
Ken,
I would like to use your scripts, but they do not seem to work on my settings: Drupal 6.15 on CentOS. Any idea why? I am not familiar with bash, but it looks like variable processing is all mixed-up:
If I echo the $SendUrl variable from the command line, I get
&key=zyOjEhY5.......c1OCKrty.com

ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
March 02, 2010, 12:21:06 pm
I'm assuming that zyOjEhY5.......c1OCKrty.com is the value that you set for TheKey?

If so, I can't see how SendUrl could take on that value.

Are you using bash as your shell? Is bash on your system (ls /bin/bash should not give you "file not found").

Alan.Guggenheim

  • Guest
Re: Useful cron jobs: sending and receiving CiviMails
March 02, 2010, 12:26:04 pm
Ken,
yes, bash is on this system, and is my shell. There must be something odd, though, as it looks like the variables are written over each other instead of being concatenated! (apparently it looks like it is a known behaviour of bash with /r). I just wanted to veryfy my string build for the wget, but the real issue is that I get the following error:
/mail-send.sh.23522.log: No such file or directory
Failed to connect to CiviCRM

---- Return-code returned by wget ----
1

---- Contents of wget log ----
/mail-send.sh.23522.log: No such file or directory

Thanks
« Last Edit: March 02, 2010, 01:01:51 pm by Alan.Guggenheim »

ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
March 02, 2010, 04:58:02 pm
The path for the .log file puts it in the root directory (/) which you are probably prohibited from writing in. Set the TmpDirectory argument.

ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
March 02, 2010, 05:03:41 pm
...known problem with /r ...

I don't know the problem you're referring to, so I suggest googling for that problem on a Linux forum and following the suggested remedial action.

/r ?

Was a windows machine involved here? (Linux uses /n to terminate lines). Either create the scripts from scratch using a Linux editor (eg wget the original forum post, and use 'nano' or another Linux editor to grab the script text) or use a Linux utility such as 'tr' to get rid of the /r characters.

Ken

Alan.Guggenheim

  • Guest
Re: Useful cron jobs: sending and receiving CiviMails
March 02, 2010, 05:35:47 pm
I tried the following path for TmpDirectory:
tmp
/tmp
/home/account/tmp
/home/account/public_html/tmp

all of them exist and I tried to give them all 777 rights, and I always get the same error:
root@server [/home/account/scripts-civimail]# bash mail-send.sh
/mail-send.sh.4702.log: No such file or directory
Failed to connect to CiviCRM

---- Return-code returned by wget ----
1

---- Contents of wget log ----
/mail-send.sh.4702.log: No such file or directory
root@server [/home/account/scripts-civimail]#

To debug, i was trying to echo the variables to check that they are all correct, and i did run into a problem: each of them display properly, but the combined one $SendUrl for example is wrong. It looks like each of the component is writing over instead of after the previous one. Like:
echo ${TheHost} gives:
http://mywebsitenamethat_is_very_long.com
but echo ${TheHost}${ThePath} gives:

/sites/all/modules/civicrm/bins_very_long.com


Doing some research on bash and this strange behavior sent me to this /r issue which might not be related but it is still a strange issue, and does not help debug.


ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
March 03, 2010, 12:39:06 am
Here's a guess ...

\r is a carriage return, meaning (possibly) "on a tty move the cursor back to the first column".

If so, and your variables have a \r at the end, then ...

http://my__web__site__name_that_is_very_long.com\r/sites/all/modules/civicrm/bin\r

... appears as ...

/sites/all/modules/civicrm/bins_very_long.com

... where the underlined text is what overwrites part of your URL.

Use 'od' to check the scripts for \r characters, and use 'tr' to remove them. These commands are documented in 'man'.

Alan.Guggenheim

  • Guest
Re: Useful cron jobs: sending and receiving CiviMails
March 03, 2010, 07:36:52 am
Ken,
Thanks for your help. I am using UltraEdit, so it was very easy to search and remove ^r. Indeed, by copying and pasting from the website, each variable ended up being terminated by a ODOA (hex) or /r/n and that was screwing up the string manipulation in bash!

As soon as I removed the /r the script works!

Thanks for your help, and sorry it took me so long to realize what was hapening.

Alan.Guggenheim

  • Guest
Re: Useful cron jobs: sending and receiving CiviMails
March 03, 2010, 03:27:16 pm
Ken,
Now that the scripts work from the command line, a few questions:
1- Which one should be executed from cron and at what rate do you recommand? Is it correct that only 2 scripts need to be in cron: mail-send.sh and mail-receive.sh?
2- do we still need to call them from cron with wget or curl and the whole name, password and key? if not, what syntax?
Thanks

ken

  • I live on this forum
  • *****
  • Posts: 916
  • Karma: 53
    • City Bible Forum
  • CiviCRM version: 4.6.3
  • CMS version: Drupal 7.36
  • MySQL version: 5.5.41
  • PHP version: 5.3.10
Re: Useful cron jobs: sending and receiving CiviMails
March 03, 2010, 06:25:35 pm
Alan,

1. Yes
2. The command that you ask cron to run is the same one you call from the command line
    <path>/mail-send.sh
    <path>/mail-receive.sh

Regarding rates, the question is threefold ...
1. How long do you want it to take for your largest mailing to be sent?
2. What is the maximum number of emails you feel you can safely send in one hit, before some email facility out there classifies you as a spammer? (I have no opinion on what this might be)
3. What fraction of emails sent will generate some form of response?

[4] = [1] divided by [2] = the time between subsequent calls to mail-send.sh
[5] = [4] divided by [3] = the time between subsequent calls to mail-receive.sh

Don't forget to visit the CiviMail Configuration and set Mailer Spool Period to "[4]" (expressed in seconds) and Mailer Batch Limit to "[2]".

My settings are 5 minutes (300 seconds) between calls to mail-send.sh; 30 emails sent in a batch (ie, very conservative); and 12 minutes between calls to mail-receive.sh.

Ken
« Last Edit: March 03, 2010, 06:28:04 pm by ken »

Pages: 1 [2] 3 4
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Post-installation Setup and Configuration (Moderator: Dave Greenberg) »
  • Useful cron jobs: sending and receiving CiviMails

This forum was archived on 2017-11-26.