Scheduled jobs - 'Daily' jobs will eventually skip a day

[ken]

There is a logic flaw with 'Daily' scheduled jobs, that will eventually lead to a day where the job is not run, skipping a day.

The logic in 4.5.8 is ...

Code: [Select]

      case 'Daily':
        $now     = CRM_Utils_Date::currentDBDate();
        $dayAgo  = strtotime('-1 day', strtotime($now));
        $lastRun = strtotime($this->last_run);
        if ($lastRun < $dayAgo) {
          return TRUE;
        }

To see the problem, imagine the cron job is run at times T1, T2, ..., Ti, Ti+1, ..., Tn through the day.

If it runs at time Ti on Day 1, then on Day 2 it will either run at time Ti or Ti+1 (depending on exactly when the job runs). Eventually a day will come when the job runs at time Tn, and the day after that the job will either run at Tn or not at all, as there is no Tn+1.

In that final case, the job will not run until T1 on the subsequent day. In that case a day has been skipped.

A correct solution for daily jobs

To avoid this, $now and $this->last_run should each be converted to a day number. The job should run if day_number($now) != day_number($this->last_run). (I'm suggesting the day number is in the form 20150315.)

A correct solution for hourly jobs

Since the logic for Hourly jobs follows the same pattern as Daily, it has the same problem of skipping an hour. This might not be as serious as skipping a day.

A similar solution also works for 'Hourly' jobs, by running the job if hour_number($now) != hour_number($this->last_run). (the hour number is 0, 1, ..., 23)

A better solution for daily jobs

The solution I propose above for Daily jobs will result in all Daily jobs being run at T1. This might be undesirable.

A better solution might be to define 'hours(x) = 24*day_number(x) + hour_number(x)' and then run the Daily job when ...

hours($now) - hours($this->last_run) >= 24

Then the job will run in the same hour every day. (To set it to run in a particular hour, run the job manually at that time, and then it will run in that hour from that time on.)

An even better solution for daily jobs

Use a job parameter "not_before=6" to prevent jobs running before 0600 hours.

$now and $this->last_run would each be converted to a day number. If the "not_before" parameter is present, use it, else treat it as zero. The job should run if day_number($now) != day_number($this->last_run) && hour_number($now) >= not_before.

This works as long as no job expects a not_before parameter. (I assume jobs ignore extraneous parameters.)

[JonGold]

This is a good solution - but if we're really taking a look at this code, I would argue that the entire approach could use some refactoring.  Especially with the UI work put in to specify recurring dates in 4.6 for the recurring event engine, it'd be great to see some of that applied here to get better flexibility in scheduled jobs.

E.g. Some daily jobs really make sense to run just after midnight - for instance, the membership status update job.  If it runs daily, and it initially ran at 8pm, you get the situation where you're looking up whether folks are active members, and for the day after their membership status should change, it hasn't yet.  The only way around this is to manually run the job after midnight or do SQL.

Some e-mailed reports should run weekly or monthly, but there's no way to specify this.

Of course, it's easier to come up with ideas than implement them...but as long as we're talking about preferred behavior of scheduled jobs, I figured I'd throw this out there.

[ken]

Jon,

From my observations of the recurring events and activities in 4.6, these are not ongoing happenings but are bound in number of recurrences, dates, etc. My gut feel is this a bit heavyweight.

An even better solution for daily jobs

Use a job parameter "not_before=6" to prevent jobs running before 0600 hours.

$now and $this->last_run would each be converted to a day number. If the "not_before" parameter is present, use it, else treat it as zero. The job should run if day_number($now) != day_number($this->last_run) && hour_number($now) >= not_before.

This works as long as no job expects a not_before parameter. (I assume jobs ignore extraneous parameters.)

[joanne]

As a user I would like to strongly support Jon's  suggestion of increased control/flexibility.

We want some membership reports run ASAP after midnight on the first day of each month so that we have a snap shot of our member numbers at the end of the previous month.  The reports need to be emailed to the relevant people.

We had to get our delevoper to schedule those jobs as I don't have that sort of control through the CiviCRM GUI.  It would be great if I did have though.

[ken]

Joanne,

Your issue is different to the one I'm discussing.

You can do what you need via the command-line features of CiviCRM. Open http://wiki.civicrm.org/confluence/display/CRMDOC/Managing+Scheduled+Jobs and look for "mail_report" on that page.

The issue I have is that "Daily" is not daily.

Ken

[joanne]

Ken, I realise you started this topic talking about daily jobs, but Jon had already broadened the topic by discussing more than that and I was responding to his post.  I probably should have started with @Jon to make that perfectly clear.

This is a good solution - but if we're really taking a look at this code, I would argue that the entire approach could use some refactoring.  Especially with the UI work put in to specify recurring dates in 4.6 for the recurring event engine, it'd be great to see some of that applied here to get better flexibility in scheduled jobs.

E.g. Some daily jobs really make sense to run just after midnight - for instance, the membership status update job...

Some e-mailed reports should run weekly or monthly, but there's no way to specify this...

but as long as we're talking about preferred behavior of scheduled jobs, I figured I'd throw this out there.

[clarkac]

See https://issues.civicrm.org/jira/browse/CRM-16276