.log files...how often do they change out, how big can they get?

[CiviTeacher.com]

Code: [Select]

files/civicrm/templates_c/en_US/ConfigAndLog/CiviCRM*.log
What is the deal with these files, how often do they rotate and/or get deleted?  In a couple of my installations the log file is approaching 100MB.  It's becoming a disk-space-usage issue.

I looked for documentation on it, couldn't find any.  If docs exist, please point me in that direction.

[Donald Lobo]

the more recent version of CiviCRM have been creating a new file when monthly or if greater than 256M. We do not delete log files right now. There are pros/cons associated with this, but definitely should be documented

Also we should document the CiviMail archival process so admins can check them on a regular basis and delete them if needed. In general the time does seem right to build a CiviCRM Maintainance component

lobo

[CiviTeacher.com]

Thanks.  Is there a way to turn OFF the generation of these .log files entirely?

[Donald Lobo]

Currently no. we do think logging is quite useful to figure out unexpected errors, but since we can also log to drupal watchdog, maybe making an option to turn off writing to file when watchdog logging is enabled is a step in the right direction

wanna investigate and submit a patch?

lboo

[mdlueck]

but since we can also log to drupal watchdog

In theory, or is this something we could configure right now in 3.3.5?

[Donald Lobo]

logging to drupal watchdog is a feature in 3.3

check

http://drupal.demo.civicrm.org/civicrm/admin/setting/debug?reset=1

lobo

[mdlueck]

Thank you so much, lobo! 'tis enabled now! 

[CiviTeacher.com]

My primary concern is that the CiviCRM.log files can get very large, several GB.  I have run into disk-space errors twice this month.

It says this in CiviCRM about Watchdog

Set this value to Yes if you want CiviCRM error/debugging messages to also appear in Drupal error logs

Watchdog tables in Drupal can be automatically cleared by Drupal with cron, but this doesn't mean the CiviCRM.log files aren't still being written to the file structure. 

I am going to post a little script I installed on a client's site recently to purge the CiviCRM.log weekly.

Eventually we could build a PHP file in <civicrm>/bin that would perform the same task (and be run by cron like UpdateMembershipStatus.php and civimail.cronjob.php) but I'm not entirely sure how to do that.  I would like to work with one of the more experienced developers to accomplish this.   

CRONTAB entry (once a week)  Note the the user this cron is run under must have permissions to delete files- in my case this is 'root'

Code: [Select]

30 1 * * 1 /bin/sh /home/<yoursite>/logs/clear_civi_log.sh
CONTENTS of .sh file

Code: [Select]

#!/usr/sh
# this script clears the civicrm log files at
rm -f /home/<yoursite>/<path_to_civiCRM_files>/civicrm/templates_c/en_US/ConfigAndLog/CiviCRM*log*


[mdlueck]

While I like the idea of Civi cleaning up after itself, I also do not want to get burned by finding something I want to check into and "Oh no the purge ran and destroyed yesterday's logs".

Is there a single place in Civi that the .log filename is defined? I would think to modify that spot to add a .YYYYMMDD. to the filename, and I can set up a daily purge of that filespec of all files older than N (30 perhaps) days.

[CiviTeacher.com]

It sort of does that already.  check your log directory and see how it looks now

[mdlueck]

Oh that is right...

Code: [Select]

CiviCRM.f266acfe3eee7422d4ccdd0394b10fee.log
So how often does Civi change that random part? I only see that one filename currently. Certainly not daily in that case.

[Donald Lobo]

from the code:

CRM/Core/Error.php, function debug_log_message

Code: [Select]

        // Roll log file monthly or if greater than 256M
                                                                                                                   
        // note that PHP file functions have a limit of 2G and hence
                                                                                                     
        // the alternative was introduce
                                                                                                                             

would be great if we can change the code and roll on a daily basis like most other open source packages. This will allow people to use utilities like rotatelogs

maybe we should just switch and store it in the DB like drupal? and then trim the table on a regular basis (user defined)

would be great if someone can take the lead on this. we can help out on IRC

lboo

[CiviTeacher.com]

I will look at the code to see what i can do.  It seems the existing script could delete .log files > 31 days old and that would solve most of the problems, without purging relevant files.   The troubles I had were from months of 256mb files building up.

[CiviTeacher.com]

@mdlueck.  That is the current log file u see.  After months there will be mire files in there with the date appended after .log

[mdlueck]

would be great if we can change the code and roll on a daily basis like most other open source packages. This will allow people to use utilities like rotatelogs

Is not that simply not inserting into the filename anything random and letting rotatelogs handle shift renaming the files? If so, removing the randomization of the filename should be simple enough that I would take on the task.

Also, perhaps like with Drupal Watchdog logging, have an on/off radio button in the UI, same URL as where Drupal logging gets enabled. If that is set to off, then skip over the Civi file logging.

The current directory the log file is stored at (/files/civicrm/templates_c/en_US/ConfigAndLog) was getting nuked by me as per the upgrade docs, I was purging /files/civicrm/templates_c/ so now I

Code: [Select]

rm -rf /files/civicrm/templates_c/en_US/%%* which leaves only the ConfigAndLog directory. Thus why I did not have much of a log file buildup before I discovered this.