Author Topic: civicrm cron job query (Read 1421 times)

eteflamingo · January 23, 2012, 07:41:04 am

Hi everyone,
We have faced last week database overload during sending emails, when we contact our hosting support center they checked the following query :

SELECT DISTINCT civicrm_email.id AS email_id, civicrm_contact.id AS contact_id FROM civicrm_email INNER JOIN civicrm_contact ON civicrm_email.contact_id = civicrm_contact.id INNER JOIN civicrm_group_contact ON civicrm_contact.id = civicrm_group_contact.contact_id INNER JOIN civicrm_mailing_group ON civicrm_group_contact.group_id = civicrm_mailing_group.entity_id AND civicrm_mailing_group.entity_table = 'civicrm_group' WHERE (civicrm_mailing_group.group_type = 'Include' OR civicrm_mailing_group.group_type = 'Base') AND civicrm_mailing_group.search_id IS NULL AND civicrm_group_contact.status = 'Added' AND civicrm_group_contact.email_id IS NULL AND civicrm_contact.do_not_email = 0 AND civicrm_contact.is_opt_out = 0 AND civicrm_contact.is_deceased = 0 AND (civicrm_email.is_bulkmail = 1 OR civicrm_email.is_primary = 1) AND civicrm_email.on_hold = 0 AND civicrm_mailing_group.mailing_id = 191 GROUP BY civicrm_email.email ORDER BY civicrm_email.is_bulkmail

and they said The distinct() function on the civicrm_email.id is not required and should be removed from the source code, since this field is defined as a primary key. A primary key value by definition will not have duplicate entries, and the distinct() function does not allow indexes to be properly utilized.

I have removed distinct(), the query completes in (6.38 sec) otherwise won't never complete. Please advise.

Thanks

Donald Lobo · January 23, 2012, 08:58:16 am

a few questions

1. how big is your DB: emails, contacts. How many groups are u sending the mailing to

2. who is the hosting provider

lobo

eteflamingo · January 23, 2012, 09:16:04 am

Well the database is large is about 3.5g.
civicrm_contacts number of rows:2,158,889
civicrm_email number of rows:1,798,680
We were trying to send about 10 to 20 groups
our hosting provider rackspace

Donald Lobo · January 23, 2012, 09:52:40 am

Can you take a look at the code in: CRM/Mailing/BAO/Mailing.php, function getRecipients

there are quite a few places where we use DISTINCT and i suspect we dont need it for many of them. If you can take a look and check various queries and rewrite them to be more optimal that would be great

lobo

eteflamingo · January 24, 2012, 11:58:41 am

Hi Lobo,

I have removed the DISTINCT function form a few queries whom they have GROUP BY civicrm_email.email, the thing is I have duplicate emails with different id and I received email twice, since I removed DISTINCT I start having this issue. Is there anywhere else i should make any changes?

Thanks

Donald Lobo · January 24, 2012, 12:15:57 pm

I dont think u can just go in and remove the distinct stuff without trying to understand what the query is doing and how to get a similar effect. If you are not familiar with sql might help to seek help from someone who is familiar with sql

lobo

eteflamingo · January 24, 2012, 12:21:03 pm

Here's what've done:

$query = "REPLACE INTO I_$job_id (email_id, contact_id)
SELECT $email.id as email_id,
$contact.id as contact_id
FROM $email
INNER JOIN $contact
ON $email.contact_id = $contact.id
INNER JOIN $g2contact
ON $contact.id = $g2contact.contact_id
INNER JOIN $mg
ON $g2contact.group_id = $mg.entity_id
AND $mg.entity_table = '$group'
LEFT JOIN X_$job_id
ON $contact.id = X_$job_id.contact_id
WHERE
($mg.group_type = 'Include' OR $mg.group_type = 'Base')
AND $mg.search_id IS NULL
AND $g2contact.status = 'Added'
AND $g2contact.email_id IS null
AND $contact.do_not_email = 0
AND $contact.is_opt_out = 0
AND $contact.is_deceased = 0
AND ($email.is_bulkmail = 1 OR $email.is_primary = 1)
AND $email.on_hold = 0
AND $mg.mailing_id = {$mailing_id}
AND X_$job_id.contact_id IS null
GROUP BY $email.email
ORDER BY $email.is_bulkmail";
$mailingGroup->query($query);

/* Query prior mailings */
$mailingGroup->query(
"REPLACE INTO I_$job_id (email_id, contact_id)
SELECT $email.id as email_id,
$contact.id as contact_id
FROM $email
INNER JOIN $contact
ON $email.contact_id = $contact.id
INNER JOIN $eq
ON $eq.contact_id = $contact.id
INNER JOIN $job
ON $eq.job_id = $job.id
INNER JOIN $mg
ON $job.mailing_id = $mg.entity_id AND $mg.entity_table = '$mailing'
LEFT JOIN X_$job_id
ON $contact.id = X_$job_id.contact_id
WHERE
($mg.group_type = 'Include' OR $mg.group_type = 'Base')
AND $contact.do_not_email = 0
AND $contact.is_opt_out = 0
AND $contact.is_deceased = 0
AND ($email.is_bulkmail = 1 OR $email.is_primary = 1)
AND $email.on_hold = 0
AND $mg.mailing_id = {$mailing_id}
AND X_$job_id.contact_id IS null
GROUP BY $email.email
ORDER BY $email.is_bulkmail");

/* Construct the saved-search queries */
$ss->query("SELECT $group.saved_search_id as saved_search_id,
$group.id as id
FROM $group
INNER JOIN $mg
ON $mg.entity_id = $group.id
AND $mg.entity_table = '$group'
WHERE
$mg.group_type = 'Include'
AND $mg.search_id IS NULL
AND $mg.mailing_id = {$mailing_id}
AND $group.saved_search_id IS NOT null");

$whereTables = array( );
while ($ss->fetch()) {
$tables = array($contact => 1, $location => 1, $email => 1);
list( $from, $where ) = CRM_Contact_BAO_SavedSearch::fromWhereEmail( $ss->saved_search_id );
$where = trim( $where );
if ( $where ) {
$where = " AND $where ";
}
$ssq = "INSERT IGNORE INTO I_$job_id (email_id, contact_id)
SELECT $email.id as email_id,
contact_a.id as contact_id
$from
LEFT JOIN X_$job_id
ON contact_a.id = X_$job_id.contact_id
WHERE
contact_a.do_not_email = 0
AND contact_a.is_opt_out = 0
AND contact_a.is_deceased = 0
AND ($email.is_bulkmail = 1 OR $email.is_primary = 1)
AND $email.on_hold = 0
$where
AND contact_a.id NOT IN (
SELECT contact_id FROM $g2contact
WHERE $g2contact.group_id = {$ss->id} AND $g2contact.status = 'Removed')
AND X_$job_id.contact_id IS null
GROUP BY $email.email
ORDER BY $email.is_bulkmail";
$mailingGroup->query($ssq);
}

/**
* Construct the filtered search queries
*/
$query = "
SELECT search_id, search_args, entity_id
FROM $mg
WHERE $mg.search_id IS NOT NULL
AND $mg.mailing_id = {$mailing_id}
";
$dao = CRM_Core_DAO::executeQuery( $query );
require_once 'CRM/Contact/BAO/SearchCustom.php';
while ( $dao->fetch( ) ) {
$customSQL = CRM_Contact_BAO_SearchCustom::civiMailSQL( $dao->search_id,
$dao->search_args,
$dao->entity_id );
$query = "REPLACE INTO I_$job_id (email_id, contact_id)
$customSQL";
$mailingGroup->query($query);
}

/* Get the emails with only location override */
$query = "REPLACE INTO I_$job_id (email_id, contact_id)
SELECT $email.id as local_email_id,
$contact.id as contact_id
FROM $email
INNER JOIN $contact
ON $email.contact_id = $contact.id
INNER JOIN $g2contact
ON $contact.id = $g2contact.contact_id
INNER JOIN $mg
ON $g2contact.group_id = $mg.entity_id
LEFT JOIN X_$job_id
ON $contact.id = X_$job_id.contact_id
WHERE
$mg.entity_table = '$group'
AND $mg.group_type = 'Include'
AND $g2contact.status = 'Added'
AND $g2contact.email_id is null
AND $contact.do_not_email = 0
AND $contact.is_opt_out = 0
AND $contact.is_deceased = 0
AND ($email.is_bulkmail = 1 OR $email.is_primary = 1)
AND $email.on_hold = 0
AND $mg.mailing_id = {$mailing_id}
AND X_$job_id.contact_id IS null
GROUP BY $email.email
ORDER BY $email.is_bulkmail";
$mailingGroup->query($query);

/* Get the emails with full override */
$mailingGroup->query(
"REPLACE INTO I_$job_id (email_id, contact_id)
SELECT $email.id as email_id,
$contact.id as contact_id
FROM $email
INNER JOIN $g2contact
ON $email.id = $g2contact.email_id
INNER JOIN $contact
ON $contact.id = $g2contact.contact_id
INNER JOIN $mg
ON $g2contact.group_id = $mg.entity_id
LEFT JOIN X_$job_id
ON $contact.id = X_$job_id.contact_id
WHERE
$mg.entity_table = '$group'
AND $mg.group_type = 'Include'
AND $g2contact.status = 'Added'
AND $g2contact.email_id IS NOT null
AND $contact.do_not_email = 0
AND $contact.is_opt_out = 0
AND $contact.is_deceased = 0
AND ($email.is_bulkmail = 1 OR $email.is_primary = 1)
AND $email.on_hold = 0
AND $mg.mailing_id = {$mailing_id}
AND X_$job_id.contact_id IS null
GROUP BY $email.email
ORDER BY $email.is_bulkmail");

eteflamingo · January 24, 2012, 02:28:18 pm

Is it right or wrong?

Donald Lobo · January 24, 2012, 02:52:06 pm

Those are a fairly complex set of queries. So i dont think anyone can tell you an answer without a fair bit of analysis and SQL knowledge. However I do think it would be good to optimize that set of queries even more.

Posting a patch rather than the entire set makes it a lot easier to know what your changes were. Also helps if you can post your reasoning behind the changes

lobo

eteflamingo · January 24, 2012, 03:11:24 pm

sorry I'm going to post a large text, this may help

It appears that the dbname_12202011.X_1463 still does not exist, which is preventing me from running an explain on the large REPLACE statements dependent SELECT statement. To get around this I removed the references to this table in the query, though the size and structure of these temporary tables can still impact the efficiency of the query.

mysql> explain SELECT DISTINCT civicrm_email.id AS email_id, civicrm_contact.id AS contact_id FROM civicrm_email INNER JOIN civicrm_contact ON civicrm_email.contact_id = civicrm_contact.id INNER JOIN civicrm_group_contact ON civicrm_contact.id = civicrm_group_contact.contact_id INNER JOIN civicrm_mailing_group ON civicrm_group_contact.group_id = civicrm_mailing_group.entity_id AND civicrm_mailing_group.entity_table = 'civicrm_group' WHERE (civicrm_mailing_group.group_type = 'Include' OR civicrm_mailing_group.group_type = 'Base') AND civicrm_mailing_group.search_id IS NULL AND civicrm_group_contact.status = 'Added' AND civicrm_group_contact.email_id IS NULL AND civicrm_contact.do_not_email = 0 AND civicrm_contact.is_opt_out = 0 AND civicrm_contact.is_deceased = 0 AND (civicrm_email.is_bulkmail = 1 OR civicrm_email.is_primary = 1) AND civicrm_email.on_hold = 0 AND civicrm_mailing_group.mailing_id = 191 GROUP BY civicrm_email.email ORDER BY civicrm_email.is_bulkmail \G*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_mailing_group
type: ref
possible_keys: FK_civicrm_mailing_group_mailing_id
key: FK_civicrm_mailing_group_mailing_id
key_len: 4
ref: const
rows: 38
Extra: Using where; Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_group_contact
type: ref
possible_keys: UI_contact_group,FK_civicrm_group_contact_group_id,FK_civicrm_group_contact_email_id,idx_contactid_status_email
key: FK_civicrm_group_contact_group_id
key_len: 4
ref: dbname_12202011.civicrm_mailing_group.entity_id
rows: 124183
Extra: Using where
*************************** 3. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_email
type: ref
possible_keys: index_is_primary,FK_civicrm_email_contact_id
key: FK_civicrm_email_contact_id
key_len: 5
ref: dbname_12202011.civicrm_group_contact.contact_id
rows: 1
Extra: Using where
*************************** 4. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_contact
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: dbname_12202011.civicrm_email.contact_id
rows: 1
Extra: Using where
4 rows in set (0.00 sec)

The distinct() function on the civicrm_email.id is not required and should be removed from the source code, since this field is defined as a primary key. A primary key value by definition will not have duplicate entries, and the distinct() function does not allow indexes to be properly utilized.

mysql> show create table civicrm_email \G

*************************** 1. row ***************************
Table: civicrm_email
Create Table: CREATE TABLE `civicrm_email` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Unique Email ID',
`contact_id` int(10) unsigned DEFAULT NULL COMMENT 'FK to Contact ID',
`location_type_id` int(10) unsigned DEFAULT NULL COMMENT 'Which Location does this email belong to.',
`email` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL COMMENT 'Email address',
`is_primary` tinyint(4) DEFAULT '0' COMMENT 'Is this the primary?',
`is_billing` tinyint(4) DEFAULT '0' COMMENT 'Is this the billing?',
`on_hold` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'Is this address on bounce hold?',
`is_bulkmail` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'Is this address for bulk mail ?',
`hold_date` datetime DEFAULT NULL COMMENT 'When the address went on bounce hold',
`reset_date` datetime DEFAULT NULL COMMENT 'When the address bounce status was last reset',
`signature_text` text COLLATE utf8_unicode_ci COMMENT 'Text formatted signature for the email.',
`signature_html` text COLLATE utf8_unicode_ci COMMENT 'HTML formatted signature for the email.',
PRIMARY KEY (`id`),
KEY `index_location_type` (`location_type_id`),
KEY `UI_email` (`email`),
KEY `index_is_primary` (`is_primary`),
KEY `index_is_billing` (`is_billing`),
KEY `FK_civicrm_email_contact_id` (`contact_id`),
CONSTRAINT `FK_civicrm_email_contact_id` FOREIGN KEY (`contact_id`) REFERENCES `civicrm_contact` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1821592 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
1 row in set (0.00 sec)

The optimizer anticipates that joining the civicrm_group_contact table will result in 124,183 rows using the foreign key constraint on the group_ip field, which is indexed by default.

I was able to reduce this to little over 11,000 by creating the following compound index.

mysql> create index idx_groupid_contactid_emailid on civicrm_group_contact(group_id,contact_id,email_id);Query OK, 0 rows affected (8.15 sec)Records: 0 Duplicates: 0 Warnings: 0

mysql> explain SELECT civicrm_email.id AS email_id, civicrm_contact.id AS contact_id FROM civicrm_email INNER JOIN civicrm_contact ON civicrm_email.contact_id = civicrm_contact.id INNER JOIN civicrm_group_contact ON civicrm_contact.id = civicrm_group_contact.contact_id INNER JOIN civicrm_mailing_group ON civicrm_group_contact.group_id = civicrm_mailing_group.entity_id AND civicrm_mailing_group.entity_table = 'civicrm_group' WHERE (civicrm_mailing_group.group_type = 'Include' OR civicrm_mailing_group.group_type = 'Base') AND civicrm_mailing_group.search_id IS NULL AND civicrm_group_contact.status = 'Added' AND civicrm_group_contact.email_id IS NULL AND civicrm_contact.do_not_email = 0 AND civicrm_contact.is_opt_out = 0 AND civicrm_contact.is_deceased = 0 AND (civicrm_email.is_bulkmail = 1 OR civicrm_email.is_primary = 1) AND civicrm_email.on_hold = 0 AND civicrm_mailing_group.mailing_id = 191 GROUP BY civicrm_email.email ORDER BY civicrm_email.is_bulkmail \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_mailing_group
type: ref
possible_keys: FK_civicrm_mailing_group_mailing_id
key: FK_civicrm_mailing_group_mailing_id
key_len: 4
ref: const
rows: 38
Extra: Using where; Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_group_contact
type: ref
possible_keys: UI_contact_group,FK_civicrm_group_contact_group_id,FK_civicrm_group_contact_email_id,idx_groupid_contactid_emailid
key: idx_groupid_contactid_emailid
key_len: 4
ref: dbname_12202011.civicrm_mailing_group.entity_id
rows: 11176
Extra: Using where
*************************** 3. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_email
type: ref
possible_keys: index_is_primary,FK_civicrm_email_contact_id
key: FK_civicrm_email_contact_id
key_len: 5
ref: dbname_12202011.civicrm_group_contact.contact_id
rows: 1
Extra: Using where
*************************** 4. row ***************************
id: 1
select_type: SIMPLE
table: civicrm_contact
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: dbname_12202011.civicrm_email.contact_id
rows: 1
Extra: Using where
4 rows in set (0.00 sec)

When I remove the disinct() function, there are over 319,944 results and the query completes in (6.38 sec) Though this particular query is still very CPU intensive, and can take up one core during its execution, so I would not execute this concurrently often. This query was also using a 70 MB temporary table for the GROUP and ORDER by sorting, so I increased the tmp_table_size and max_heap_table_size to 100 MB.

A summary of the slow query log can be found in the /root/qd_results.txt file. Looking over the slow query logs it appears the following queries were being executed rather frequently prior to the MySQL daemon restart. Both of these queries are running a LEFT JOIN causing the query optimize sort by the civicrm_contact first, which will prevent any additional compound indexes from potentially changing the table join order.

The entries of this table that also meet the criteria for is_deleted are also close to a full table scan.

The civicrm_contact.is_deleted is not very selective, and the distinct() function is being used on the civicrm_contact.id field, both of these will greatly limit the efficiency of the index on this field. Since the civicrm_contact.id field is a primary key, the disinct() function is not necessary.

There is little that can be done from a MySQL perspective to make these more efficient.

# Time: 120121 9:48:22
# User@Host: root[root] @ serverWEB2 [10.178.230.22]
# Query_time: 20.983836 Lock_time: 0.000077 Rows_sent: 1 Rows_examined: 4724700
mysql> explain SELECT COUNT(DISTINCT contact_a.id) FROM civicrm_contact contact_a LEFT JOIN civicrm_group_contact `civicrm_group_contact-16,49,44,96` ON contact_a.id = `civicrm_group_contact-16,49,44,96`.contact_id LEFT JOIN civicrm_group_contact_cache `civicrm_group_contact_cache_44` ON contact_a.id = `civicrm_group_contact_cache_44`.contact_id WHERE (((`civicrm_group_contact-16,49,44,96`.group_id IN (16, 49, 44, 96) AND `civicrm_group_contact-16,49,44,96`.status IN ("Added")) OR (`civicrm_group_contact_cache_44`.group_id = 44))) AND (contact_a.is_deleted = 0);
+----+-------------+-----------------------------------+------+------------------+------------------+---------+------------------------------------+---------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------------------------+------+------------------+------------------+---------+------------------------------------+---------+--------------------------+
| 1 | SIMPLE | contact_a | ref | index_is_deleted | index_is_deleted | 1 | const | 1141357 | Using index |
| 1 | SIMPLE | civicrm_group_contact-16,49,44,96 | ref | UI_contact_group | UI_contact_group | 4 | dbname_12202011.contact_a.id | 1 | |
| 1 | SIMPLE | civicrm_group_contact_cache_44 | ref | UI_contact_group | UI_contact_group | 4 | dbname_12202011.contact_a.id | 1 | Using where; Using index |
+----+-------------+-----------------------------------+------+------------------+------------------+---------+------------------------------------+---------+--------------------------+
3 rows in set (0.00 sec)

# Time: 120121 9:54:36
# User@Host: root[root] @ serverWEB2 [10.178.230.22]
# Query_time: 24.831919 Lock_time: 0.000131 Rows_sent: 36519 Rows_examined: 14760882

mysql> explain SELECT DISTINCT(contact_a.id) as id FROM civicrm_contact contact_a LEFT JOIN civicrm_group_contact `civicrm_group_contact-16,49,44,96` ON contact_a.id = `civicrm_group_contact-16,49,44,96`.contact_id LEFT JOIN civicrm_group_contact_cache `civicrm_group_contact_cache_44` ON contact_a.id = `civicrm_group_contact_cache_44`.contact_id WHERE ( ( ( `civicrm_group_contact-16,49,44,96`.group_id IN ( 16,49,44,96 ) AND `civicrm_group_contact-16,49,44,96`.status IN ("Added") ) OR ( `civicrm_group_contact_cache_44`.group_id = 44 ) ) ) AND ( 1 ) AND contact_a.id NOT IN ( SELECT contact_id FROM civicrm_group_contact WHERE civicrm_group_contact.status = 'Removed' AND civicrm_group_contact.group_id = 104 ) UNION SELECT contact_id as id FROM civicrm_group_contact WHERE civicrm_group_contact.status = 'Added' AND civicrm_group_contact.group_id = 104;

+----+--------------------+-----------------------------------+-----------------+----------------------------------------------------+-----------------------------------+---------+------------------------------------+---------+-------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-----------------------------------+-----------------+----------------------------------------------------+-----------------------------------+---------+------------------------------------+---------+-------------------------------------------+
| 1 | PRIMARY | contact_a | index | NULL | index_is_deleted | 1 | NULL | 2282626 | Using where; Using index; Using temporary |
| 1 | PRIMARY | civicrm_group_contact-16,49,44,96 | ref | UI_contact_group | UI_contact_group | 4 | dbname_12202011.contact_a.id | 1 | Distinct |
| 1 | PRIMARY | civicrm_group_contact_cache_44 | ref | UI_contact_group | UI_contact_group | 4 | dbname_12202011.contact_a.id | 1 | Using where; Using index; Distinct |
| 2 | DEPENDENT SUBQUERY | civicrm_group_contact | unique_subquery | UI_contact_group,FK_civicrm_group_contact_group_id | UI_contact_group | 8 | func,const | 1 | Using where |
| 3 | UNION | civicrm_group_contact | ref | FK_civicrm_group_contact_group_id | FK_civicrm_group_contact_group_id | 4 | const | 1 | Using where |
| NULL | UNION RESULT | <union1,3> | ALL | NULL | NULL | NULL | NULL | NULL | |
+----+--------------------+-----------------------------------+-----------------+----------------------------------------------------+-----------------------------------+---------+------------------------------------+---------+-------------------------------------------+
6 rows in set (0.00 sec)

I am also seeing a large amount of full table scans taking place on the civicrm_mailing_event_queue table, which will be very CPU intensive. I would recommend adding WHERE conditions to these queries to help reduce the overall amount of rows being scanned.

# Time: 120121 10:20:57
# User@Host: root[root] @ serverWEB2 [10.178.230.22]
# Query_time: 7.589489 Lock_time: 0.000033 Rows_sent: 1 Rows_examined: 15253869
SET timestamp=1327162857;
SELECT id, email_id
FROM civicrm_mailing_event_queue

Donald Lobo · January 24, 2012, 04:47:51 pm

sorry am a bit slammed with other client work to look into this in the short term.

My advice would be to take a look at the queries and see what they intend to do and then write an optimized version of those queries and see if they give you the same results etc. Once done, please attach your patch and analysis here so others can take a look at it

thanx for working on this. it really will help scale civimail. if you have any specific questions on the queries ping us on irc

lobo

CiviCRM Community Forums (archive)

News:

Author Topic: civicrm cron job query (Read 1421 times)

eteflamingo

civicrm cron job query

Donald Lobo

Re: civicrm cron job query

eteflamingo

Re: civicrm cron job query

Donald Lobo

Re: civicrm cron job query

eteflamingo

Re: civicrm cron job query

Donald Lobo

Re: civicrm cron job query

eteflamingo

Re: civicrm cron job query

eteflamingo

Re: civicrm cron job query

Donald Lobo

Re: civicrm cron job query

eteflamingo

Re: civicrm cron job query

Donald Lobo

Re: civicrm cron job query