CiviCRM Community Forums (archive)

Hello,

I am thinking that the Drupal role CiviMember sync could be improved. The site I am using is on 3.4.7 with ~18,000 membership records.

One of the problems I am encountering is running the manual sync - it seems like this isn't a great candidate for a web based script.

I was having some issues getting this script to complete so I increased the memory and time limit with the following

+++ b/drupal/sites/all/modules/civicrm/drupal/modules/civicrm_member_roles/civicrm_member_roles.module
@@ -188,6 +188,9 @@ function civicrm_member_roles_manual_sync()
  */
 function civicrm_member_roles_manual_sync_submit($form, &$form_state)
 {
+    set_time_limit( 600 );
+    ini_set( 'memory_limit', '256M' );
+
     if ( ! civicrm_initialize( ) ) {
         return;
     }

That conked out the production server so I moved to my local development environment... 

I tried it multiple times and got it to finish once I think but most of the time i got a mixture of various timeout, memory 502 errors. The one time that (I think) it finished, the memory usage was 371mb I did a memory_get_usage() at the start of the while loop in _civicrm_member_roles_sync which does all the updating, and it was 42mb.

I'm not sure what is chewing up the memory or how to go about reducing the usage (apart from installing XDebug which I could do) but any pointers/ideas? do other people update this number of contacts without problems?  Thoughts appreciated.

The other thing I am wondering about is if we are doing a good enough job on the different synchronisation methods.

I understand that the problem with this method is that when someone who is logged in gets a new membership, they have to log out and log in again before they get that option, which sucks a fair amount.

It seems like sync when a membership is updated is much better since you could 1) run the mass sync 2) select this option and 3) forget about it. One thing I haven't investigated yet is does this also get fired when we run the membership update cron script??  That script does call civicrm_contact_membership_create. I'm not sure if civicrm_contact_membership_create calls hook_civicrm_post() but imagine if it did, that would make the mass update redundant.

So my next question is doesn't having the 'Synchronize when membership is updated' option make the 'Synchronize whenever a user logs in or logs out. This action is performed only on the user logging in or out' redundant? Maybe we should deprecate it / remove it.

Also, 'Synchronize when Drupal cron is ran. This action will be performed on all users and contacts.' seems to me like a crazy way to do the sync if the drupal cron is run about once an hour (which i think is a pretty standard option) or so and you have any number of contacts.  I think we should deprecate it as well (unless there are some use cases where it isn't a crazy choice).

So to summarise the suggestions / questions

1) is the mass sync. called when we do a bin/UpdateMembershipRecord.php and if so, should we
 a) remove the mass sync button from the UI and
 b) suggest that people do this using the membership sync

2) shall we remove the sync when someone logs in out, or at least change it to an additional option so it can be belt and braces rather than at the expense of the better 'Synchronize when membership is updated' method

3) why is the script so memory hungry and can we make it less so? if it is going to get called for each contact when we run bin/UpdateMembershipRecord.php, am i right in thinking that we need it to use less memory

_civicrm_member_roles_sync didn't get called from bin/UpdateMembershipRecord.php.

In 3.4, that script is calling an apiv2 function which doesn't call the hook.  Since that script has been rewritten in a later version, it didn't seem worth touching.

I did a work around which I have detailed below.  I still think we should improve that module along the lines mentioned in my previous post though, and will come back to that once they have upgraded to 4.1 which i imagine will be in the next 6 months.

Additionally, I think that they two methods should be checkboxes, not radios, so you can choose to update the status both when the log in and out and when membership is updated - low overhead and belt and braces.

My work around in the mean time in case people are interested...

i did a kludgy hack on admin/settings/civicrm_member_roles/manual_sync page so that it could be chunked via the ui in the following format:

admin/settings/civicrm_member_roles/manual_sync?offset=0&limit=500

i then called a series of URLs one after the other (automated via a shell script on a local machine to deal with the fact that you need to be logged in)

--- a/drupal/sites/all/modules/civicrm/drupal/modules/civicrm_member_roles/civicrm_member_roles.module
+++ b/drupal/sites/all/modules/civicrm/drupal/modules/civicrm_member_roles/civicrm_member_roles.module
@@ -179,7 +179,9 @@ function civicrm_member_roles_manual_sync()
                                                        '#type' => 'submit',
                                                        '#value' => t('Synchronize CiviMember Membership Types to Drupal Roles now'),
                                                        );
-   
+    if(isset($_GET['limit']) && isset($_GET['offset'])){
+      _civicrm_member_roles_sync();
+    }
     return $form;
 }
 

@@ -443,6 +446,9 @@ function _civicrm_member_roles_sync( $ext_uid = NULL, $cid = NULL )
         $memTypes = implode( ",", $all_rule_membership_types );
         // OPTIMIZATION: "SELECT DISTINCT contact_id" would be better if field were indexed
         $sql ="SELECT DISTINCT contact_id FROM civicrm_uf_match";
+        if(isset($_GET['limit']) && isset($_GET['offset'])){
+          $sql.=" LIMIT {$_GET['limit']} OFFSET {$_GET['offset']}";
+        }
         $contacts = CRM_Core_DAO::executeQuery ( $sql,CRM_Core_DAO::$_nullArray );
         if ( $contacts->N == 0 ) {
             return FALSE;