Duplicate Matching - caching issue?

[tomslee]

It looks to me like when I change the duplicate matching rules the change does not take effect right away. On the demo site I changed the individual rule to use the First Name (length 1, weight 5), Last Name (weight 7), and Middle Name (weight 3) with a threshold of 14. I create Jane Robin Smith and John Robin Smith, which should trigger the threshold - but they don't get flagged as potential duplicates.

I've had this kind of behaviour on my site, and it seemed to go away after a while. Is there some caching of rules happening that I could flush somehow?

Tom

[Piotr Szotkowski]

Hm, they do seem to appear in the ‘Find Duplicate Contacts’ search results. I added Jack Robin Smith and it got included right away, so I’m at a bit of a loss here.

Try deleting the Smarty template cache next time this occurs to you and let me know if it changes anything.

[tomslee]

I think I see the problem, which has two parts.

1. I assumed that the check for matching contacts done on the New Individual form was a check for potential duplicates. I think it is not - it's a check for drupal users instead. Am I right?

2. I was using the check on the New Individual form because, when I run a 'Find Duplicate Contacts' on my data set it fails. The reason (I see in the drupal logs) is that it goes beyond the memory that my server allows (I'm running on an ISP). Here is the log entry.
[Mon Sep 10 08:18:34 2007] [error] [client 72.142.181.97] PHP Fatal error:  Allowed memory size of 41943040 bytes exhausted (tried to allocate 6140 bytes) in /home/kwndp/kwndp.ca/public_html/sites/all/modules/civicrm/CRM/Dedupe/Finder.php on line 94, referer: http://activist.kwndp.ca/civicrm/admin/dedupefind?reset=1
Maybe I should post this separately, but I have a data set of 1500 individuals.

• Is there something I can do (simpler rules perhaps?) to minimize the memory requirements of a full duplicate contact search?
  Is there a piece-wise alternative?
  

[Piotr Szotkowski]

1. I assumed that the check for matching contacts done on the New Individual form was a check for potential duplicates. I think it is not - it's a check for drupal users instead. Am I right?

It’s a bit more complicated than that. In CiviCRM 1.8 we have two mechanisms for finding similar contacts – the new (de)duplicate engine that’s used in the admin and the old matching mechanism, that’s still used on new contact creation and import. To edit the matching rules, go to Administer CiviCRM → Contact Matching.

2. I was using the check on the New Individual form because, when I run a 'Find Duplicate Contacts' on my data set it fails. The reason (I see in the drupal logs) is that it goes beyond the memory that my server allows (I'm running on an ISP). Here is the log entry.
[Mon Sep 10 08:18:34 2007] [error] [client 72.142.181.97] PHP Fatal error:  Allowed memory size of 41943040 bytes exhausted (tried to allocate 6140 bytes) in /home/kwndp/kwndp.ca/public_html/sites/all/modules/civicrm/CRM/Dedupe/Finder.php on line 94, referer: http://activist.kwndp.ca/civicrm/admin/dedupefind?reset=1
Maybe I should post this separately, but I have a data set of 1500 individuals.

• Is there something I can do (simpler rules perhaps?) to minimize the memory requirements of a full duplicate contact search?
  Is there a piece-wise alternative?
  

If your dataset is similar (from the rules’ point of view), then it will generate many potential duplicates; if this is why you cross your memory limit, then creating a more complicated rules might be the solution.

Also, when you go to the ‘Use Rule’ URL (say, http://civicrm-18/civicrm/admin/dedupefind?reset=1&action=update&rgid=1), you can limit the scope of the search to a given group by adding &gid=7 to the end of the URL; try whether that makes CiviCRM use less memory on searches.

[tomslee]

Thanks Piotr. Assuming last name is pretty much a good indicator I split my individuals into two groups (A-M, N-Z) and deduped each of them using your URL tip. That worked a treat.

Plus, I now know how to prevent future duplicates on import. So I'm happy.

Tom