Depuping empty custom fields

[rocxa]

CiviCRM appears to be setup to consider empty custom fields as matches when deduping..

This makes sense in one sense. <empty> === <empty>

but..

If you have a rule such as:

Match 'email' and 'custom ID'  any records with the same email but no ID will match.  These records are generally created when allowing multiple participant bookings using the same email address.

For a party of 10 booking, you soon get 10x10 matches.. in your dedupe results for each group booking as they are all compared with each other (assuming they booked 10 people in)

Am I missing a setting somewhere that disables considering blanks as a match? or any other suggestions are welcome.

[petednz]

can you work around by making a (smart) group of those with custom field is null

then another group of the contacts you want to dedupe excluding the above group (using the include/exclude custom search)

then use that group as your target for deduping and you might then avoid those records with the blank

sorry if that isn't the type of solution you need but thought it might help

[rocxa]

That sounds like a reasonable workaround and thanks for the idea.  I will definitely try it but...

we have 250'000 records and running almost any dedupe rules on groups tends to timeout and lockup mysql.  Which then needs to be restarted.

The real solution is probably going to be added a check box to rules which can ignore blank values.

[JonGold]

It sounds like you may want to tune your MySQL settings/feed MySQL more RAM.  You may want to talk to Vermont Progressive Party, who have a similar situation in which they need a very large amount of RAM for dedupe during import, but much less RAM for the rest of the time, and see how they do it.

With that many contacts, you may also want to consider doing deduping via Lucene.  The Wikimedia folks have written Civi integration code for this scenario.

[rocxa]

A pull request has been made to skip blank fields from the deduping progress.  It speeds up deduping no end and stops all the false positives.

https://github.com/civicrm/civicrm-core/pull/5592

[JonGold]

rocxa - this is awesome, thank you!