CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Core CiviCRM Functions (Moderator: Yashodha Chaku) »
  • Depuping empty custom fields
Pages: [1]

Author Topic: Depuping empty custom fields  (Read 771 times)

rocxa

  • I post occasionally
  • **
  • Posts: 40
  • Karma: 4
  • CiviCRM version: 4.5.5
  • CMS version: Drupal 7.34
  • MySQL version: 5.1.71
  • PHP version: 5.3.3
Depuping empty custom fields
January 28, 2015, 08:03:55 am
CiviCRM appears to be setup to consider empty custom fields as matches when deduping..

This makes sense in one sense. <empty> === <empty>

but..

If you have a rule such as:

Match 'email' and 'custom ID'  any records with the same email but no ID will match.  These records are generally created when allowing multiple participant bookings using the same email address.

For a party of 10 booking, you soon get 10x10 matches.. in your dedupe results for each group booking as they are all compared with each other (assuming they booked 10 people in)

Am I missing a setting somewhere that disables considering blanks as a match? or any other suggestions are welcome.

petednz

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4899
  • Karma: 193
    • Fuzion
  • CiviCRM version: 3.x - 4.x
  • CMS version: Drupal 6 and 7
Re: Depuping empty custom fields
January 28, 2015, 02:11:05 pm
can you work around by making a (smart) group of those with custom field is null

then another group of the contacts you want to dedupe excluding the above group (using the include/exclude custom search)

then use that group as your target for deduping and you might then avoid those records with the blank

sorry if that isn't the type of solution you need but thought it might help
Sign up to StackExchange and get free expert advice: https://civicrm.org/blogs/colemanw/get-exclusive-access-free-expert-help

pete davis : www.fuzion.co.nz : connect + campaign + communicate

rocxa

  • I post occasionally
  • **
  • Posts: 40
  • Karma: 4
  • CiviCRM version: 4.5.5
  • CMS version: Drupal 7.34
  • MySQL version: 5.1.71
  • PHP version: 5.3.3
Re: Depuping empty custom fields
January 29, 2015, 07:35:44 am
That sounds like a reasonable workaround and thanks for the idea.  I will definitely try it but...

we have 250'000 records and running almost any dedupe rules on groups tends to timeout and lockup mysql.  Which then needs to be restarted.

The real solution is probably going to be added a check box to rules which can ignore blank values.

JonGold

  • Ask me questions
  • ****
  • Posts: 638
  • Karma: 81
    • Palante Technology
  • CiviCRM version: 4.1 to the latest
  • CMS version: Drupal 6-7, Wordpress 4.0+
  • PHP version: PHP 5.3-5.5
Re: Depuping empty custom fields
January 30, 2015, 12:49:38 pm
It sounds like you may want to tune your MySQL settings/feed MySQL more RAM.  You may want to talk to Vermont Progressive Party, who have a similar situation in which they need a very large amount of RAM for dedupe during import, but much less RAM for the rest of the time, and see how they do it.

With that many contacts, you may also want to consider doing deduping via Lucene.  The Wikimedia folks have written Civi integration code for this scenario.
Sign up to StackExchange and get free expert CiviCRM advice: https://civicrm.org/blogs/colemanw/get-exclusive-access-free-expert-help

rocxa

  • I post occasionally
  • **
  • Posts: 40
  • Karma: 4
  • CiviCRM version: 4.5.5
  • CMS version: Drupal 7.34
  • MySQL version: 5.1.71
  • PHP version: 5.3.3
Re: Depuping empty custom fields
April 09, 2015, 03:45:15 am
A pull request has been made to skip blank fields from the deduping progress.  It speeds up deduping no end and stops all the false positives.

https://github.com/civicrm/civicrm-core/pull/5592

JonGold

  • Ask me questions
  • ****
  • Posts: 638
  • Karma: 81
    • Palante Technology
  • CiviCRM version: 4.1 to the latest
  • CMS version: Drupal 6-7, Wordpress 4.0+
  • PHP version: PHP 5.3-5.5
Re: Depuping empty custom fields
April 09, 2015, 07:38:16 am
rocxa - this is awesome, thank you!
Sign up to StackExchange and get free expert CiviCRM advice: https://civicrm.org/blogs/colemanw/get-exclusive-access-free-expert-help
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Core CiviCRM Functions (Moderator: Yashodha Chaku) »
  • Depuping empty custom fields

This forum was archived on 2017-11-26.