CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion (Moderator: Donald Lobo) »
  • Dedupe Large Number of Contacts
Pages: [1]

Author Topic: Dedupe Large Number of Contacts  (Read 782 times)

crsikes

  • I’m new here
  • *
  • Posts: 2
  • Karma: 0
  • CiviCRM version: 4.0.1
  • CMS version: Joomla 1.6.3
  • MySQL version: 5.1.5
  • PHP version: 5.3
Dedupe Large Number of Contacts
March 02, 2012, 11:16:00 am
Greetings,

I am an administrators for a site with over 27,000 contacts, many of which are duplicates.  Contacts and contributions have been generally imported via CSV by a wide variety of volunteers, the result being several different methods for default deduping (i.e. some chose to use email address while others have used first and last name).

The result is a database with up to 8 duplicate entries for the same contact. Using the admin dedupe takes an unacceptable long time to run, invariably timing out in the process.  Attempts to update via querying tables have been frustrated by the inherent complexity of the system.

Any help/experience with deduping would be greatly appreciated.  Thanks.

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: Dedupe Large Number of Contacts
March 02, 2012, 12:13:57 pm
there is a dedupe hook which can be used to construct optimized queries for running dedupe rules. it can greatly reduce the load and prevent timeouts.

also -- coming in 4.2 is a bulk merge tool whereby you run your dedupe rule then do a "safe" merge on the result list (a safe merge is one in which there are no direct fields in conflict)
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

crsikes

  • I’m new here
  • *
  • Posts: 2
  • Karma: 0
  • CiviCRM version: 4.0.1
  • CMS version: Joomla 1.6.3
  • MySQL version: 5.1.5
  • PHP version: 5.3
Re: Dedupe Large Number of Contacts
March 02, 2012, 12:39:36 pm
Thanks for the reply. Do you have a link to the hook you mentioned?

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: Dedupe Large Number of Contacts
March 02, 2012, 12:51:54 pm
http://wiki.civicrm.org/confluence/display/CRMDOC41/CiviCRM+hook+specification#CiviCRMhookspecification-hookcivicrmdupeQuery
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Dedupe Large Number of Contacts
March 03, 2012, 12:21:02 am
You might want to update to 4.1 to see if the newest version works better (it has been improved, not sure if it was for 4.0 or 4.1)

X+
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion (Moderator: Donald Lobo) »
  • Dedupe Large Number of Contacts

This forum was archived on 2017-11-26.