smart automatic merge dedupe?

[Erich Schulz]

mmmm 

btw Xavier, just so you know I can use the v3 api:

Code: [Select]

/**
   * Automaticlly merge two contacts.
   *
   * @see autoMergeSQL()
   *
   * @param $param array
   *  - 'id' integer contact_id of record to keep (injection attack safe)
   *  - 'id_duplicate' integer contact_id of record to be assimilated (injection attack safe)
   * @return array result
   *  - is_error
   *  - sql
   *  - error_message
   *  - report
   */
  static public function merge($param) {
    $keep = (int)$param['id'];
    $lose = (int)$param['id_duplicate'];
    // attempt to generate SQL:
    $result = AgcAutoMerge::autoMergeSQL($keep, $lose);
    if (!$result['is_error']) {
      // perform merge:
      require_once 'CRM/Core/Transaction.php';
      $transaction = new CRM_Core_Transaction( );
      foreach ($result['sql'] as $sql) {
        CRM_Core_DAO::executeQuery( $sql, CRM_Core_DAO::$_nullArray, true, null, true );
      }
      $transaction->commit( );
      // add notes to both contacts: //todo fix note created by
      $note = array(
        'entity_table' => 'civicrm_contact',
        'entity_id' => $keep,
        'note' => "This contact has been merged from the duplicate $lose",
        'subject' => 'Target of automerge',
        'version' => 3,
      );
      $result['report'] .= "Added note to $keep: ". json_encode(civicrm_api( 'note', 'create', $note));
      $note = array(
        'entity_table' => 'civicrm_contact',
        'entity_id' => $lose,
        'note' => "This contact was merged to $keep",
        'subject' => 'Duplicate. Deleted during automerge',
        'version' => 3,
      );
     $result['report'] .=  "Added note to $lose: " . json_encode(civicrm_api( 'note','create', $note));
    }
    return $result;
  }
   

This wrapper does the merge and adds a brief note to both contacts, I've updated the git-hub with the latest edition,

if anyone else is planning on putting this to use soon they may wish to pm me, as I maybe motivated to package it up better (ie that drupal module Lobo suggested)

any hints on how to access the current user id? I'm assuming if i pass that to the create note api as contact_id it gets stored in the 'created by' field??

[xavier]

Might be better to store it as an activity than a note (easier to search?)

Give me few more min and I try to chat with you
X+

[Erich Schulz]

for xavier - this is the current interface

Code: [Select]

/**
 * Automatically merge two contacts.
 *
 * Creates a note in both main and outer contact
 *
 * @returns string html
 */
function _agc_dedupe_merge($mainId, $otherId) {
  require_once 'class.automerge.php';
  $params = array(
    'id' => $mainId,
    'id_duplicate' => $otherId);
  $result = AgcAutoMerge::merge($params);
  $html = "<h3>Planned merger of $otherId into $mainId</h3>";
  if ($result['is_error']) {
    $html .= "<h4>Error</h4><p class='error'>$result[error_message]</p><p>$result[report]</p>";
  } else {
    $html .= "<h4>SQL:</h4>" . AgcAutoMergeDevWindow::arrayFormat($result['sql']);
  }
  return $html . '<p>Report:' . $result['report'] . '</p>';
}
menu item for .module file

Code: [Select]

  $items['agc/dedupe/merge'] = array( 'title' => 'Perform merge',
    'description' => 'Merge a pair of duplicate contacts',
    'file' => 'agc.automerge.inc',
    'page callback' => '_agc_dedupe_merge',
    'page arguments' => array(3,4), // $mainId, $otherId
    'access arguments' => $adminRole,
    'type' => MENU_CALLBACK);

I'm currently working on another function that will take an sql statement and loop through those... it's a bit fast and loose but gets this thing going.

[Erich Schulz]

just a little update that we've been running this code

and have now merged 1300 duplicates automatically

just a few glitches 

will be motivated to expand if others need this process

[Erich Schulz]

have now added in functionality to merge custom multivalue fields (in beta test)

[xavier]

That's great! Thx for all your work.

X+

[daven]

Hi Erich

Thank you for sharing your code. I will be using it for a project I'm on at CivicActions with around 80k contacts and guessing 10% duplicates right now. Auto-merging will certainly enable us to actually dedupe that many. I'll let you know how it goes.

[Erich Schulz]

Thanks Daven - good to know someone is giving it go... how'd you go?

[Donald Lobo]

This is now a Make It Happen for 4.2: http://civicrm.org/mih#massdedupe

Hopefully we'll meet the target and ship it with 4.2. We'll read this thread again when spec'ing the issue. Blog post and issue coming soon

lobo

[Erich Schulz]

thats nice to here Lobo - hopefully some of my stuff will be useable - I'm afraid I'm now only a few months out from an election so will be a bit distracted but happy to reply to messages etc

poor data quality, poor data entry of nicknames and desire not to overwrite customised greetings are currently major roadblock for us...

provision of a customisable nickname table may be a tremendous boost

[Donald Lobo]

Our current focus is on making mass merge quick and efficient.

I suspect data cleanup / improving data quality and data normalization will come in later releases

lobo

[Erich Schulz]

understand and laudable priority!

i decided that rather than making the automerge more complicated to deal with inconsistent data I was better just cleaning the data (and then setting crons to keep it clean)... this meant two simple and clean processes

be interesting to see how you decide to go