CiviCRM Community Forums (archive)

Hey Xavier!

Thanks for replying

Some of your questions are answered in the code, but I’m very happy to clarify as as I’d love your help to properly assimilate this code…

autoMergeSQL is the primary interface and it’s what you’d interact with from the cli – and this is the function that needs an api wrapper around it. Note that this function only generates the sql – it does not execute it. (?yet) but it’s the function you want (I hope!!). this function returns a plain text error report on failure.

By data dictionary, I guess I mean the civicrm xml files… I read somewhere that is what civi uses to generate the some of the code.

Re the html, you’ll hopefully be pleased to hear its not as bad as you think. The three html function are for development and debugging purposes – run them to explain how the class is proposing to perform the merge and how it got there – these functions merely provide a window into the logic called upon by autoMergeSQL – if you wanted I could easily tweak them to return a nested array instead of a html report. Given the need to do more work on this I wanted to build some proper windows into the machinery as I when along.

(don’t worry about those additional tables – I didn’t make them, but they are part of the legacy environment I’m in – they would be harmless if they’re missing)

re “schema” – good question, sorry I didn’t realise would cause confusion (maybe showing my oracle roots) yep schema = database in mysql see http://lists.mysql.com/mysql/211616 

Again, I am sorry for not reusing the existing code – but (1) its very scantly commented and I couldn’t follow it (2) it looked like it needed major surgery to extract form dependencies and (3) it is full of significant ‘todos’ and ‘fixmes’

sorry you're correct there are no todos

but there are five more fixme in civicrm\CRM\Dedupe\Merger.php

thanks Xavier 

Thanks for the api links, but what i'm suggesting is that each conctact_id foreign key field gets a new attribute of "automerging behaviour", ie how should this record be handled during an automated contact merge?

- block automerge (ie because mergeing this information isn't safe without human review)
- updated (to point to target conctact_id),
- deleted altogether
- ignored
- examined in detail

if someone in the core adds these attributes then the automerge class just needs to read them (ie in the *_list functions)

the same applies to the columns of civcrm_contact that are examined in columnAutomergeBehaviour()... those behaviours eg "BlockIfGreater"  for do_not_email or "BlockOnDifferentValue" is_deceased should also go into the datadictionary then be read from there


re the entity_id being a reference to a participant - i *think* i have that covered with excludeEntitiesSQLList() which lists the entities which I thought are not a synonym for the a contact

Hey all, I would appreciate some more feedback on this class – I understand the importance of code reuse, but
in reality the only code I could have reused was to replace the functionality between lines 175-190. And there didn’t seem to be a simple way to refactor the many hundred lines of code that are designed to do something quite different into the actually very simple 15 lines.

Xavier autoMergeSQL() is ready to go – it just needs an api wrapper around it. It returns a single array with elements: is_error, error_message (on failure), sql (on success) and report.

A fair amount of effort went into making automerge class transparent and easy to refine and maintain by others. Lines 57 – 137 do nothing other than provide a trace report to make it easier for ongoing development. These function are purely there to quickly generate reports for the eyes of fellow programmers (these are ‘reports’ – they are not data, which is why they generate html), if you want the information reported by these functions then call the same function that the html reporter is accessing. 

Lines 208 – 254 clearly document the key settings.

Lines 255-322, 394-439 and 442-459 are designed to be replaced by meta-data reads of “data automerge behaviour”, defining (1) what to do with foreign key relationships (2) what to do with civicrm_contact fields (3) and any special record level processing of particular fields, respectively. As I’ve said I’m happy to work with someone in core to implement that. If you want…

Lines 364-393 implement a range of potential system reactions to non-identical data in records proposed to be automatically merged.

Most of the remaining lines are either simple api-wrappers or schema reading and filtering functions not available in the current code.

I’ll swap out the drupal api calls for civicrm. So I’m happy to keep working on this, but I need to know I’m not wasting my time.

xavier - so the wrapper you want is something like this - the return value is the contact_id on success,

Function automerge_api_interface($param) {
  $result = AgcAutoMerge::autoMergeSQL($param['id'], $param['id_duplicate']);
  if ($result['is_error']) {
    return array($result['error_message']);
  }
  else {
    // where is the api definition for execute query?
    CRM_Core_DAO::executeQuery($result['sql']);
    if {{{no error returned by executeQuery}}} {
      return ($param[id];)
    }
    else {
      return {{{errors returned by executeQuery();}}}
    }
  }
}
I know you’re all vollies and are only working for warm fuzzies too, but sorry guys when I contributed this I did it on the assumption that someone who is intimately familiar with civicrm would pick it up and go with it, I have a full time job, 3 kids and a wife, for a “hobby” I’m going to be sysadmin supporting hopefully about 20-30 users managing 20000 contacts in a major campaign culminating in March. I have a lot to do. I could have just hacked this de-dupe and got on with the other 20-50 things on my to-do list but this seemed like a way I could contribute to civicrm.

Sorry this is a long and windy way of saying I don’t have time to write formal test cases (even tho i understand their value!) and you’ll need to be creative to write a test that proves an entity has disappeared without leaving orphan entities or corrupted references.

Re using the sql vs api, I think the API is a great and amazing and really useful thing, but so is SQL! if you want to re-write the 15 lines of code that implement the writes (updates and deletes) be my guest (more specifically replace lines 183 and 188 with an api call if you want, but you'll lose functionality and make the code less transparent as far as i can tell). I'm also happy to help integrate datadictionary reads to find the foreign key references if they exist. The benefit of constructing the update as an sql script, other than scalability and efficiency is that an sql script can run as a single transaction and can also simply be aggregated into batches. The existing manual merge code uses a good smattering of sql - so I'm assuming that 'safe entity transfer' isn't a current api call

so my question about the entitiy_id columns is:

*  how to identify which ones are a direct foreign key into civicrm_contact.id ?

I've already got a list of all collumns with those names and i've had a stab at eliminating the entity id's that don't point at contacts my stab at the logic is in getContactKeyFields(). if i can do this with the api cool, how do i? sorry i currently i can't see how, i can only get as far are pulling a list of field names, not the foreign key relationships

Thanks Xavier for the many pointers

 

At Xaviers suggestion I've reposted the code here: https://github.com/ErichBSchulz/CiviCRMAutoMerge

I've done a major reorder of the functions around a bit into a more logical clustering - but the code itself is little changed since the first version. Only code changes are to remove the "support code" to a separate class, and de-drupalise the db calls.

 

Hi,

I really would like to have it as an API. Ok, I need to work on a hook to add/override API methods then, let me think about it.

X+