CiviCRM Community Forums (archive)

I want to programmatically import batches of CRM data from an external source into Civi.  The existing interfaces are pretty nice, but could be improved by hiding several implementation details from the calling code:

• Hide object encapsulation design, accept input data rows with any mix of Contact, Address, Contribution, or other fields
• Process batches using an internal loop which can be optimized according to particular Civi needs
• Implement import in the API layer (see also http://issues.civicrm.org/jira/browse/CRM-6273), accept strictly declarative data and return errors and warnings as a list

The algorithm will iterate over the input data in several passes, building up data according to the entity dependency graph.  The first pass will create contacts and supporting location, email records and so on.  The next pass will import contribution, participant and membership data and link to the previously created contacts.  Finally, module hooks are run, and a dedupe is performed.

My plan so far is to implement a Civi API which has something like the following call semantics:
$crm->Import->mixed(
    /*
     * Note that Civi field names might contain some
     * magic like "billing_" to assign basic secondary
     * attributes.
     */
    mapper: {
        "name": "contact.display_name",
        "email": "contact.primary_email",
        "bill_mail": "contact.billing_email",
        "amount": "contribution.total_amount",
    },
    /*
     * Records can be specified in heterogeneous formats, may
     * not completely fulfill the mapping schema, or can be
     * written as an array without the key names.
     */
    records: {
        { name: "Mikhail B", email: "intl@congress.it" },
        [ "Nom dePlume", "ndp@example.fr", null, null ],
        { bill_mail: "anonymous@admirer@scumlabs.corp", amount: "9999.44" },
    },
    options: {
        stopOnError: false,
    },
);


Output in this case would indicate that the third row could not be imported at all, but otherwise the import was processed successfully.  Custom, third-party code might have been run as hooks, or a subsequent api call might have access to the contact and contribution ids created during this call.
$result = {
    numSuccessfulRows: 2,
    errors: [
        { 
            message: "Invalid email address",
            rowOffset: 2,
            fieldName: "bill_mail",
            source: { bill_mail: "anonymous@admirer@scumlabs.corp", amount: "9999.44" },
        },
    ],
}

Obviously, this is a very rough outline.  Please share any thoughts!
-Adam

So, firstly, if create could accept either a params array or an array or params arrays & do multiple creates that would be step in the right direction? Or would it be better as a separate generic function ie.

Perhaps we should add an action to api/v3/Generic.php

function civicrm_api3_generic_import($apiRequest) {

}

which just provides a wrapper for the api create functions.

ie. suspect we should have 2 separate mechanisms / wrappers for multiple create + error reporting & for mapping.

(Writing on phone -- apologize for terseness)

A few thoughts:

1. Agree that it's important to support importing mixed entity data, but I don't think it should be encapsulation -- the " encapsulated" form seems like it will become complex and leakyy. Its  better to do something like API chaining which allows you to describe entity relations while exposing the canonical data model.

 2. Also think it's important decouple the mappings from the main loop. There are different strategies for running an importer's main loop (web based progress bars, cli, batching, queueing,etc.


Heres an alternative interface for definif mappings - ask the dev to provide as callback function: 

Function ($row) => $apiParams

Where $row is an array representing a raw, flat row of data, and the result is an array that can be passed to the API. The result array can be a tree (with nested API calls to update related entities).).  The decision about which main loop to use (all-at-once; batching; queueing) can be made a runtime