Author Topic: Invoking the Import Classes (Read 737 times)

xcf33 · May 18, 2010, 11:49:07 am

Hi guys, I'm in the process of creating a scalable import process for civiCRM and is running into blocks when it comes to the Import Classes

So in CRM/Import/

There's

Code: [Select]

ImportJob.php
Which contains interesting methods such as

Code: [Select]

public function runImport(&$form, $timeout = 55)
public function isComplete( $dropIfComplete = true )

Etc etc,

It seems this will be the class to call when you want to run a import job

Additionally, I found Importer.php in the same folder which contains the following:

Code: [Select]

    public function runIncompleteImportJobs( $timeout = 55 ) {
        $startTime = time();
        $incompleteImportTables = CRM_Import_ImportJob::getIncompleteImportTables();
        foreach ($incompleteImportTables as $importTable) {
            $importJob = new CRM_Import_ImportJob( $importTable );
            $importJob->runImport(null, $timeout);
            $currentTime = time();
            if ( ( $currentTime - $startTime ) >= $timeout) {
                break;
            }
        }
    }

This seemed to be tremendously useful for me because the contact import process hangs up for me after step 2 (Mapping), however, the temporary import table will be populated. If I can just run this script on a cron it could potentially help me solve the timeout issue.

My question for this class is, What is considered an incomplete job? Where are the mapping and import configuration saved? Where can we set the grouping and tagging configuration if we run this. And most importantly, how to invoke this class in an independent script outside of CiviCRM. (I'm getting a lot of path errors due to the require_once path dependencies)

Any help would be much appreciated!

Donald Lobo · May 18, 2010, 02:16:10 pm

hey chang:

i just spoke with wes (who wrote the sql importer part of the code) and we tried to track back who wrote the code and did not succeed (the branch it was developed on was deleted after the merge). If i had to guess, a lot of it was trying to write a good abstraction that we would move into at some point

So i dont think we store the various config options (group/tag/mapping etc) in the db currently

I've been thinking about the batch import process, i think we need 3 calls:

civicrm_import_create_table( $fileName ); // creates and return the table name

// import rows from $offset to $offset + $limit from $tableName
// the import parameters are in the params array
civicrm_import_import_rows( $tableName, $offset, $limit, $params )

civicrm_import_delete_table( $tableName ); // drops the table

things are getting a bit better now, so i can work with you on the above interface and get it going. Would be good to design a set of unit tests that we can use to test functionality before we write code. will help us improve the import code this way

ping me on IRC if the above makes sense and we can get the ball rolling

lobo

CiviCRM Community Forums (archive)

News:

Author Topic: Invoking the Import Classes (Read 737 times)

xcf33

Invoking the Import Classes

Donald Lobo

Re: Invoking the Import Classes