CiviCRM Community Forums (archive)

This is currently not doable out of the box, BUT i dont think its too hard for a developer to be able to extend this to allow the below.

A few thoughts on how this can be done:

a. extend the core import code (both preview and summary) to allow an offset/limit combination sent in via $_REQUEST

b. after processing the lines offset to offset + limit, the code redirects back to the same page with new offset till we are done processing

c. all the stats we collect will need to be saved during the redirection

if you / your org have the resources to help make the above happen that would be great. Alternatively doing a command line version of the import would be a good alternative

lobo

Do you wanna tell your dev team to get on IRC and we can chat and come up with a plan

we currently do the import via AJAX. to begin with, i would kill the ajax part and basically keep redirecting to same screen till done (varying offset). I think you can modify: templates/CRM/Import/Form/Preview.tpl to kill the ajax part

Some files to check:

CRM/Import/ImportJob.php, function run. I suspect this is the only one to modify and redirect back to same page if import not yet done

lobo

Not sure if you / your developers are attending civicrm developer training (http://civicrm.org/civicrm/event/info?reset=1&id=25) but definitely recommended and worth it for a folks integrating / extending civicrm. If so, we can chat more about it that time

if not, we are thinking of doing a few code sprints / BoF sessions during drupalcon and we could meet there

lobo

Another possibility that might be worth investigating (but Drupal-centric) is creating a handler for the migrate module. The migrate module already has drush integration & a useful interface. It is a possibility that would work if you already had your data in a table in mysql - it uses the table wizard module to expose the tables & choose fields and relationships between tables. Thus you can create an import set from a bunch of legacy tables, design an import & then test it & reverse it (a few times). When you are ready to go you can run it through drush.

It works at the moment for nodes, cck, taxonomies etc