CiviCRM Community Forums (archive)

Attempting to import with no customl fields - 116 records.

Preview (step 3 of 4)
The information below previews the results of importing your data in CiviCRM. Review the totals to ensure that they represent your expected results.

Click 'Import Now' if you are ready to proceed.

Importing records...

0%0%

  No processing status reported yet.
We encountered an unknown error: [object Object]

Why would I get an unknown error?  I have used this list to import in drupal, so I know it is fine.

Nope, well...I already know this data works on the demo - I have done this step previously on another issue with this data.

What would be the apparent reasons as to why I would receive this error on Joomla 1.0 with civicrm.1.8.10954-joomla-php5.

Thanks,
Kevin

I had the same problem when importing a large database (13K records) which appear to be memory related.

Oddly, processing continued even after the error, if I waited long enough a large number of records actually imported and the status bar appeared, before it ultimately timed out on an incomplete import.

By trial and error, I found that I could import the file in batches of about 2K records apiece without encountering the problem.

I wish it had been as simple to clean up the raw data (voter registration roll from our County Clerk's office who had entered it inconsistently, and with various date formats) so that we could eliminate having huge numbers of rejects.  Still beat handkeying by a country mile.

J-M +

First off, I want to say that CiviCRM has far exceeded our expectations, and has allowed us to get way out ahead of our opposition in organizing for what has become an extremely heated local campaign.  Most of our glitch-related inconveniences are the result of our own calculated decisions: (1) to install on a Joomla! platform instead of Drupal with which CiviCRM is more tightly integrated, and (2) to install a beta version on a production site.  I won't bore you with the details of why we made these choices, but it was risk-benefit stuff with full knowledge of the potential complications.  Now, back to your regularly scheduled broadcast:

I do agree that import is quite inefficient in CiviCRM and we can (and will) improve it significantly in  future 2.x release. That said a few comments

I've made do with much worse.

1. Make sure that geocoding and other network services (USPS address cleanup) are disabled when importing. This is even more time consuming (network latency) and will slow down things significantly

I've made not of this great tip, even though we have not yet implemented such services.

2. Increase PHP memory and execution limit to some high numbers. We've fixed import to not leak a lot of memory, so 64M should be enough, but php execution limit should be raised to 60 minutes or more depending on the size of the file

This, of course, is not always possible.  Working in a largely rural community on local projects, I usually have to work within the limits of a shared hosting environment.

3. Cleanup your data before import. I think it is better/faster to write a script to do data cleanup on the csv file rather than expect CiviCRM to be all powerful and all encompassing. In the 2.x release we will be pushing for a simpler faster import and potentially obsoleting a few features rather than adding any new ones. Keeping things simple allows us to optimize code much better

I think this should be emphasized in the docs.  We figured it out the hard way, when our reject rates were through the roof.  Throwing together a simple Excel macro for cleanup and export to a readable CSV was our quick-and-dirty fix.

4. I suspect at some point we will need a command line importer which can break large files into smaller chunks and import based on a specific mapping. This would be a great contribution from the community. Please contact us if you have the time and resources to make this a reality

I wish I had the time (and the skill) to volunteer for this.  Whoever does might want to take a look at the import routines employed by phpGedView.  Their development team wrote a batch processing import which doesn't require leaving their admin panel, and is configurable to cope with large CSV's and draconian time and memory limits on shared servers.  They're on sourceforge, and maintain a presence at http://phpgedview.net 

J-M +