CiviCRM Community Forums (archive)

We are finding problems with the CiviCRM batch import function for individuals.

If we import with "update" as the option for handling duplicates, the system just returns us to the Step 2 screen, with no error messages. Whereas, with the "skip" option, it works.

We are running CiviCRM 1.8 on Drupal (through the CivicSpaceLabs Online system).

Has anyone else found glitches in the batch import function like this?

lobo,

but then, won't that make my contacts' live data visible to the public?

or, will it just be administrators like you that see it? i have to ask that.

thanks!

Lobo,

That won't work, because some of our files loaded in fine (after the system gave us good error messages and we cleaned up the problem data fields).

Others loaded in with inconsistent glitches and lack of error messages.

When we loaded the last file set to "update" duplicates, it just kept returning to the Step 2 screen, without any error messages, progress reports, or other explanations.

When we loaded that file set to "skip" duplicates, it seemed to make it all the way through and then gave a fatal error message. (All the data seemed to be loaded in, though).

We also saw inaccurate progress messages like "loading next 200 out of 100" (then 300, 400, etc. out of 100). It starts to erode my confidence in the system.

What we could do is set you up with a temporary account on our system and email the file to you so you could try loading it into our system and watch its behavior. Would that work for you?

We know we have bad "country" data in a couple of records -- for Germany & Nigeria -- since CiviCRM seems to enforce the "alpha-2" version of the ISO standard for country code abbreviations (which the documentation does not tell us anywhere, by the way -- at least, we haven't found it).

So, we should see some good clean error messages if the system is working right. The first batch that we loaded by the way (when there was no data in our system) worked great -- wonderful error messaging, etc. It seems that managing duplicates is where the CiviCRM batch file import function is choking?

Thanks!

Bill

For anyone who may have a similar problem, here is what we learned to work around it.

Selecting the "no duplicate checking" helped a lot. De-duping is a complicated art and MySQL is a limited search engine. CiviCRM documentation does not show its de-duping logic to the users. So, taking out this unknown helped a great deal.

We also learned that the CiviCRM import function seemed to break down when attempting to load more than 500 records at one time. Up to that point, with de-duping off, it functioned well -- returning good error messages for any bad data.

Hope this helps? It worked for us.

Thanks! I found that earlier, but after my first posting on this topic.

We never changed the default setting, so the fact remains that the import function failed when the de-duping option was flagged, but performed quite nicely (up to 500 records per batch) when it was not.

Anyway, now that our data is all in the system, is there any way within CiviCRM to check for duplicates? So far, I have not found any canned function to de-dup or even check for them.

thanks.

de-duping works okay for small batches, but seems to time out (blank screen) for larger ones.

i will ask the csod people if we can increase some of my settings. that should do the trick.