CiviCRM Community Forums (archive)

I'm encountering several non-repeatable annoyances (well, that's not the first word that comes to mind) trying to import data into various instances of CiviCRM.

First, with memory_limit set to the default 64M, I am lucky to be able to import 1000 rows at a time before I encounter the infamous "500" error -- at different (random) points in the import process, for the exact same import file! 

I made the following changes to my php.ini and now I'm at least able to consistently import 1500 or so rows at a time (though I'm not sure which change is most responsible).  Not great, but better than before... though these values certainly appear to be overkill for files ranging from a few hundred to 4500 records and sizes up to 700K bytes:
        max_execution_time = 3000
        max_input_time = 900
        memory_limit = 512M

Next, the CiviCRM import process never imports as many records as there are lines in the original csv file.

After the import completes, when I export what made it into CiviCRM and diff that with my original import file (to create a "missing" file which I can then re-import), I see:
a) a random number of records are imported, then a random number of records are skipped, then another bunch of records are imported, then another bunch skipped, etc. (i.e. the skipped records aren't all at the beginning or at the end, they come in "bunches")
b) sometimes when I import the "missing" file, everything imports successfully -- and there are no differences between the records in the original file and the records in the "missing" file.  This is usually when only a hundred or so records are missing.
c) other times when I import the "missing" file, not all of the records in that file are recognized (same as 'a' above) and the ones that *are* recognized are flagged as duplicates.  This is usually when several hundred records are missing.

Finally, one other thing (though I don't think it's related):  After moving my CiviCRM instance from the development site to the production site, and following the procedure in http://wiki.civicrm.org/confluence/display/CRMDOC/Moving+an+Existing+Installation+to+a+New+Server+or+Location, I noticed that the record in the civicrm_domain table still had the old location specified in the config_backend field for the "configAndLogDir" location.  Where is the configAndLogDir location specified?

It might be possible that there were errors in few rows hence skipped. Check your error file generated at the end of import.

There's no error file generated... it's just an incomplete import.  For example:

Original import file:
record1
record2
record3
record4
record5
record6
record7
record8
record9
record10

records imported:
record1
record2
record3
record7
record8
record10

No errors reported, no rows flagged as duplicates, no nothing (except a line of output telling me how many rows were imported, which I know is an incomplete import).

second import file:
record4
record5
record6
record9

Sometimes, all records in this second import file are imported successfully (begging the question of why they didn't import successfully the first time around), sometimes a subset of these records are imported and I have to create a third import file containing the subset of records that did not import either of the first two times.

Wash, rinse, repeat as necessary until all records are imported.  Way too timeconsuming a process!

There might be same contact in your CiviCRM db, hence it is flagged as duplicate.

I'm importing into an empty database... so, no chance of duplicates (and anyway nothing is being flagged as even a possible duplicate). 

Does it crash in the middle, or something when you import ?

That's weird, there is something wrong in your server/configuration because it should happen (obviously) and hasn't for all the other installs.

Ajax is used in the import, might hide some errors. Do you have access to the logs ? Any error ?

What CMS are you using ? I'm sure there is a way of increasing the verbosity of the log.

I just tried an import of a CSV file with 3332 records (intending to add them all to a specific group)

After the first 900 records imported, I got:

500 Server Error
The web server encountered an error or misconfiguration and was unable to complete your request. If this error persists, please contact the webmaster, and inform them of the time the error occurred, and anything you might have done that may have caused the error.

No other error message.

At this point, I hit the back button of my browser, respecified the group I wanted to add all the imported records to, and clicked "Import".  After 950 records I got the same 500 error.

Again I repeated the "back button, respecify group, import" and got another 950 records imported.

The last time through, I got a message to the effect that 386 records were imported and added to the group.  But (fortunately) when I did a query of all records that have a particular field set (i.e. a field which I knew got set during the import process) the results included all 3332 records.  I was able to add them to the group from the search results (at least, all but the 386 that were already in the group). 

Of course, I'd have preferred all 3332 records to have been imported all at once, but this is probably the next best thing.

I would estimate it to be 80 - 100 hours.

I would commit to funding up to 10% of that up to 10 hours if this is webaccess work/prices