CiviCRM Community Forums (archive)

I have been having a lot of problems with the import feature lately. I am running 3.3.0 with Drupal 6.

I am practicing on a few records before moving my entire contacts (20K+ records)

First off, if I do beyond 150 records at once, I get a Bad Response From Server, which I assume is my browser timing out.  Any tips for remedy to this?

Secondly, I am getting a lot of import record errors, that are really not errors, like saying problem with urls (the urls are fine) and such.  Can I turn off record checking?

Third, when the I go through to Step 4, it asks me to download index.php and import freezes at 0% completed, nothing happens.

I can't get even 50 records to run through, much less 2000!

My php settings:
execution time: 250 seconds
memory: 256 M

I will try upgrading to 3.3.2

Ok, so I reduced the size of my CSV, look for any weird characters, and tried importing again.

Same thing, asks me if I want to download index.php at Step 4, nothing else happens.

I've seen that issue of it trying to download index.php a few times on the forums before, not just with imports. This might be relevant too if you haven't already seen it - http://forum.civicrm.org/index.php/topic,17228.0.html - no clear answers though.

With the other import issues, imports are a bit of a nightmare. A lot of people have ended up coding their own task-specific import alternatives using the APIs. I think some of these are shared and might be suitable - have a search on the forums.

When I use the import script, my most common cause of bogus errors is when there are trailing or preceding spaces in a cell. There are some other weird issues with lists of allowed values having funny suprises in them - particularly countries, where some of the allowed/disallowed countries are just strange. Also there are sometimes some issues with csv cells being read incorrectly by PHP if they aren't encased in quotes. I made an Excel macro that avoids this last issue which might help if nothing else works: http://forum.civicrm.org/index.php/topic,16442.msg70182.html#msg70182

The other essential import tip is, imports usually fall over and die if you try to import 'reference' data such as like relationships (e.g. current employer) at the same time as anything else. To get around this limitation I usually do imports in a multi step process like this:

1) Import basic data (names, addresses, emails, notes...)
2) Export these contacts' CiviCRM contact IDs and basic data to identify them (names, email addresses)
3) Line these IDs up with the original records
4) Import relationships one at a time with a 2 column sheet containing nothing but the two contact IDs (e.g. 'employee ID | employer ID')
5) Import activities as a seperate sheet using one column of contact IDs and various columns of activity data
6) Etc...

It's fiddly and frustrating but it works and avoids the numerous possible internal server failures that are otherwise possible. Imports are definitely something of a weak link.

Unless you've got a crazy beast of a high-powered server, if you are going to do a lot of imports, you generally need to configure PHP and MYSQL to cater for the import script, which generally means whacking up every buffer or memory limit to between 2 and 20 times the typical value. Have a search for this - I think some people have posted some examples of the settings that work for them.

Ok, I got things working after tweaking somethings, and found a custom field hat was causing the issue.

So, now it works fine with test data, but if I try and import like 900 records, I get Bad Server Response, sometimes at Step 2, sometimes at Step 3, and sometimes during the actual import at step 4.

I've increase all memory and timeout settings o both php and mysql to try and get it to work, but nothing helps.  It was working with 1,000 at a time, but now it doesn't.

Well spotted.

Do you have the IT knowledge to modify the code and add a check that external iD isn't >32 ?

X+