Author Topic: Import never completes (Read 4003 times)

fearlsgroove · July 25, 2008, 08:34:16 pm

Hi, I'm trying to import ~500 contacts, and the import never quite completes. In fact I think the import does finish doing all it's work, but the form never completes and sends you to the "I'm done" screen. This is happening on a VPS where the site will be hosted, but I don't have that problem on my lab machine.

It seems likely its a server config issue since it works fine in my lab. The VPS with problems is a cPanel/WHM machine; I've tried a bunch of things to work around the problem. I've reproduced the issue with all possible combinations of Apache 1.3.x, 2.0.x and 2.2.9 with PHP 5.2.6 running as both a CGI and a module.

Incidentally I also have a separate issue with the import attempting to run PHP as a FastCGI: I get a 500 error promptly at 40 seconds on the POST the importer does to /civicrm/contact/import -- this post is the same post that fails to complete and notify the screen that the import is done with the other problem. But here it aborts after 40 seconds.

Can anyone shed any light on what may be causing this issue? TIA!

Dave Greenberg · July 26, 2008, 01:45:00 pm

Not sure what is happening - but there were several issues w/ import stalling before summary screens in earlier 2.0 versions on Joomla. Please post the version of CiviCRM you are using and on what version of Drupal or Joomla. Also useful to see if import runs properly on a much smaller set of data (since this might be a memory issue).

fearlsgroove · July 26, 2008, 04:55:40 pm

Hi, thanks for the reply. I'm using CiviCRM 2.0 with Drupal 5.8. I tried a number of variations to see if I could narrow down the issue, with strange results.

First, I solved the fastCGI thing. Since I'm using cpanel I had to modify /var/httpd/conf/php.conf and add relevant fastcgi directives there. The key in this case was adding this directive:

Code: [Select]

IPCCommTimeout 600
This prevented FastCGI processes from being squished for running to long.

I should mention I'm uploading the same file over and over again telling it to update records it finds. The problem remains with the stalled imports however. At first I tried the first 100 lines of the same file I'd been experimenting with. This attempt stalled at the very end.

Next I tried just the first 20 lines. This stalled in the same way.

So next I deleted the database, renamed the civicrm settings file and reinstalled CiviCRM (lucky for me I'm just trying to get transitioned so I can do this). This time the 20 line upload worked fine. However the 500 line upload had the same issue.

So next I deleted and reinstalled civi again, and tried the whole 500 line file. Stalled again.

So I tried another file I need to import, this one would import just the organization names from a file with more fields. There were about 300 records out of which about 140 had organization names, so it should end up importing about 140 records. This had the very bizarre behavior of loading the initial import screen into the progress update area along with a javascript pop up that asked I'f i'd like to reload and try again. So it looked like a page within a page. Hope that makes sense.

Another interesting fact is that the imports are completing successfully. I didn't verify all the actual data, but the record counts are correct as far as the number of contacts and organizations that should have been imported. So this is merely a UI issue. Nonetheless I'm anal and have to find a fix

One more note: while I didn't see this error in the round of testing I just described, I did find a pile of this error message repeated in Drupal's log from my previous round:
array_key_exists() [<a href='function.array-key-exists'>function.array-key-exists</a>]: The second argument should be either an array or an object in [home directory]/sites/all/modules/civicrm/CRM/Import/Parser/Contact.php on line 1228.

Related? Dunno. Guess I'll start looking into the code next if no one can shed some light.

fearlsgroove · July 26, 2008, 04:56:56 pm

Sorry -- Using CiviCRM 2.0.5 to be exact.

Thanks

Dave Greenberg · July 27, 2008, 09:29:30 am

If you can get the problem to replicate w/ a small de-sensitized set of rows on our public sandbox - that would be great. Sounds like there might be something in the data triggering this bug which could be good to catch.

http://sandbox.civicrm.org

If you can recreate it - please file an issue on the issue tracker w/ the attached sampled file. (Of course if you dug in to the code a found the culprit - all the better - u can attach a patch to the issue :-) ).

fearlsgroove · July 30, 2008, 08:22:04 am

So it seems this problem is server specific, not code specific. Some of my issues were being caused by "MySQL has gone away" problems, so some performance tweaking there helped, however the hang at the end of the process I still haven't figured out.

I broke my lists out into smaller groups and each group individually works fine. I was able to get it to occassionally blow up on some groups; this seems to have been related to MySQL errors above. However I'm certain it has nothing to do with the import code itself.

Unfortunately I'm no closer to solving the underlying server configuration issue. The only experiment left I can think of is to do a network trace, which I haven't gotten around to. The real problem here is that we're trying to perform a long running job in an http context, which is generally a bad idea.

I haven't looked into the code that runs the ajax update job, but some real solutions may be to change the architecture of the import process. For example, if the main import request doesn't send ANY data back at all until the process completes, you'll hit timeouts in several places: the browser, the apache server, etc. This can be solved by occassionally sending a few bytes back just to let both sides know the process is still alive and not hung.

Ideally, the import process would check to see if a shell exec or similar process is available and process the job asynchronously. The ajax status check is already setup to be able to handle this; you'd just have to change it so that it actually posts "I'm done" instead of not updating status anymore when the import is finished.

If a shell exec isn't available, perhaps the import could be broken up into manageable batches? The import could split it up into 50 record chunks, process the 50 and return to the initial caller with info that there's more to come, which would then call again. There's some extra overhead here but you'd never time out.

I suppose maybe I'll mosey on over to the dev boards and see if I can figure out how to get a 2.1 Drupal install from SVN setup.

Donald Lobo · July 30, 2008, 11:50:33 am

You might also want to check and comment on this thread: http://forum.civicrm.org/index.php/topic,4148.html

Currently import is basically a multi-part process:

setup -> verify -> import

I suspect it would be fairly easy to write a PHP script to do the above in one step

The parameters could be the normal import parameter, the field mapping and the line range the script should import.

We have most of the code to do this. Someone just needs to stich it together. Wanna take the lead and do it? You can jump on IRC and we can help u get started

lobo

CiviCRM Community Forums (archive)

News:

Author Topic: Import never completes (Read 4003 times)

fearlsgroove

Import never completes

Dave Greenberg

Re: Import never completes

fearlsgroove

Re: Import never completes

fearlsgroove

Re: Import never completes

Dave Greenberg

Re: Import never completes

fearlsgroove

Re: Import never completes

Donald Lobo

Re: Import never completes