Import Errors, Times Out, or Repeats Screen for Step 3 of 4

[philiphart]

I've eventually managed to get our data imported in chunks but thought I'd report the issues in case others are having similar problems.

We're on CiviCRM 2.0.2 with eApps hosting.

I was trying to import 580 individual contact rows from a CSV file with around 24 columns of straightforward data.

If I tried all the data at once, the import process seemed to time out after a minute or two and returned a blank screen after Step 2 of 4 at the URL: http://xx.xx.xx.xxx/civicrm/import

With blocks of 50 contacts, Step 2 worked successfully but after processing Step 3 for a minute or two, the screen returned to showing the initial Step 3 of 3 screen instead of Step 4. The processing status never changed from 0%. However, usually the data had actually imported ok but was just not reported as such.

When I tried to import 100 records, after a minute the processing status was reported as 50% ("50 of 100 records - 72 seconds remaining") but then shortly afterwards I got the error "We encountered an unknown error in setError: Error: bad http response code:500". Around 97 records were actually imported but the last few were not.

The only error in the httpd logs (repeated many times over a long period of time) was "mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed" but I don't think it is relevant.

With small numbers of contacts (~30 or less) the process seemed to work as intended.

cheers
Phil

[sodder]

I'm at 2.1.2 hosted on godaddy and am running into a similar problem.

I have 838 records to import, 12 columns per record.  I've scrubbed the data to know for sure that there are no unprintable characters etc but still cannot import the data successfully.  As suggested doing 30 at a time is my next attempt but any ideas on what may be wrong or data that I can collect to help debug the issue.

In the directory .../sites/default/files/civicrm/upload the file CiviCRM.log has the following (Note I've removed the real email address:

Nov 26 13:07:06  [info] $Fatal Error Details = Array
(
    [callback] => Array
        (
           

• => CRM_Core_Error
[1] => handle
        )

   

Code: [Select]

=> a515ac9c2796ca0e23adbe92c68fc9fc
    [message] => DB Error: a515ac9c2796ca0e23adbe92c68fc9fc
    [mode] => 16
    [debug_info] => INSERT INTO civicrm_email (contact_id , location_type_id , email , is_primary , is_billing ) VALUES ( 4271 ,  1 , 'someone@emailaddress.com' ,  1 ,  0 )  [nativecode=1213 ** Deadlock found when trying to get lock; try restarting transaction]
    [type] => DB_Error
    [user_info] => INSERT INTO civicrm_email (contact_id , location_type_id , email , is_primary , is_billing ) VALUES ( 4271 ,  1 , 'someone@emailaddress.com' ,  1 ,  0 )  [nativecode=1213 ** Deadlock found when trying to get lock; try restarting transaction]
    [to_string] => [db_error: message="DB Error: a515ac9c2796ca0e23adbe92c68fc9fc" code=0 mode=callback callback=CRM_Core_Error::handle prefix="" info="INSERT INTO civicrm_email (contact_id , location_type_id , email , is_primary , is_billing ) VALUES ( 4271 ,  1 , 'someone@emailaddress.com' ,  1 ,  0 )  [nativecode=1213 ** Deadlock found when trying to get lock; try restarting transaction]"]
)


Nov 26 13:07:06  [info] $backTrace = /home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/Error.php, backtrace, 194
, handle,
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/PEAR.php, call_user_func, 912
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB.php, PEAR_Error, 969
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/PEAR.php, DB_Error, 574
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB/common.php, raiseError, 1903
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB/mysql.php, raiseError, 898
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB/mysql.php, mysqlRaiseError, 327
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB/common.php, simpleQuery, 1216
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB/DataObject.php, query, 2394
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/DB/DataObject.php, _query, 1039
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/DAO.php, insert, 279
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/BAO/Email.php, save, 75
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/BAO/Block.php(313) : eval()'d code, add, 1
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/BAO/Block.php, eval, 313
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/BAO/Location.php(80) : eval()'d code, create, 1
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/BAO/Location.php, eval, 80
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Contact/BAO/Contact.php, create, 255
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Contact/BAO/Contact.php, create, 1413
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Import/Parser/Contact.php, createProfileContact, 1160
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Import/Parser/Contact.php, createContact, 560
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Import/Parser.php, import, 379
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Import/Form/Preview.php, run, 311
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/Form.php, postProcess, 247
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/StateMachine.php, mainProcess, 166
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/QuickForm/Action/Next.php, perform, 63
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/HTML/QuickForm/Controller.php, perform, 203
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/packages/HTML/QuickForm/Page.php, handle, 103
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/Controller.php, handle, 232
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/CRM/Core/Invoke.php, run, 197
/home/content/T/h/e/Theatreink/html/internal/sites/all/modules/civicrm/drupal/civicrm.module, invoke, 315, civicrm_invoke,
/home/content/T/h/e/Theatreink/html/internal/includes/menu.inc, call_user_func_array, 348
/home/content/T/h/e/Theatreink/html/internal/index.php, menu_execute_active_handler, 18




[Kurund Jalmi]

Sodder:

Can you replicate this on http://drupal.demo.civicrm.org, If you are able to replicase, do file an issue in jira and also attach sample csv file to replicate the issue. It would be helpful if you also specify your mapping.

thanx

Kurund

[sodder]

Kurund,

Thanks for your reply,  I will try and reproduce the issue. It may take some time since I'll need to 'invent' some data etc.   I did split the data into 30 records at a time and was able to import the data.  Will post a reply when I've tried to reproduce the issue.

Arnold

[brightbyte]

I have the same error: all goes fine until the last step, which fails when showing 0% progress with:

We encountered an unknown error in setError: Error: Unable to load https://xx.xxx.xx/civicrm/standalone/index.php?q=civicrm/import/contact status:500

Importing samll amounts of data (25 records) at once works, but even slightly more (50 records) fails. Importing 900 contacts in batches of 25 would be a bit annoying.

I also tried the seconds 25 records, to make sure it's not related to the data contained. It isn't. And it fails consistently. It seems that some limit is hit - but the error occurrs relatively quickly, way before php's max execution time. And 90MB should also be plenty of room. I see no errors in the PHP log nor in CiviCRMs log.

Any ideas?

PS: i can't access CiviCRM on the Drupal demo site. It tells me I'm not authorized. How do I get in?

[Kurund Jalmi]

brightbyte:

PS: i can't access CiviCRM on the Drupal demo site. It tells me I'm not authorized. How do I get in?

Username = demo password : demo , you need to use this credentials.

Hth

Kurund

[petednz]

My experience for reasons I can't explain have been that imports have worked better via Safari than via Firefox in case it is worth trying different browsers in your case.

[businessbroker]

on demo this go without problems, in my bluehost.com it was going and today not, I got this error at 43% import of 110 contacts
on site http://www.rnhotels.com with drupal 6.6 and civicrm 2.1.2
what i can do? i need to import 40.000 records to start..

on www.xopeka.biz importing only pue emails I can import maximum 600-1200 emails per time, to not have time out error, i have done a dozen of imports, I'm not sure, but it look that it slow down relatively with number of TAGS

[brightbyte]

I was told that in 2.1.0 there's some debug code let in line 792, CRM/Import/Parser/Contact.php which might cause trouble, samley, a sleep statement that makes everything slow. It's removed in 2.1.2.

That sounded like a good guess at the cause, and I found the line and removed it.

This did not help however. I'm still getting this error. And I get it after only about 20 seconds or so, so a PHP timeout is extremly unlikely. So, what gives?! 

[sodder]

My attempts to reproduce the issue have also not been successful on the demo site.  I'm running 2.1.2.

[Donald Lobo]

This did not help however. I'm still getting this error. And I get it after only about 20 seconds or so, so a PHP timeout is extremly unlikely. So, what gives?! 

can you please reproduce this issue on the demo server

lobo

[brightbyte]

Like the others, I was unable to reproduce the error on the demo site. The most obvious difference in my settup is that it's a standalone install. Also, PHP is running as a CGI module (it's suPHP). There's probably another dozen differences. So, the question goes to other people with this problem...

* Are you using a standalone install?
* Are you using PHP via suPHP? via CGI at all?

I know it's hard to fix an error if you can't reproduce it. So I'm asking for advise as to how to debug it. First of all, there seems to be some magic going on in order to make the progress indicator work... The error indicates its calling itself via HTTP? Is that right? Can this somehow be bypassed, so the response is more immediate, and we could see more of what's going on?

[sodder]

My previous attempt to reproduce the issue was with 500 records.  Just retried it with 5000 records and still could not reproduce the issue.  I agree that we need to debug this on our systems.  Can someone point me to the source code files for the import function?

[OwenRogers]

FWIW, I encountered this error (setError: Error: bad http response code:500) using Google Chrome at exactly 50 records out of a batch of 656. I found this thread and after a bit of puzzling tried again in Internet Explorer 7 and the run completed successfully with no errors. Using 2.0.1 on Drupal.

[businessbroker]

it's time to upgrade to 2.2 big works are done on import tool, here info how to upgrade:
http://civicrm.org/node/495