CiviCRM Community Forums (archive)

We have a set of data that regularly needs to be imported into CiviCRM from another application.  We generate csv files to be imported using Kettle.

I find when the sets of import records is too large, I get errors like this one:

#0 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/Error.php(148): CRM_Core_Error::backtrace()
#1 [internal function](): CRM_Core_Error::handle(Object(DB_Error))
#2 /home/crm/public_html/sites/all/modules/civicrm/packages/PEAR.php(931): call_user_func((Array:2), Object(DB_Error))
#3 /home/crm/public_html/sites/all/modules/civicrm/packages/DB.php(969): PEAR_Error->PEAR_Error("DB Error: no such table", -18, 16, (Array:2), "SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#4 /home/crm/public_html/sites/all/modules/civicrm/packages/PEAR.php(564): DB_Error->DB_Error(-18, 16, (Array:2), "SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#5 /home/crm/public_html/sites/all/modules/civicrm/packages/DB/common.php(1905): PEAR->raiseError(NULL, -18, NULL, NULL, "SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...", "DB_Error", TRUE)
#6 /home/crm/public_html/sites/all/modules/civicrm/packages/DB/mysql.php(898): DB_common->raiseError(-18, NULL, NULL, NULL, "1146 ** Table 'civicrm.civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816' d...")
#7 /home/crm/public_html/sites/all/modules/civicrm/packages/DB/mysql.php(327): DB_mysql->mysqlRaiseError()
#8 /home/crm/public_html/sites/all/modules/civicrm/packages/DB/common.php(1216): DB_mysql->simpleQuery("SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#9 /home/crm/public_html/sites/all/modules/civicrm/packages/DB/DataObject.php(2421): DB_common->query("SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#10 /home/crm/public_html/sites/all/modules/civicrm/packages/DB/DataObject.php(1613): DB_DataObject->_query("SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#11 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/DAO.php(155): DB_DataObject->query("SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#12 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/DAO.php(917): CRM_Core_DAO->query("SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...", TRUE)
#13 /home/crm/public_html/sites/all/modules/civicrm/CRM/Import/ImportJob.php(127): CRM_Core_DAO::executeQuery("SELECT * FROM civicrm_import_job_3e6e9f85b1c6340c64b8f4fde887f816\n          ...")
#14 /home/crm/public_html/sites/all/modules/civicrm/CRM/Import/Form/Preview.php(334): CRM_Import_ImportJob->isComplete(TRUE)
#15 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/Form.php(246): CRM_Import_Form_Preview->postProcess()
#16 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/StateMachine.php(160): CRM_Core_Form->mainProcess()
#17 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/QuickForm/Action/Next.php(60): CRM_Core_StateMachine->perform(Object(CRM_Import_Form_Preview), "next", "Next")
#18 /home/crm/public_html/sites/all/modules/civicrm/packages/HTML/QuickForm/Controller.php(203): CRM_Core_QuickForm_Action_Next->perform(Object(CRM_Import_Form_Preview), "next")
#19 /home/crm/public_html/sites/all/modules/civicrm/packages/HTML/QuickForm/Page.php(103): HTML_QuickForm_Controller->handle(Object(CRM_Import_Form_Preview), "next")
#20 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/Controller.php(316): HTML_QuickForm_Page->handle("next")
#21 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php(292): CRM_Core_Controller->run((Array:3), NULL)
#22 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php(70): CRM_Core_Invoke::runItem((Array:13))
#23 /home/crm/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php(52): CRM_Core_Invoke::_invoke((Array:3))
#24 /home/crm/public_html/sites/all/modules/civicrm/drupal/civicrm.module(436): CRM_Core_Invoke::invoke((Array:3))
#25 [internal function](): civicrm_invoke("import", "contact")
#26 /home/crm/public_html/includes/menu.inc(517): call_user_func_array("civicrm_invoke", (Array:2))
#27 /home/crm/public_html/index.php(21): menu_execute_active_handler()
#28 {main}

What must I do to make these go away?  This was importing a set of 1000 individual contacts.  Importing 500 does not cause the error.  I haven't narrowed it down lower than that.

If it were a one time thing or if I were the one doing it, 15 batches of 500 records each would be OK.  But we want to turn this over to someone else to run whenever needed, and it shouldn't be something that needs to be baby-sat and have files fed in every 3 or 4 minutes.  We want to get this tuned so that each import (there are 3) can be done with one file each, and let the person do some other task while it's running.

I've made a variety of changes and the hard crash with backtrace has disappeared.  I am unsure which of the changes was effective but happy to figure that out later.  Changes included increasing apache timeout and max connections settings.  Although I don't have all the imports in a single file, the batch sizes have doubled and that means progress.

But now I am getting some strange results.  I am unsure if something is really wrong, or if the user interface just isn't reporting correctly.

The javascript progress dialog is the issue.  For example, on a batch of 800 things will progress normally until it gets past 600, then, in sequence,

600 of 800

no processing status report yet

50 of 800

700 of 800

100 of 800

750 of 800

150 of 800

and then it completes.  The final total of records imported reflect the number of records remaining when the craziness above starts.  For the above, it would be something like

150 records imported

But there are no real errors.  Most of these records are "updates" with no changes, so it's hard to dig into the data and conclude that the records at the end of the batch have in fact been processed.

Any suggestions?