CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Import Issues
Pages: [1]

Author Topic: Import Issues  (Read 668 times)

DerekL

  • I post frequently
  • ***
  • Posts: 132
  • Karma: 1
  • CiviCRM version: 4.5.5
  • CMS version: Drupal 7.34
  • MySQL version: 5.1.54
  • PHP version: 5.2.17
Import Issues
August 14, 2011, 01:33:26 pm
Hi Guys and Gals,

Long time first time. I have a large scale import on my hands, something like 3000 organizations with 6000 members split amongst them.

I'm beginning to notice that the web based import tool is perhaps not the right tool for jobs of this size. I was initially working off of a hosted staging server, and have since moved to a local VM running ubuntu 11, the most recent LAMP available, Drupal 7.7 and Civi 4.05.

I was wondering if anyone out there could point me in the direction of any known resources available that provide some insight as to the bulk importing of orgs, contacts, and relationships. 

For the time being, I'm splitting up my CSV's into files about 1.5 megs a piece and spoon feeding the web import tool, but since I'm working on a tool that needs to more or less synchronize civi with an external sql DB nightly, I am encountering some frustration in the slow nature of the web based import tool.

I am surprised that even my local ubuntu vm chokes on larger CSVs...perhaps I am missing some kind of config tweak that will let apache/drupal/civi use as many resources as required to properly digest sizeable csvs?

Thanks for reading! 

Hershel

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4640
  • Karma: 176
    • CiviHosting
  • CiviCRM version: Latest
  • CMS version: Mostly WordPress and Drupal
Re: Import Issues
August 16, 2011, 07:08:47 am
What do you mean that it chokes? What precisely is the error you are getting?
CiviHosting and CiviOnline -- The CiviCRM hosting experts, since 2007

See here for the official: What to do if you think you've found a bug.

bpmccain

  • I post frequently
  • ***
  • Posts: 255
  • Karma: 5
  • CiviCRM version: 4.1
  • CMS version: Drupal 7.12
  • MySQL version: 5.2
  • PHP version: 5.2
Re: Import Issues
August 16, 2011, 08:18:43 am
I just finished a 550,000 contact import using the API. Took 24 hours, but other than that no problem.

I just posted a similar response here:
http://forum.civicrm.org/index.php/topic,21184.0.html

Check out some of those resources, and I'd be happy to provide you an example php script I used to import. It may not be the most efficient, but it worked no problem.

DerekL

  • I post frequently
  • ***
  • Posts: 132
  • Karma: 1
  • CiviCRM version: 4.5.5
  • CMS version: Drupal 7.34
  • MySQL version: 5.1.54
  • PHP version: 5.2.17
Re: Import Issues
August 17, 2011, 11:06:07 am
Ah, thank you for the pointers!

I've been working off a local VM with ridiculously long timeouts and I've been able to successfully import contacts using SQL queries for the past few days.
 
I would much rather be able to interact with my production site via the API rather than depend on all out mySQL restores.

I'll look into these and report back, I figure the more exposure this side of things gets on the forums, and therefore google, the better for everyone involved.

Edit: Hershel, my production site's host: Civihosting.com seems to be incapable of changing settings on their side to allow for a timeout greater than 100 or so seconds. Whenever I try to import (CSV or SQL), I get 500 server errors approximately half way through the final import progress bar.
I have been working with them since 8/12 without any luck.
 
« Last Edit: August 17, 2011, 11:21:09 am by DerekL »

Hershel

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4640
  • Karma: 176
    • CiviHosting
  • CiviCRM version: Latest
  • CMS version: Mostly WordPress and Drupal
Re: Import Issues
August 17, 2011, 11:56:04 am
Hi Derek. CiviHosting provides shared hosting and our priority is to maintain server stability for all users at all times. We thus do enforce CPU usage limits and we are unable to change those, due to the nature of shared hosting.

The solution to doing large imports is simply to break up the import into smaller chunks.

Generally speaking, the CiviCRM import feature is not intended for daily synchronization. For that, the API would certainly be a better option I would think.
CiviHosting and CiviOnline -- The CiviCRM hosting experts, since 2007

See here for the official: What to do if you think you've found a bug.
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Import Issues

This forum was archived on 2017-11-26.