CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Can't import more than 400 records
Pages: [1]

Author Topic: Can't import more than 400 records  (Read 931 times)

crayon

  • I post occasionally
  • **
  • Posts: 69
  • Karma: 1
  • CiviCRM version: 3.3.6
  • CMS version: Drupal 6.2
  • MySQL version: 5.0.91
  • PHP version: 5.2.17
Can't import more than 400 records
May 02, 2011, 02:24:16 pm
I am trying to import about 10,000 records.  200 at a time goes without a hitch but more than 400 or so crashes.  It imports over 400 and then I get an Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request.  Is there something I can fix?

Hershel

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4640
  • Karma: 176
    • CiviHosting
  • CiviCRM version: Latest
  • CMS version: Mostly WordPress and Drupal
Re: Can't import more than 400 records
May 03, 2011, 02:48:38 am
The message "Internal Server Error" usually means a CPU timeout or perhaps a RAM limit. These can fixed if you are able to adjust those parameters in your php.ini file. Ask your host about this if you don't know what it is.
CiviHosting and CiviOnline -- The CiviCRM hosting experts, since 2007

See here for the official: What to do if you think you've found a bug.

crayon

  • I post occasionally
  • **
  • Posts: 69
  • Karma: 1
  • CiviCRM version: 3.3.6
  • CMS version: Drupal 6.2
  • MySQL version: 5.0.91
  • PHP version: 5.2.17
Re: Can't import more than 400 records
May 03, 2011, 08:16:13 am
I found the file.  I am using PHP 5.2.17.  My status report show a 64MB memory limit. 

Below are the php.ini settings.  Do I need to change something or add something?

register_globals = off
allow_url_fopen = off

expose_php = Off
max_input_time = 60
variables_order = "EGPCS"
extension_dir = ./
upload_tmp_dir = /tmp
precision = 12
SMTP = relay-hosting.secureserver.net
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=,fieldset="

Hershel

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4640
  • Karma: 176
    • CiviHosting
  • CiviCRM version: Latest
  • CMS version: Mostly WordPress and Drupal
Re: Can't import more than 400 records
May 03, 2011, 08:38:03 am
The About CiviCRM page http://civicrm.org/aboutcivicrm saays:

Quote
128mb PHP memory limit recommended

So I would suggest you ask your host if that can be increased from the 64 you have. This is set via:

Code: [Select]
memory_limit = 128MB
but on many/most shared hosts, you don't have the ability to set this via your own "private" php.ini file.
CiviHosting and CiviOnline -- The CiviCRM hosting experts, since 2007

See here for the official: What to do if you think you've found a bug.

CiviTeacher.com

  • I live on this forum
  • *****
  • Posts: 1282
  • Karma: 118
    • CiviTeacher
  • CiviCRM version: 3.4 - 4.5
  • CMS version: Drupal 6&7, Wordpress
  • MySQL version: 5.1 - 5.5
  • PHP version: 5.2 - 5.4
Re: Can't import more than 400 records
May 11, 2011, 02:04:54 pm
CiviCRM has robust hosting requirements.  Hershel knows a lot about this.  Ignore the CiviCRM recommendations for 128MB of PHP ram at your own peril.

In my experience, I rarely attempt to import more than 2000 rows of data at a time, even on sites with 128MB of PHP memory.  Generally with a large import I split it into several different files of < 2000 rows and import in batches.   Save your field mapping and you'll get better results this way.   I've been doing CiviCRM imports for about 3 years now.

Try CiviTeacher: the online video tutorial CiviCRM learning library.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Can't import more than 400 records
May 11, 2011, 04:48:58 pm

seems like in the long run we do need an import api / a scalable import tool. Hopefully we'll get a group of people who collaborate and make this a priority :)

Some relevant blog articles:

http://civicrm.org/blogs/xcf33/scaling-contact-import-process-civicrm-perspective-and-solution
http://civicrm.org/blogs/lobo/implementing-batch-import-api
http://civicrm.org/blogs/eileen/civi-migrate-proof-concept

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Can't import more than 400 records

This forum was archived on 2017-11-26.