CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Core CiviCRM Functions (Moderator: Yashodha Chaku) »
  • Only imports a few hundred at a time
Pages: [1]

Author Topic: Only imports a few hundred at a time  (Read 1963 times)

TLange

  • I’m new here
  • *
  • Posts: 20
  • Karma: 0
  • President of California Clean Money Campaign
    • California Clean Money Campaign
Only imports a few hundred at a time
May 10, 2007, 02:21:12 pm
We're trying to import a database that has about 23,000 records into CiviCRM 1.7 running in Drupal 5.1.  Even when I break the import down into chunks of 1,000 rows (only 89k because we're only importing 11 columns), it's importing less than 400 rows at a time before quickly claiming to finish all 1,000 rows.  If I then re-import the same file telling it to skip matching, it will then successfully import the next 400 or so.  The amount it successfully imports each time gets smaller as the database gets bigger.  At the beginning it was importing 1,000+ successfully but the number imported each time gets a little smaller, and is under 400 now that we have about 11,000 records in.

An ideas on what we could do to make it successfully import more at a time?

Thanks,

- Trent

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Only imports a few hundred at a time
May 10, 2007, 02:52:22 pm

trent:

a few suggestions and thoughts:

1. Im hoping that u r not attempting the below on a shared hosting solution and have a VPS or decent hardware

2. Make sure you have your php timeout set to a high number (1 hour at least). This will enable you to import more stuff

3. Would it be possible for us to get a copy of your db and field mapping. We obviously will keep it confidential etc and destroy when we are done. Would be great and allow us to optimize import and speed it up for 1.8/2.0

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

TLange

  • I’m new here
  • *
  • Posts: 20
  • Karma: 0
  • President of California Clean Money Campaign
    • California Clean Money Campaign
Re: Only imports a few hundred at a time
May 11, 2007, 12:35:08 am
Thanks for the tip.  Increasing the PHP timeout from 10 minutes to 60 made a huge difference.

I'd be happy to provide a version of our db and mapping (after changing the names to protect the innocent!) if it will help to optimize import.  Just let me know how to get them to you.

- Trent

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Only imports a few hundred at a time
May 11, 2007, 01:07:28 am

can u obfuscate and email the file to me at ( lobo at yahoo dot com )

thanx

lobo


A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

syoumans

  • Guest
Re: Only imports a few hundred at a time
May 29, 2007, 05:49:04 am
It could also be a memory issue. The default memory limit for PHP is 8M. Try increasing it to 24M.
- Scott

global_asia

  • Guest
Re: Only imports a few hundred at a time
July 12, 2007, 01:26:44 pm
I think I am having a similar issue - this may not be an issue of this forum - but can somone tell me how to change the php limit - otherwise I imagine I must just import less contacts at once

Dave Greenberg

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 5760
  • Karma: 226
    • My CiviCRM Blog
Re: Only imports a few hundred at a time
July 12, 2007, 05:20:14 pm
The FAQs have some info / links on this:
http://wiki.civicrm.org/confluence/display/CRMDOC/Error+PHP+Memory
Protect your investment in CiviCRM by  becoming a Member!
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Core CiviCRM Functions (Moderator: Yashodha Chaku) »
  • Only imports a few hundred at a time

This forum was archived on 2017-11-26.