CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Importing Large Files
Pages: [1]

Author Topic: Importing Large Files  (Read 2589 times)

8ballsaysiwin@gmail.com

  • Guest
Importing Large Files
May 20, 2009, 03:53:39 pm
So I am trying to import a very large database into CiviCRM (350,000 records). I run into a lot of problems using the import page in CiviCRM (many of which have been described in this forum elsewhere). So, my thought was that I would import the files directly into MySQL, but I am now running into errors with this as well (time-limit and file size errors).

Has anyone out there done this before? Can you please describe your methods for getting all that data in?

Or if anyone has suggestions on how I should import hundreds of thousands of records, I would love to hear them.

Chris Burgess

  • Ask me questions
  • ****
  • Posts: 675
  • Karma: 59
Re: Importing Large Files
May 20, 2009, 05:27:53 pm
2.2 has an import process (contacts only I believe) where you can define an SQL SELECT as your import statement.

So you can import your 350K records to MySQL as a single flat table, then use CiviCRM to import from there.

Once it's there, you can choose "SQL Query" as the data source at civicrm/import

To get the data into a single table in MySQL:
  • If you have console access, you can use MySQL's "LOAD DATA INFILE" to load the 350K CSV file rapidly and circumvent PHP timeouts.
  • Or you could use PHPMyAdimn - which should be faster going direct into a single table than CiviCRM importing contact records with many related table writes
  • Or you could ask your hosting admin to load the data into a DB for you to work with
  • Or connect to MySQL via networked ODBC/JDBC if that's available to you.
« Last Edit: May 20, 2009, 05:32:28 pm by xurizaemon »
@xurizaemon ● www.fuzion.co.nz

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Importing Large Files
May 20, 2009, 06:16:20 pm

also check

bin/ContributionProcessor.php (http://svn.civicrm.org/civicrm/branches/v2.2.report/bin/ContributionProcessor.php)

we recently used it to upload 240K contacts and 260K contributions at the same time. Its a custom script etc, but should be easily tailored for your needs. ping us on IRC if you need help or assistance

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

Denver Dave

  • Ask me questions
  • ****
  • Posts: 471
  • Karma: 9
Re: Importing Large Files
July 03, 2009, 05:55:19 pm
I previously ran into a similar issue trying to reload a phpBB forum.  The solution there, which might work here is a utility called BigDump:

http://www.hotscripts.com/listing/bigdump/
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Importing Large Files

This forum was archived on 2017-11-26.