CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • 108MB Tab Delimited text file - 300,000+ entries 60+data fields
Pages: [1]

Author Topic: 108MB Tab Delimited text file - 300,000+ entries 60+data fields  (Read 2338 times)

thesailor

  • Guest
108MB Tab Delimited text file - 300,000+ entries 60+data fields
January 30, 2009, 05:54:48 pm
I have very large data file - about 108MB that is full of our entire county voter data - I would like to import all contacts to CiviCRM and am having problems figuring out how to do so. Any idea?

I know file upload limits at 2MB are prohibitive... is there a way to run a script on our server to insert all fields? Data mapping is also a problem for that idea... I need some ideas... I have seen some software on googe search that is paid that will take tab delimited files, convert them to csv then slit them down to specific file sizes - but this seems like it would make 50+ files for me to go in manually upload and then import - are there any tools for this you all know of?

Kurund Jalmi

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4169
  • Karma: 128
    • CiviCRM
  • CiviCRM version: 4.x, future
  • CMS version: Drupal 7, Joomla 3.x
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: 108MB Tab Delimited text file - 300,000+ entries 60+data fields
January 30, 2009, 09:53:04 pm
Is it not possible to open file in spreadsheet and save as CSV ? Well do import such a huge file you will have to manually change fie upload limit, also increase execution time and memory limit for PHP. Then try importing... might work

I would recommend you to split file into smaller files and then import those..

HTh

Kurund
Found this reply helpful? Support CiviCRM

Dave Greenberg

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 5760
  • Karma: 226
    • My CiviCRM Blog
Re: 108MB Tab Delimited text file - 300,000+ entries 60+data fields
January 31, 2009, 08:14:06 am
You might want to experiment with new new SQL-based import method available in 2.2 (currently in alpha). With this technique you can potentially load all your records into a temporary table in your DB - and then provide the query to read these into the import. You could use a sequence of LIMIT statements to grab xxxx records at a time....

http://civicrm.org/node/493
Protect your investment in CiviCRM by  becoming a Member!

thesailor

  • Guest
Re: 108MB Tab Delimited text file - 300,000+ entries 60+data fields
January 31, 2009, 06:21:33 pm
I'd sure like to pay someone that has a clue to help me with this... I am still very new to civicrm - trying to show some of the things it can do to my city council and library board - we are doing  a lot of fund raising and I think this system is the best... but I do not have a lot of time to figure out the data import...

I looked into several options to split file into smalled chunks... none of the software I found has documentation and none have worked so far.

I am looking into mySQL gui tool by sun that will allow import of large files like this... create a new DB - then pull jsut the info I am looking for to import for now... but I need help figurig it all out - creating all the data fields we want to track and be able to sort by...

Bounty: $50 - $100 to show me how and help me out! Help with intigration of civicrm with joomla and users... I am trying to see if there is a way to create a custom module to integrate with the two so that a user can log in, create a profile with joomla or CB Joomla and check the large database of citizens in the area for his/her information to verify - then claim and make part of their profile - autofill or associate... I think a lot of people would use something like that - I'll pitch in $500 if someone can get it started...

anyhow - just getting the info into our system is a challenge for me...
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • 108MB Tab Delimited text file - 300,000+ entries 60+data fields

This forum was archived on 2017-11-26.