CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Scalability (Moderator: Donald Lobo) »
  • Contact Importing scalability, and scalability of the system in general
Pages: 1 2 [3]

Author Topic: Contact Importing scalability, and scalability of the system in general  (Read 15164 times)

fen

  • I post frequently
  • ***
  • Posts: 216
  • Karma: 13
    • CivicActions
  • CiviCRM version: 3.3-4.3
  • CMS version: Drupal 6/7
  • MySQL version: 5.1/5.5
  • PHP version: 5.3/5.4
Re: Contact Importing scalability, and scalability of the system in general
September 23, 2011, 07:46:47 am
Quote from: davej on July 27, 2011, 08:45:12 am
"One thing they could try would be to program a 'progress bar' of sorts into their application so that the connection at least has some data going through it, forcing it to stay open."

Upvote.  A progress bar or - simpler - even just a counter updated every 100 contacts imported would prevent a lot of import timeout issues.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Contact Importing scalability, and scalability of the system in general
September 23, 2011, 08:39:25 am

import is done via ajax, and it does have a progress bar.

However there is a long runnung import process in the background (via ajax) that is consuming the resources. the status bar just updates where the import is periodically

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

The Next Wave

  • I post occasionally
  • **
  • Posts: 111
  • Karma: -9
  • I'm just here to change the world.
  • CiviCRM version: 4.4.4
  • CMS version: Wordpress 3.8.1
  • MySQL version: 5.1.72-cll
  • PHP version: 5.4.25
Re: Contact Importing scalability, and scalability of the system in general
June 07, 2013, 04:52:29 pm
Not that this is totally on topic- but I found a tool that splits .csv files quickly via this post:
http://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/
the first link to the splitter is 404- but this one worked:http://www.fxfisherman.com/forums/forex-metatrader/tools-utilities/75-csv-splitter-divide-large-csv-files.html#post727

The Next Wave

  • I post occasionally
  • **
  • Posts: 111
  • Karma: -9
  • I'm just here to change the world.
  • CiviCRM version: 4.4.4
  • CMS version: Wordpress 3.8.1
  • MySQL version: 5.1.72-cll
  • PHP version: 5.4.25
Re: Contact Importing scalability, and scalability of the system in general
February 24, 2014, 09:36:16 am
Another helpful tool- is Google Refine- or Open Refine: http://openrefine.org/
works for dealing with big data- and prepping it for import. It's fractal function- takes the long list of suffixs- and allows you to fix things like "111" instead of "III" and even pull out Dr and PhD from the Suffix and move it to title if need be.

The drupal module solution doesn't help those of us with Wordpress installs.
Pages: 1 2 [3]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Scalability (Moderator: Donald Lobo) »
  • Contact Importing scalability, and scalability of the system in general

This forum was archived on 2017-11-26.