CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Suggestions of improvements, what do you think ?
Pages: [1]

Author Topic: Suggestions of improvements, what do you think ?  (Read 1128 times)

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Suggestions of improvements, what do you think ?
February 14, 2011, 04:20:53 am
3 big pain on my side:

For practical purpose UK should be handled like GB instead of rejecting country not exist (long story about northern ireland, bottom line, no one uses GB over here, but UK).


If there is only one column in the css, try the other separator instead of trying to create a table with a super long column name and crashing DB error.

If the column name in the csv is > 32, truncate it to avoid the import to crash on a db error.

add a field "source" in the last step of the import (when you choose the group & tags). so you can add "import list X" and put it as the source of the new contacts.

X+


-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Erik Hommel

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1773
  • Karma: 59
    • EE-atWork
  • CiviCRM version: all sorts
  • CMS version: Drupal
  • MySQL version: Ubuntu's latest LTS version
  • PHP version: Ubuntu's latest LTS version
Re: Suggestions of improvements, what do you think ?
February 15, 2011, 12:18:07 am
All improvements make sense to me!
Consultant/project manager at EEatWork and CiviCooP (http://www.civicoop.org/)

SweetTooth

  • I post occasionally
  • **
  • Posts: 70
  • Karma: 1
  • CiviCRM version: 4.1
  • CMS version: Drupal 6.20
  • MySQL version: 5.1.54
  • PHP version: 5.2.17
Re: Suggestions of improvements, what do you think ?
February 25, 2011, 04:44:22 am

Scalability and speed has to be improved IMHO. My organisation often has to import just shy of 200,000 records in one go, to have to do 200,000 broken down into 10,000 batches is a pain, especially if you forgot where you were or forgot to assign 30,000 of them to a group etc!

Ultimately, something that can split the job between processors would also be good.

If I have a box with 16 cores and I'm importing several million records, I would want at least 8 cores on the job.

Some people might think a million records is a lot, but during a Euro Election, a Euro region has millions and I wouldn't be surprised if around the world, there are cicrumstances where political parties end up having to import tens of millions within the space of a week.

You could argue that we should try and cut down the data before we load it into CiviCRM, but in my world, CiviCRM is supposed to be doing everything so I can centralise my operation.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Suggestions of improvements, what do you think ?
February 25, 2011, 08:13:24 am

I dont think anyone is arguing against improving import vis-a-vis scalability and speed :)

I think the main question is: who will do the work and who will fund it? Maybe the political parties who are importing millions of records in a short timeframe should help make this happen.

if we do get a few seed funders for this, we'll introduce this as a make it happen. I suspect we'll do things in small pieces, so potentially do the project as a few 100 hour make it happens in subsequent releases

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

SweetTooth

  • I post occasionally
  • **
  • Posts: 70
  • Karma: 1
  • CiviCRM version: 4.1
  • CMS version: Drupal 6.20
  • MySQL version: 5.1.54
  • PHP version: 5.2.17
Re: Suggestions of improvements, what do you think ?
February 25, 2011, 09:00:35 am
Lobo,

My organisation does raise a nice sum online.

Sadly, none of it is through a CiviCRM setup. We've been frustrated by various problems (most not civiCRM related) and now we are at the last hurdle.

If we solve this last problem, I believe the amount of money that we raise online would at least double and then I would be able to approach my treasurer with a view to part funding some CiviCRM projects.

This is all that stands in our way :

http://forum.civicrm.org/index.php/topic,16972.0.html

If I can get the gross amounts to appear on our CiviCRM installation...we're away...and to us this means anywhere between $500 - $3000 in extra revenue a day, who knows!

alanms

  • I post occasionally
  • **
  • Posts: 72
  • Karma: 5
Re: Suggestions of improvements, what do you think ?
March 07, 2011, 05:03:56 am
Hey,

I'm starting a discussion on some import changes I'm proposing to a client to fund - it's the opposite end of the scale to SweetTooth's issue, target audience of small organisations doing average sized imports on lower-spec machines without worrying about their servers falling over, so I don't want to off-topic this discussion, but it is relevant to this general topic so I'll drop a link: http://forum.civicrm.org/index.php/topic,18923.0.html

Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using Import (Moderator: Yashodha Chaku) »
  • Suggestions of improvements, what do you think ?

This forum was archived on 2017-11-26.