CiviCRM Community Forums (archive)

I am hoping someone with some experience importing largish amounts of data into civicrm can speak up.

I am having an endless streak of frustrations trying to import some pretty trivial data: duplicates getting made that aren't in the csv files, rows not beign imported because of phantom (read: nonexistent) existing contacts), the wrong relationships getting pulled from the data.

So, at this point I don't really care if the importing is not feature complete.  Am I going to save myself a headache in the end by first exporting my data to a custom db of my creation, then importing to civicrm from the db?

Am I better off just importing straight from my csv files into the the DB directly, skipping the civicrm import GUI?

Basically what I have is this:

employees.csv (~20k rows)
--
4 Columns:  First, Last, Employer, Position

I need to tag and add all of these employees as a specific group (and all the same tag).  When I try and import them mapping First->First, Last->last, Employer-> Employee of - Organization name, Position -> Job :: Employee Information

What happens is a bunch of Employees end up being employees of multiple organizations that they are not actually employees of according to the file.  Many of them won't import because they are supposedly duplciated (even though I have deleted every individual before starting the import).

After that file, I have 4 increasingly smaller files that fill in more data about the employees or employers, and then some 'activities' between members of the organization for which the civicrm install is for and the employees.

Basically my question comes down to, is the import known to be horrible and I am just it's latest victim, or am I doing something wrong?  And, if the import *is* horrible, how do I best get around it?

Thanks for your time
Geoff

thanks Donald.

Not sure if you came back to IRC but I had to jet.  In the meantime I looked more into the duplicate issue and I believe I found the main issue with the duplicates.

I am sorry also if I came across a bit peavish.

In any case, another problem I am having refers to tagging and grouping on import.  It's not really critical as I can get around it coincidentally because of the way the data is set up, but if I ever need to do an import on top of existing data this could turn into an issue.

Say I had some data like:

Jim, Johnson, Compaq, Programmer
Salley, Johnson, Gateway, Secretary
George, Wallburg, Compaq, Janitor
Cynthia, Lopez, Gateway, CEO

Now I want to import this csv and add all imported contacts to a group, "Employees of Crappy Computer Makers" It seems to also add Gateway and Compaq to that group. 

Is there a way around this, or am I doing something wrong, or both?

we add all the contacts including the relationship contacts to the group. no way to get around it from a UI perspective

in your case, you can get around it by creating another group post import and storing just the Individual contacts from the above group (since the relationship contact created is an organization)

lobo