CiviCRM Community Forums (archive)

My goal is to migrate mailing lists from a legacy system to CiviCRM. The individual and organization contacts have already been migrated but now I need to create several hundred group lists. I thought I had this figured out but as I attempt it now, I run into many problems...

I am creating a CSV file for import that contains all the contact information including Company Name which I map to the related organization contact by Employed by. I perform an Individual, Fill import and create a new group. Then chaos ensues.

1) my existing contact records are corrupted with duplicate data. Every successive import results in more and more phone numbers and email addresses for the contacts (all the same data).
2) the country is being updated with random countries. I tried do-not-import on the country field, but still, I now have contact with addresses  in Azerbajan and Namibia.
3) the related organizations of the contacts are added to the groups along with the individuals. So, for example I have 12 contacts to import, but then I end up with 24 "members" of the group because the organizations are also added to the groups. I can delete these by searching for organizations within the group, but it would be much easier if they were never imported.

The reason I was doing a full import of all columns of data was that sometimes the data in the legacy system is more up to date and people are adding new data all the time. However, if I simply change the import file to be email, organization, then I still get duplicate email addresses added.

There are many variables at play here; I will continue to troubleshoot by trial and error, but it would be great if someone had written up a how-to for this kind of task. It MUST be a common situation to have to migrate from one system to another.

If there is a way to do this directly in the database, I would be open to that as well. It would be so much easier to just put all the email addresses in a temporary table and then insert rows into a group--avoiding this dangerous import process.

I don't know that I consistently selected a De-dupe rule. Email address should be the primary key. I see now there is a rule for strict individual email...I suppose I should select that?

With respect to an external identifier, maybe. There is a field, but it is not exportable; have to pull directly out of the database.