CiviCRM Community Forums (archive)

Using current CiviCRM 1.8 on Joomla 1.0.13 (stable)

I've been working on importing my data into CiviCRM. All was going well for a while, but I've run into some recent problems.

I first imported all the organization records into CiviCRM, being careful not to have dups. Great, no prob. Then I imported my individuals, and used the relationship option to say the individual was an employee of the organization name field. Based on the documentation, I think this should have imported the records in such a way so as create the parent-child relationships. Unfortunately, it didn't---at least not consistently. In a few cases it seemed to have created that, but for the most part it just created duplicate org records.

Now I have a database full of duplicate orgs. I hoped the deduping mechanism in CiviCRM would let me hunt down the dups and combine them or something, but when I run the dedupe tool from the admin menu, the page breaks on me:

500 Server Error
A misconfiguration on the server caused a hiccup. Check the server logs, fix the problem, then try again.
URL: http://www.....org/administrator/index2.php?option=com_civicrm&task=civicrm/admin/dedupefind&reset=1&action=update&rgid=2
I've tried deduping both indiv and org contacts, and I get the same error msg.

Any suggestions on what is the best way to backtrack and fix things?

-Brian

Scott,
I figured out a couple problems in my process that cleared things up --

1) When creating your .csv file for import, be sure to use UTF-8 as the export standard. I was exporting from MS Access to .csv, and it defaults to a Windows code format that can create problems when importing into CiviCRM.

2) I broke my import into chunks of about 200-300 (originally 2,000). That's a bit of a pain, but is worth it. I ran into no problems, saw the full list for each file successfully import, and when I did the individuals (after the orgs), the relationships were built perfectly.

I still have issues with the dedupe functions (they crash on me). But I handled most of my deduping before I imported, so it's not an immediate need.

-Brian

1) When creating your .csv file for import, be sure to use UTF-8 as the export standard. I was exporting from MS Access to .csv, and it defaults to a Windows code format that can create problems when importing into CiviCRM.

You can setup CiviCRM to use a non-UTF encoding on imports: Administer CiviCRM → Global Settings → Localization → Legacy Encoding, but it’s better to have a UTF-8 file in the first place.

I still have issues with the dedupe functions (they crash on me). But I handled most of my deduping before I imported, so it's not an immediate need.

Is there a chance you could send me (shot@civicrm.org) your database? I won’t disclose it to anyone, of course, and will delete it as soon as I’m done with the forensics. You can encrypt the email with my GPG key, D128F14A.

Yes, it's running utf8_unicode_ci

And sure, I'll send you a copy of the db dump to take a look at a little later today.
-Brian

Before importing either orgs and indivs, I was rigorous about making sure the names matched perfectly (obviously necessary since org name is essentially the key index field). As for quantity, I imported in chunks of 200. I'm no expert on the code for this function -- just sharing what worked for me.
-Brian