Author Topic: Import not giving me error reports but not all rows are turning in to records (Read 2685 times)

petednz · March 09, 2008, 09:46:38 pm

Today I have been trying to import a list where i have <first name><last name><email><custom data>
Just when I thought I was getting the hang of this i have gotten my self really confused and am wondering if i have hit a bug in the midst of my confusion.

When I import the list (using Fill or Update - I still don't quite get the difference

it tells me that 153 rows imported succesfully - when i then do an AdvSearch on that custom data I only get around 60. And I am not getting an error report even though there are some rows that have no surname - and there are some where first and last name will match but it has a different email. What setting do i need to use so that
- if <first name><last name><email> all match then it puts the data in to my custom data field and
- if any of the fields don't match it reports them so i can sort anomolies by hand instead of it creating a new contact with same names and new emails.
FYI our "Contact Matching' Rules are first_name AND last_name AND email

petednz · March 09, 2008, 11:10:12 pm

Spending more time trying to sort this out. The data we are adding is a 'subscription' to a publication. There are a list of newsletters in a custom data group - each one has a multichoice <email><no thanks><print><bulk>. Some of the newsletters had a default set. I realise that whenever a single piece of data eg email, is added to anyone of the newsletters, the default is tripped on all the others!! Is that expected?
And could it relate to why some of my import isn't working, because it is finding data already in that field (triggered by putting data in one of the other newsletter fields) and therefore my data won't update this field

Searching for some thing to explain the current random nature by which some contacts are being updated in this import and others arent.

Dave Greenberg · March 10, 2008, 01:24:50 pm

Quote from: peterd on March 09, 2008, 09:46:38 pm

(using Fill or Update - I still don't quite get the difference

"Update" means over-write existing data in existing contact record fields IF the import row has data mapped to that field. So if contact 3 has eye-color = blue, and a matching import data row has eye-color = green - update the DB value.

"Fill" means leave existing DB field values alone on existing contact record fields. So in the case above, eye-color would still be blue after the import. BUT if an existing record had no value for eye-color - it would be filled with an incoming value.

Quote from: peterd on March 09, 2008, 09:46:38 pm

What setting do i need to use so that
- if <first name><last name><email> all match then it puts the data in to my custom data field and
- if any of the fields don't match it reports them so i can sort anomolies by hand instead of it creating a new contact with same names and new emails.
FYI our "Contact Matching' Rules are first_name AND last_name AND email

There is currently no setting on import to force it to UPDATE only mode - which is what you're trying to do (i.e. you expect all rows to match existing DB records and want to know when any of them don't). I'm not sure why the data source you're importing from has such different values for key fields (first, last, email) from your CiviCRM DB? Is there an option to export to existing CiviCRM values and then merge the newsletter / other values you want to update into that file?

Dave Greenberg · March 10, 2008, 01:26:22 pm

Quote from: peterd on March 09, 2008, 11:10:12 pm

Spending more time trying to sort this out. The data we are adding is a 'subscription' to a publication. There are a list of newsletters in a custom data group - each one has a multichoice <email><no thanks><print><bulk>. Some of the newsletters had a default set. I realise that whenever a single piece of data eg email, is added to anyone of the newsletters, the default is tripped on all the others!! Is that expected?
And could it relate to why some of my import isn't working, because it is finding data already in that field (triggered by putting data in one of the other newsletter fields) and therefore my data won't update this field
Searching for some thing to explain the current random nature by which some contacts are being updated in this import and others arent.

If updating a value in custom field 1 within a custom data group causes a value to be set / updated for a different field - that sounds like a bug to me. Would you be able to create a simple example of this on the demo and recreate / post a bug report?

petednz · March 10, 2008, 01:39:15 pm

Hi Dave - thanks for the further clarification on Update v Fill - I will try and write it down in a way that makes it really obvious to me, and post it back in case it might help others.
RE demo site. I was hoping you wouldn't ask me that but knew you would have to. I do think there was a problem with the import beyond my incompetence. I will have a go at doing a test upload again on our site and if it also fails then will let you know. The thing was that my test uploads (less then 10 members) all worked fine, it was only when i went for the longer list that it seemed to me that it just missed/skipped some and failed to report.
I have been through my list v what is now in civi and can see no pattern.
Some existing members with same name and email were updated, others weren't (may have been my trying Fill combined with fact that the default on that field had been triggered by data for another field in the same group - which I think is a bug but may be intended)
Some non-existent contacts were added, as hoped, others weren't.
The ball is in my court. $:-\$
I think the UPDATE only (ie don't add records if they don't exist) could be a useful addition and isn't that what Speleo was suggesting elsewhere?

petednz · March 10, 2008, 01:43:01 pm

Quote

not sure why the data source you're importing from has such different values for key fields (first, last, email)

some examples
our Civi data has X as Michael or Cate, the file i was importing from was Mike or Kate
our cividata had X as @gmail.com, the file i was importing from was @yahoo.com
Both of these examples create unwanted new records which then have to be removed, with the correct data manually updated in the correct existing record.
Much better would be for me to have the option of excluding such actions, have them reported as an error, and either enable me to check if there is spelling differences and change my file and reimport, or just go for the manual update to the correct contact, without the hassle of having to delete as well.

petednz · March 10, 2008, 01:45:42 pm

Nurrgh - just realised you are asking me to recreate the field issue - will put it on my list.

petednz · March 10, 2008, 02:00:52 pm

Okay - on demo I have set up Tab = petetest. There are 3 x Custom Fields here. One has Default = Orange. Others don't have default set.

If you pick on someone who hasn't had this tab used, or set up a new person, you will see that the page shows nicely blank for all 3 fields.

Now presume you want to change status of PeteTest2 to Apple, go Edit - PeteTestA will now show Orange as the default is triggered by the Edit button, change PeteTest2 to Apple and save, the result is that PeteTestA is saved as Orange (i didn't want to do that!) and TestB is Apple (yay)

Obviously (?) I should not set a default if I don't mean it to work, but it bugrd up my data when I had a whole lot of fields in this sort of set up, uploaded data for one of them, and without realising, caused the default to be applied to all the other fields I wasn't wanting to have change.

Hope that's clear enough. i doubt this is a bug - and may be intentional or unavoidable - but functions as a nuisance in our case. I have now gone and removed all those defaults - but have a bit of a clean up job to do. FOrtunately most of those fields haven't been populated yet so should be able to negate all the default settings that were applied to the contacts.

Donald Lobo · March 10, 2008, 02:20:29 pm

Its supposed to work this way

our 'default' definition is: if the person does not have a value than use this value. Since we use mysql column defaults to implement this, i suspect that whenever a new row is inserted, all the defaults are set for that row (if they are not overrriden)

I dont think i'd consider this a bug

lobo

Andrew Perry · April 15, 2008, 04:19:18 pm

I am having a similar problem here with CiviCRM 1.9.13019

I have exported a bunch of records to add additional Home and Postal addresses and some new custom data fields.

We have also added some new contacts (they are all individuals), so the import csv has a column with the contact id that is sometimes blank.

I have done an import with UPDATE set, and adding all imported people to a new group. It says that all 888 contacts are imported successfully, but when I search on the Group I added them to there are only 825!!!

This is bizarre! I had a similar problem importing a csv file from excel where due to foreign characters it would fail part way through even though it didn't report that (it would just say it had imported 276 people when there were a couple of thousand in the import).

Any assistance greatly appreciated!

Dave Greenberg · April 15, 2008, 05:00:28 pm

This sounds like it might be a bug relative to how the "Add to Group" is handled for updated vs. created (inserted) contact records. It would be helpful if you can run a small subset of your import on the current public demo (with desensitized contact info - of course). Include one or two records that are updates, and one or two inserts.

If this recreates, please post a bug report to the issue tracker - including your sample csv file and steps for recreating. A screenshot of the import mapping you used should be attached to the issue. thx!

CiviCRM Community Forums (archive)

News:

Author Topic: Import not giving me error reports but not all rows are turning in to records (Read 2685 times)

petednz

Import not giving me error reports but not all rows are turning in to records

petednz

Re: Import not giving me error reports but not all rows are turning in to record

Dave Greenberg

Re: Import not giving me error reports but not all rows are turning in to record

Dave Greenberg

Re: Import not giving me error reports but not all rows are turning in to record

petednz

Re: Import not giving me error reports but not all rows are turning in to record

petednz

Re: Import not giving me error reports but not all rows are turning in to record

petednz

Re: Import not giving me error reports but not all rows are turning in to record

petednz

Re: Import not giving me error reports but not all rows are turning in to record

Donald Lobo

Re: Import not giving me error reports but not all rows are turning in to record

Andrew Perry

Re: Import not giving me error reports but not all rows are turning in to record

Dave Greenberg

Re: Import not giving me error reports but not all rows are turning in to record