CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Discussion (deprecated) »
  • Alpha and Beta Release Testing »
  • 3.2 Release Testing »
  • default dedupe rules
Pages: 1 [2]

Author Topic: default dedupe rules  (Read 11656 times)

Dave Greenberg

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 5760
  • Karma: 226
    • My CiviCRM Blog
Re: default dedupe rules
August 07, 2010, 09:52:56 am
No great ideas yet for renaming "STRICT" - but thinking on it (we could use Fussy and Fuzzy as the 2 names - but doesn't quite capture the purpose :-) ).

I've pinged the core team for feedback on this - so keeping the conversation going a bit to collect more perspectives. If one of u is inspired to write a quick blog post with proposed modifications - that would probably get a bit more exposure and input from the broader community.

On a related note, I like Eileen's idea for using New Individual profile as a default for online event registration and online contribution pages. Currently we ship some sample Event Templates which do use New Individual - so if folks use / modify those templates and create events from there - they already get that default. A "hesitation" (especially for online contribution pages) is that pages configured w/ a payment processor where billing fields are collected will have 2 sets of name fields. Thoughts on that?

BTW - Eileen, I hope you noticed all the new links and shortcuts in the Manage Events screens inspired by your suggestions over the past year(s) :-)
Protect your investment in CiviCRM by  becoming a Member!

SarahG (FountainTribe)

  • Ask me questions
  • ****
  • Posts: 782
  • Karma: 29
  • CiviCRM version: 4.4.7
  • CMS version: Drupal 6, Drupal 7
  • MySQL version: 5.5
  • PHP version: 5.3
Re: default dedupe rules
August 07, 2010, 05:59:02 pm
I am with Eileen - I think the default script rule should be email + first name + l last name.  Plus add the "new individual" profile as a default on new events and donation pages.   I may be biased, but I  think the situation of 2 people sharing an email is pretty common.     

Did I help you? Please donate to the Civi-Make-It-Happen campaign  CiviCRM for mobile devices! 

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: default dedupe rules
August 07, 2010, 08:23:00 pm

maybe having 2 rules with "wierd names" is confusing to most users? maybe we should just get rid of the distinction between strict and fuzzy and just use the "default" rule

at some point in time (when someone contributes code and/or sponsor the feature), it would be good to allow profiles/event reg forms/contribution forms etc to specify what dupe check rule to follow.

with regard to shared email addresses, it would be nice if someone could see if there is "some actual data" to see how widespread this issue is. I suspect to a large extent it depends on the demographic and geographic location that an org serves.

the answer to the above question, will determine if your dupe check rule can be only email or email + first + last. the nice thing about only email is, many of the orgs are not aware of (or dont care about) doing any dedupe currently

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: default dedupe rules
August 08, 2010, 12:00:22 am
I see value in having the two types - I think of them as 'front-end' and 'back-end' rules so perhaps something that denotes the difference between the fact that 'strict' are basically used by people on the outsite of the system and fussy  ;) by people on the inside would make more sense.

I agree the shared e-mail issue is demographic related - we mostly hit it in families but I imagine that it is also common in communities where there are limited internet resources (which would probably be a geographic thing). Having said that I am not aware of any organisation that uses events where I would find e-mail only adequate. Events seem to be particularly prone to people signing other people up in many and varied ways and giving their e-mail both for themselves and other people they sign up (not necessarily using the multiple participants feature). I'm not sure I would see huge value in specifying a rule per event / profile / contrib but others might.

Regarding the first name / last name on the billing screen. I would tackle this by putting the first-name/ last-name/email profile as the top of page profile & putting in javascript to autofill the billing based on this profile - either automatically or - like Ubercart does - give the option to select 'my billing details are the same as my details' or similar - this would presumably copy address details in as well

Dave - I think the 3.2 navigation buttons are really useful & well laid out

 



Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

petednz

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4899
  • Karma: 193
    • Fuzion
  • CiviCRM version: 3.x - 4.x
  • CMS version: Drupal 6 and 7
Re: default dedupe rules
August 08, 2010, 02:50:21 am
A really simple tweak that might help some of us who are suffering early onset - would be at least a warning notice on the Import Wizard to 'check your dedupe rules before starting the import' , or even better a checkbox to confirm that 'yes i have checked my settings in DeDupe before I commence my import" (and probably a nice message at the end to suggest that you may need to change them back again!
Unfortunately I doubt I am the only one to forget to check if the strict dedupe default has been reset by someone else, or worse not even checked by me, before firing off an import.
Sign up to StackExchange and get free expert advice: https://civicrm.org/blogs/colemanw/get-exclusive-access-free-expert-help

pete davis : www.fuzion.co.nz : connect + campaign + communicate

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: default dedupe rules
August 09, 2010, 06:00:48 am
Quote from: peterd on August 08, 2010, 02:50:21 am
A really simple tweak that might help some of us who are suffering early onset - would be at least a warning notice on the Import Wizard to 'check your dedupe rules before starting the import' , or even better a checkbox to confirm that 'yes i have checked my settings in DeDupe before I commence my import" (and probably a nice message at the end to suggest that you may need to change them back again!

Asking to approve the whatever without providing infos about the whatever is very much the microsoft way (eg. for the EULA, the security pop up, wizards... you end up with users that are trained monkeys that know that they should click on "yes" -or "next"-, even so they don't have a f**king clue of what the question is, nor what their answer means.

Just went back from Guatemala and Mexico, saw a lot of monkeys in the jungle. Let's leave them there and not in front of the keyboard using civicrm.

Quote from: peterd on August 08, 2010, 02:50:21 am
Unfortunately I doubt I am the only one to forget to check if the strict dedupe default has been reset by someone else, or worse not even checked by me, before firing off an import.

And I'm willing to bet that you won't check more, just that you will train yourself to check whatever box or ignore whatever message.

As for the dedupe rules and duplicate, my experience is completely different than Brian's: virtually each of the contact has a different email, and duplicates are a big issue that takes them a lot of time to fix.

X+

P.S. Trying to gauge if we can get community support. How much work would it be to:
1) add the list of potential dedupe rules (with the default/strict being selected) on the import
2) apply this rule instead of the strict/default one
3) same goes for profile (that should cover all the other use cases, eg. online event registration/donation...

-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: default dedupe rules
August 09, 2010, 06:45:26 am

i suspect tweaking the code to use configurable dedupe rules in various scenarios is a 20-30 hour job. We will need a join table to specify what dedupe rule to use in what scenario (for profiles and other cases). There is the potential that the same profile will be used in multiple registrations (name and address) but each registration would prefer its own dedupe rule (a bit more general IMO)

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: default dedupe rules
August 09, 2010, 07:09:25 am
Quote from: Donald Lobo on August 09, 2010, 06:45:26 am

There is the potential that the same profile will be used in multiple registrations (name and address) but each registration would prefer its own dedupe rule (a bit more general IMO)


That's already an issue today with the boolean on the profiles, what happens if you have two profiles, one asking to match existing contacts, the other that doesn't ?

Wouldn't it be simplier/cheaper (for the profile) to convert the boolean into an int, if null no dedupe applied (like the boolean today) if a value use that rule ?

Same goes for the import,  instead of having an option "update/skip/create dupe" offering the option to choose the dedupe rule.

ie. no new table, altering  civicrm_uf_group to have an int update_dupe foreign civicrm_dedupe_rule null instead of a boolean is_update_dupe. Modifying the ui/wizard on the import to add the choice of rule and no impact on the DB, as import jobs aren't stored anyway (I think)

X+

P.S. Trying to get funding for that isn't going to be easy, as it's an issue that isn't a concern for the users (that aren't aware of dedupe rules in the first place) and given the length of this discussion, not a trivial one to explain, let alone convince it needs their funding to fix.
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

lcdweb

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1620
  • Karma: 116
    • www.lcdservices.biz
  • CiviCRM version: many versions...
  • CMS version: Joomla/Drupal
  • MySQL version: 5.1+
  • PHP version: 5.2+
Re: default dedupe rules
August 09, 2010, 05:45:59 pm
We wouldn't want to remove the update/fill/skip control. Those are important controls for how the dedupe rules should be used.
But I really like the idea of setting a dedupe rule on a per import and per profile basis. I could potentially chip in to help sponsor some hrs.
support CiviCRM through 'make it happen' initiatives!
http://civicrm.org/mih

petednz

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4899
  • Karma: 193
    • Fuzion
  • CiviCRM version: 3.x - 4.x
  • CMS version: Drupal 6 and 7
Re: default dedupe rules
August 09, 2010, 05:56:26 pm
The other tweak that would be advantageous if remotely possible would be as well as the update/fill/skip - only update or fill for records where a match is made, do not import and create new contacts if no match is made.

Maybe an extra checkbox - Do not create new records. Update or fill only on matched contacts.
Sign up to StackExchange and get free expert advice: https://civicrm.org/blogs/colemanw/get-exclusive-access-free-expert-help

pete davis : www.fuzion.co.nz : connect + campaign + communicate

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: default dedupe rules
August 09, 2010, 07:43:43 pm
Quote from: lcdweb on August 09, 2010, 05:45:59 pm
We wouldn't want to remove the update/fill/skip control. Those are important controls for how the dedupe rules should be used.


Thanks for clarifying, didn't mean to replace all the options indeed, but instead of having the simple choice update or fill or skip, being able to set the dedupe rule



X+
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: default dedupe rules
August 14, 2010, 06:26:02 am
As per Dave's request:

http://civicrm.org/blogs/xavier/how-civicrm-avoid-creating-duplicates-and-suggestions-improvements
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Pages: 1 [2]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Discussion (deprecated) »
  • Alpha and Beta Release Testing »
  • 3.2 Release Testing »
  • default dedupe rules

This forum was archived on 2017-11-26.