Multi-Org: Summary of data segregation failures

[davej]

Hi,

As we've gone further in using multi-org, further data segregation failures have come to light. Here's a summary from Oliver Gibson:

----8<----
·        Events - fully shared

·        CiviEvent public listings are shared

·        Public Subscription pages for contact groups (i.e. Newsletters) are shared

·        Mailing Headers and Footer templates are shared

·        Message templates and messages are shared

·        Reports are shared (I think there's a new ACL fucntion in 3.1 that can sort this)

·        I think Custom searches are shared

·        Duplicate checking is shared

·        Membership types are shared (not a priority)

·        Activity Types are shared (not a priority)
----8<----

Duplicate checking affects imports, too.

A user on an an L2 site with Administer CiviCRM and upload contacts, without administer Multiple Organizations, view all contacts or edit all contacts, finds that during import duplicates are checked across all domains.

Expected behaviour: unless the user has view/edit all contacts and/or administer Multiple Organizations perms (we're still a bit hazy on how the "all contacts" perms interact with "administer Multiple Organizations"), then duplicate checking (including within import) should be carried out entirely within the site's parent group.

Without Administer CiviCRM perm, the same user finds that the import fails on every contact, with the same error on every line in Import_Errors csv:

"Invalid value for field(s) : field ID; field ID; field ID; field ID; field ID; field ID; "

Administer CiviCRM perm shortcuts custom group ACL: see CRM_Core_Permission::customGroup line 127:

Code: [Select]

        // check if user has all powerful permission
        // or administer civicrm permission (CRM-1905)
        if ( self::check( 'access all custom data' ) ||
             self::check( 'administer CiviCRM' ) ) {
            return array_keys( $customGroups );
        }

- This is not good; L2 admins need 'administer CiviCRM' but should not see custom groups that are ACL'd for other sites only.

One last heave for multi-org...

Dave J

[Dave Greenberg]

Dave - you should hookup w/ deepak directly on these issues (if you haven't already).

[Donald Lobo]

lets have an IRC chat on this

some of this is potentially scope creep too how about monday morning PST (6:00 am?) would be good for oliver and dave m to be present too. i can do later, but not much earlier

lobo

[petednz]

Would be interested in being part of any discussion or at least knowing the outcomes - not suggesting you change times to suit us though. But as we are in active development mode with multisite  another 'use case' might be useful. And am certainly interested in any timeframes involved and of course in any possibilities of helping to push/resolve/contribute some of the items that are higher on our clients wishlist.

I haven't had much time to play with our set up yet so am surprised to see Events sitting at the top of the list as I thought this had been ticked off but maybe I got the wrong end of that stick - it can happen  .

Maybe once I have had a good look around willsbrownberger set up and the powerbase latest iteration I can spend some time back in multi-land.

[davej]

lets have an IRC chat on this

some of this is potentially scope creep too how about monday morning PST (6:00 am?) would be good for oliver and dave m to be present too. i can do later, but not much earlier

Sorry, I didn't get a notification of this post & didn't see it. Would today be possible? Awaiting reply from Olly to see if he's free too.

Dave J

[davej]

Discussed with Lobo on IRC, recording the conversation here for posterity...

Feb 17 16:18:42 <davej_>   dlobo: davem_ has had to go to a meeting and olly can't be here but would be useful to look at ACL issues if you're free.
Feb 17 16:18:51 <dlobo>   yes
Feb 17 16:18:54 <dlobo>   we can
Feb 17 16:18:59 <dlobo>   just wrapping up a few thngs
Feb 17 16:19:01 <dlobo>   but lets start
Feb 17 16:19:31 <davej_>   I've had a go at multisite_civicrm_aclGroup for events...
Feb 17 16:20:03 <davej_>   .. seems to work in that it's called and sets correct $currentGroups
Feb 17 16:20:39 <davej_>   - but has no effect, e.g. civicrm/event?reset=1 still shows all events.
Feb 17 16:21:02 <davej_>   - likewise civicrm/event/ical?reset=1&page=1
Feb 17 16:21:19 <dlobo>   davej_: this is in 3.1.2?
Feb 17 16:21:29 <dlobo>   can u post on forum, and we'll check and potentially fix in 3.1.3
Feb 17 16:21:39 <davej_>   Yes, 3.1.2
Feb 17 16:21:40 <dlobo>   seems like  a bug if that is happening
Feb 17 16:21:59 <dlobo>   also that code goes in the multisite module eventually right
Feb 17 16:22:30 <davej_>   dlobo: yes, I put it in multisite.module
Feb 17 16:22:46 <dlobo>   cool, you might also want to attach the code there
Feb 17 16:22:54 <dlobo>   did u commit it?
Feb 17 16:22:57 <dlobo>   into svn?
Feb 17 16:23:04 <dlobo>   not sure if u have svn access
Feb 17 16:23:23 <davej_>   dlobo: no, I meant our local copy.
Feb 17 16:23:37 <davej_>   Do I need to do multisite_civicrm_aclWhereClause too?
Feb 17 16:24:03 <dlobo>   davej_: no
Feb 17 16:24:13 <dlobo>   the aclwhereclause is used only for contact cls
Feb 17 16:24:15 <dlobo>   acls
Feb 17 16:24:35 <davej_>   dlobo: OK, I'll post on the existing leakage thread.
Feb 17 16:24:49 <dlobo>   davej_: create a new thread in civievent
Feb 17 16:24:56 <davej_>   OK
Feb 17 16:24:57 <dlobo>   lets split it into smaller thread
Feb 17 16:25:06 <dlobo>   do u have the forum link for the lekage thread
Feb 17 16:25:06 <davej_>   Not in multi-org forum?
Feb 17 16:25:16 <dlobo>   multi-org forum is better
Feb 17 16:26:13 <davej_>   dlobo: leakage thread: http://forum.civicrm.org/index.php/topic,12171.0.html
Feb 17 16:26:33 <dlobo>   davej_: wanna chat on that thread a bit
Feb 17 16:26:57 <dlobo>   since i dont think our original list had included many / most of those things
Feb 17 16:27:13 <dlobo>   and the goal was to keep it as small as possible
Feb 17 16:27:22 <davej_>   The dupe match / import issue is a particular problem
Feb 17 16:28:03 <dlobo>   can u elaborate on it?
Feb 17 16:28:18 <davej_>   - as it effectively stops any imports from working once we have one site's data in civi.
Feb 17 16:28:33 <davej_>   Dupe matching operates across all sites
Feb 17 16:28:44 <davej_>   So no way to match within a site
Feb 17 16:29:06 <davej_>   - which is needed in order to import.
Feb 17 16:30:22 <davej_>   For GM2, it's quite common to find matches across sites as the contacts are in the same general locality.
Feb 17 16:30:47 <dlobo>   ok, thinking
Feb 17 16:30:57 <dlobo>   so u're saying dupe checking should use acl's
Feb 17 16:31:03 <dlobo>   (which kinda makes sense)
Feb 17 16:31:41 <davej_>   I think dupe checking needs to respect siloing in the same way as e.g. search
Feb 17 16:33:05 <davej_>   - so only users with view/edit all contacts should get dupe-matching across domains (& we don't currently have a use case for this AFAIK)
Feb 17 16:34:14 <davej_>   I.e. there may one day be a use case for dupe-matching across domains but the important thing is getting it working within domains.
Feb 17 16:34:41 <dlobo>   ok, wanna create another forum thread for this
Feb 17 16:35:27 <davej_>   dlobo: will do.
Feb 17 16:35:27 <dlobo>   the admin can dupe match across domains
Feb 17 16:36:14 <dlobo>   i think we should be able to do that for 3.1.3
Feb 17 16:36:21 <dlobo>   i suspect it will be an easy patch
Feb 17 16:36:38 <davej_>   dlobo: D'you mean the L1 admin with view/edit all contact &/or administer multi-org?
Feb 17 16:37:12 <dlobo>   davej_: the L1 admin with view/edit all contacts
Feb 17 16:39:43 <davej_>   dlobo: OK, so I'll start 2 new threads for Event ACL, Match/import ACL. Next on Olly's list is: Public Subscription pages for contact groups (i.e. Newsletters) are shared
Feb 17 16:39:52 <dlobo>   yeah
Feb 17 16:40:02 <davej_>   Haven't gone into this, any immediate thoughts?
Feb 17 16:40:26 <davej_>   Think he means CiviMAil groups
Feb 17 16:40:33 <davej_>   SO surprised they don't work
Feb 17 16:40:45 <davej_>   - groups are generally sorted.
Feb 17 16:41:35 <dlobo>   not sure what he means by that
Feb 17 16:41:47 <dlobo>   so might need more details
Feb 17 16:42:49 <dlobo>   we can chat more on that when u get more details
Feb 17 16:43:07 <dlobo>   i suspect the main issue will be for "anon" subscription pages, what groups do u display?
Feb 17 16:43:25 <dlobo>   since we probably display ALL public groups (since groups by themselves do not have a domain id)
Feb 17 16:44:08 <davej_>   dlobo: I'd have thought subgroups of the site parent group?
Feb 17 16:45:13 <dlobo>   i dont think we changed groups code to accomodate multi-site as yet
Feb 17 16:45:25 <dlobo>   lets get clarification from olly
Feb 17 16:45:33 <dlobo>   and that might make it a bit simpler
Feb 17 16:46:14 <davej_>   BAils has just pointed me to an example - it's civicrm/mailing/subscribe?reset=1
Feb 17 16:46:35 <dlobo>   yeah, ok the use case i suspected
Feb 17 16:46:41 <dlobo>   yeah, start a 3rd thread on that
Feb 17 16:46:50 <davej_>   - shows all mailing groups, not restricted to subgroups of parent group.
Feb 17 16:46:51 <dlobo>   i suspect all 3 threads will result in issues
Feb 17 16:47:03 <dlobo>   once we agree on the solution
Feb 17 16:47:17 <davej_>   OK, will do!
Feb 17 16:49:00 <davej_>   Right, mailing templates, headers & footers - suspect that wasn't spec'd?
Feb 17 16:52:49 <dlobo>   that we'll probably want to punt to a future relaase
Feb 17 16:52:55 <dlobo>   for now
Feb 17 16:54:28 <davej_>   OK. Olly is saying L2 users (including L2 admins) shouldn't know they're on a shared system, which is a higher standard of segregation than we initially discussed I think...
Feb 17 16:54:44 <davej_>   ...but he's pragmatic & is happy to hide stuff for now.
Feb 17 16:55:49 <dlobo>   lets get the bugs fixed first and then we can discuss more features, IMO
Feb 17 16:56:22 <davej_>   Yes. What's current situation with reports ACL?
Feb 17 16:57:44 <davej_>   - ah, CRM-5104
Feb 17 16:57:50 <davej_>   - should be OK then.
Feb 17 16:58:03 <dlobo>   we can easily add it
Feb 17 16:58:14 <dlobo>   will need to do it on a case by case basis
Feb 17 16:58:21 <dlobo>   since some reports are truly global
Feb 17 16:58:27 <davej_>   5104 marked as fixed.
Feb 17 16:59:36 <davej_>   Custom searches - these do their own SQL so would need addressing individually, is that right?
Feb 17 17:00:15 <dlobo>   yes
Feb 17 17:00:50 <davej_>   - so they're a pain, will check with Olly which if any are needed now, will have to hide others.
Feb 17 17:01:27 <davej_>   Membership types - we discussed with dgg a while back & he had some good ideas IIRC
Feb 17 17:02:34 <dlobo>   yeah, did that go anywhere?
Feb 17 17:02:52 <davej_>   - I'll check.
Feb 17 17:03:03 <davej_>   Membership types shouldn't be hard I'd have thought, as we have the membership org
Feb 17 17:03:16 <davej_>   - so can check if it's in site parent group
Feb 17 17:03:25 <davej_>   - so can do with acl hook?
Feb 17 17:03:40 <dlobo>   yeah, but we are also tryng to keep it contained to either the acl system and/or existing / new hooks
Feb 17 17:03:58 <dlobo>   which is the main problem, since some of the things dont have hooks as yet
Feb 17 17:05:24 <davej_>   Is it that civicrm_aclGroup is only called for a limited set of entities?
Feb 17 17:11:20 <dlobo>   yes
Feb 17 17:11:37 <dlobo>   for the entities that we offer acls for (which are 4-6 objects?) davej_
Feb 17 17:12:40 <davej_>   OK so membership types need further thought.
Feb 17 17:13:25 <davej_>   Activity Types are custom_values so the infrastructure is there for these, is that right?
Feb 17 17:14:29 <dlobo>   yes
Feb 17 17:15:15 <davej_>   Great. Finally: Administer CiviCRM perm shortcuts custom group ACL
Feb 17 17:16:21 <davej_>   A quick hack comes to mind: check for multisite being on & if so, check "administer multi orgs" instead. Any good?
Feb 17 17:16:45 <davej_>   - in CRM_Core_Permission::customGroup line 127 ish
Feb 17 17:18:09 <dlobo>   yeah, we can do that
Feb 17 17:18:12 <dlobo>   that seems reasonable
Feb 17 17:18:43 <davej_>   Fab.
Feb 17 17:18:49 <davej_>   As I said, we're still a bit hazy on how the "all contacts" perms interact with "administer Multiple Organizations"
Feb 17 17:19:26 <davej_>   Maybe we should just grep the source!
Feb 17 17:19:46 <dlobo>   hopefully does not happen in too many places
Feb 17 17:20:21 <dlobo>   the goal is to keep it in as few places as possible
Feb 17 17:20:30 <dlobo>   so hopefully u dont see that string in a lot of places
Feb 17 17:20:33 <davej_>   I mean we're not quite clear what "administer Multiple Organizations" does, as distinct from allowing access to all contacts.
Feb 17 17:20:56 <dlobo>   ahh
Feb 17 17:20:58 <dlobo>   not sure
Feb 17 17:21:05 <dlobo>   i'll need to check with deepaks
Feb 17 17:21:14 <davej_>   We tend to enable or disable them together.
Feb 17 17:21:20 <dlobo>   i suspect it mainly does the parent/child group stuff
Feb 17 17:22:06 <davej_>   dlobo: THanks, this has been really useful. I'll write this up on the existing forum topic & create new ones for the issues we'e discussed.

[Deepak Srivastava]

<davej_>   I mean we're not quite clear what "administer Multiple Organizations" does, as distinct from allowing access to all contacts.

Currently a user with "administer Multiple Organizations" permission -
- can see an additional organization column on manage groups screen.
- gets to associate an organization with a group on add/edit groups screen.
- can see a link on organization's contact summary screen, pointing to associated group
- has site-group visible in groups selector of contact search screen (via multisite module).

[davej]

Thanks Deepak, that's a really useful summary.

Dave J