Multi-organization via nestable groups discussion and proposed implementation

[cap10morgan]

dlobo and I discussed how to better support multi-organization installations (where each organization needs to keep their data somewhat separate but a service provider or federation organization needs to be able to easily work with aggregated data).

We discussed some options and came to the conclusion that allowing groups to be nestable would help support this configuration in a much better way. (TODO: Link to implementation wiki page here.)

Here's a transcript of our IRC discussion (note that we later determined that we cannot use the nested set model because we need multiparent support, i.e. this isn't hierarchical data so much as network/graph data):

cap10morgan: we're meeting now to figure out the multi-org stuff we want to implement
cap10morgan: we confirmed your warning that our smart groups based on relationships idea was not going to perform fast enough
cap10morgan: so now we're looking at making regular groups nestable

dlobo: wanna gimme a 5 line description of the problem

cap10morgan: ok, so what we're trying to do is support multiple organizations in the system
cap10morgan: and keep them separate as far as the staff of those organizations are concerned
cap10morgan: but then we also want to have sub-groups of those organizations (so like donors, online activists, alumni, staff, etc.)
cap10morgan: and we need to be able to centrally aggregate those sub-groups at a national level
cap10morgan: so, for example, pull up all donors of any group
cap10morgan: sorry, i'm using "group" and "organization" sort interchangably
cap10morgan: but does that make sense?
cap10morgan: the other option, which is probably more work, is to implement some sort of intelligent caching of smart groups
cap10morgan: which would be cool, but may be a bit much to bite off at this point

dlobo: do u wanna post the above on the forum
dlobo: and we'll bounce off some ideas there and how people can approach it now
dlobo: and what we need to change to do a better job
dlobo: so to recap:
dlobo: each sub-org has a few groups: donors, volunteers, members etc
dlobo: and they want to live in that silo
dlobo: i.e. view/edit those contacts
dlobo: national needs to view/edit from all sub-orgs

cap10morgan: yes, and aggregate the groups. so, for example, we need to be able to, in one step, pull up every group's donor list as one big list

dlobo: ok
dlobo: so lets assume for now

cap10morgan: so if we had each org's donor list as a sub-group of both that org and the national "donors" group, we'd have that

dlobo: that each org has only the donor list

cap10morgan: ok

dlobo: so u have your group of contacts C composed of C1 ... CX
dlobo: C1 == donor list of org 1, CX = donor list of org X
dlobo: C = total donor list
dlobo: if your org staff members are S1 ... SX (acl roles, and civirm groups)
dlobo: and S for national
dlobo: u give S view/edit access on all contacts
dlobo: and for each S1 ... SX u give them view/edit access on only C1..CX

cap10morgan: yep, that's how i'm planning to do the permissioning

dlobo: its not very scalable if X is a large number
dlobo: > 25 or so (IMO)
dlobo: but we'll come back to that
dlobo: so why dont u organize it the above way, meets your needs

cap10morgan: because the orgs have more than just donors
cap10morgan: we need to have multiple groups inside each org
cap10morgan: but aggregate based on group-type at the national level

dlobo: ok
dlobo: how many more groups besides donors
dlobo: i.e. how many groups does each sub-org create

cap10morgan: oh, there will probably 3-7 for each organization

dlobo: and how many orgs are there in total

cap10morgan: 75-100

dlobo: argh
dlobo: ok
dlobo: so a totally different approach
dlobo: (can u please post all this on the forum)

cap10morgan: yep

dlobo: so we can record it etc
dlobo: and respond there with other thoughts at a later stage
dlobo: so now for option 2 for your case
dlobo: which might be better
dlobo: each org has a seperate instance of civicrm
dlobo: so that solves the org issues
dlobo: they can create groups etc
dlobo: (does national ever edit contacts?)

cap10morgan: yeah, routinely

dlobo: or is it read only for national
dlobo: ok
dlobo: so then u have a protocol that on a "regular" basis
dlobo: updates the master db
dlobo: which is national
dlobo: and its a 2 way sync

cap10morgan: i'm having a hard time seeing how this is anywhere near as good as a solution as making groups nestable. that seems like a small change that gets us what we need. but am i just being naive?

dlobo: can u talk a bit about nestable groups

cap10morgan: yeah

dlobo: so u'll have a donor group D (national)

cap10morgan: yep

dlobo: and then each org level donor group D1 .. DX is a sub group of D

cap10morgan: yep

dlobo: for now lets assume that all subgroups are static groups

cap10morgan: yep

dlobo: (makes it a wee bit easier conceptually)
dlobo: so

dlobo: then the big modification to support this is in search
dlobo: to handle sub-groups
dlobo: if u choose a parent group
dlobo: u r right
dlobo: this is a better method

cap10morgan: cool; i was thinking we'd use a nested set model at the db level

dlobo: what does that mean
dlobo: since 2.0 will be 5.0.x only

cap10morgan: http://dev.mysql.com/tech-resources/articles/hierarchical-data.html
cap10morgan: it's a model for storing hierarchical data in a relational database
cap10morgan: keeps the performance and query complexity sane

dlobo: most excellent

[8ballsaysiwin@gmail.com]

Just came across this... What about extending the functionality of tagged data? So, ACL's could be based off tags and there would be the concept of "smart tags". Basically, extend all the functionality of a group over to a tag, so that they are essentially equivalent.

So, you could have one group to house each of your organizations, and tag donors, volunteers, alumni, etc with the separate "tag" system.  The separate ACL structures that overlap, i believe, allow for all the functionality described above. Within a group, those who have group-level access to do so, can view/edit contacts that are tagged a certain way. And nationally, the only difference is that the tags extend across all groups, so those are all view/editable.

It gets around the concept of nesting and sub-grouping and extends functionality, but i am new at this, so i don't know how flexible it would be or how easy to develop. I am working on hierarchical level organizational implementation and i thought that this would be the way we might want to go.

[Donald Lobo]

You can basically do the below via ACL hooks. Basically the hooks allow you to implement ACL's in whatever manner best fits your organization, hence you can make it super efficient also

lobo