Improve the profile/custom field admin interface

[xavier]

Hi,

Dealing with profiles & custom fields, for instance for an event registration or a survey is confusing and takes a lot of time.

I have been through a long (70 fields) event registration organised with 3 big organisations that changed dozen of time the help or label, and changed the order of the questions. The existing interface didn't make it a pleasant experience.

PTP users have difficulties creating surveys for their campaigns.

Part of  the problem is that the interface is imposing the opposite of the natural order: you'd expect creating a survey, then adding questions, then adding answers to the questions. Currently, what you have to do is create a custom data set, create the fields/questions (with "internal short names") and answers, create a profile, select the fields your just created (and put the long "external" names), create the survey and chose the profile you just created.

side note: as the name of the profile is the same as the one displayed to the end user, you have to navigate through tons of "more about you" "your details" and other non descriptive names.

The good folks at PTP wants to contribute to fix that and improve the workflow and I think we should go the full "edit in place" way. The general idea is that you have almost the questionnaire as displayed to the end user, but that you can directly edit the text in this page, and update the questionnaire without having to switch between loads of forms.

So you got your list of questions (profile fields) and you can click on the help texts or labels and edit them directly, save and voila. You stay on the same page with the complete questionnaire, but the text has been updated.

For some fields (eg radio/checkboxes), modifying the answers means modifying the custom field values (options), not the profile fields, but hiding that complexity is rather a good thing IMO, at least that's where I often loose my users when they have to juggle between custom fields, profiles and options on the custom fields

I don't think we can cover all the field types directly (eg adding a new gender or any of the options...) on phase 1, but should focus on the 80% and have a single page that allows to (without any refresh)

1) being able to change the texts (label/pre&post help)
2) being able to re-order
3) being able to change mandatory/not
4) being able to add an existing field
5) being able to modify/add one of the option in a custom field (radio/checkbox)

We might not be able to implement everything in place directly, but at least link better so you can directly reach the option list admin form from within the profile field for instance.

One of the most common, and trickier bit is to create a new question (a new custom field). you can add it to and existing activity data set, create a new activity data set and add the field there, add it to a contact data set, create a new one and add the field there, and you have to multiply that with the option to make the dataset for all contacts, only individual, limit it further to contact subtypes and same issue to decide beetween all activities or only for a type...

These are complex choices, that have potential big impacts down the road and seeing how my users do that, not fully mastered (and to be fair, it needs a conceptual understanding of the db schema and that's hard anyway to predict if the same question might be used again in the future)

Add in the mix the questions that have pre-defined answers (radio/checkboxes) and the added complexity to decide if you want a new option group or re-use an existing one for extra fun. And of course the chance to modify an option list and mess up another survey

So, my questions for you:
1) what do you think about the edit in place idea (that could/would be embedded in the page where you currently choose the profile in the select.
2) Do you have a satisfying way of explaining how to create custom fields, and an idea on how to do a clearer interface for that one?

[jamie]

Thanks for launching this discussion Xavier.

I think we need a balance between an easy user interface and a clean, well organized data schema.

My thinking on this topic is constantly evolving... here's is my latest thoughts (as of this morning  )

The danger of allowing people to re-use custom data fields and change their properties is that they may unknowingly break a different form. Especially with the ability to edit in place, which gives the impression that you are only acting on something that will affect your survey.

Even allowing users to change the profile related settings could be confusing when it comes to analyzing the data. Suppose I create a field and I gave the cusotm data field the short name "needs_hotel_2011_conf" and then I create a profile and give the field the label "Needs Hotel 2011 Conf." I collect the data. Then, the next year, I create a new survey, re-use the field and change the label to "Needs Hotel Room 2012 conf". This is all great.... until someone in the organization sees the custom data field with the label needs_hotel_2011_conf and does a search on that field thinking they will find everyone who needed a hotel for the 2011 conference. But - instead, they get everyone who needed a hotel in either the 2011 or 2012 conference.

I think the safest (and most extreme) way would be to dis-allow the re-use of custom data fields, period. Every survey has it's own custom data fields. The advantage is no mistakes or surprises and it's very clear and logical and simple to the end user. The disadvantage is an explosion in custom data fields and some user frustration because they have to create the same field over and over again.

I wonder if there is some middle ground?

Perhaps - allow re-use of custom data fields (people will have to be smart about when it makes sense and when it doesn't), but prevent users from changing re-used cusotm data fields. So - any modification to the profile is allowed (re-ordering fields, changing labels, etc.) but you can't change the underlying custom data field (I think most of what we would want to change is in the profile - by preventing the editing of the underlying field we'd only stop people from changing value options)

If a user finds they need to change the custom data field, it means that they should instead create a new one.

Perhaps, to ease the new custom data field creation process, a custom data set can be automatically created for the survey questions (an activity type custom data set). Every survey/petition has a one-to-one corresponding custom data set.

Then, the only custom data fields that can be changed are ones in the custom data set specifically linked to this survey.

That leaves one more piece: what if people want to ask questions that should be an extension of the contact (not an extension of the activity)? I think this is common for petitions (less so for surveys) - the page displays some questions about who you are (name, email, etc.) and then asks the particlar questions for the survey/petition.

For this purpose, for our first phase, I think it's acceptable to have people choose an existing profile (as the petition user interface already handles it). I imagine it would be quite common for people to re-use existing profiles for this purpose, so I think the flexible user interface need is less urgent for this functionality.

[xavier]

My thinking on this topic is constantly evolving... here's is my latest thoughts (as of this morning  )

I don't think there is a universal answer indeed. It's a complex problem to have a proper DB structure, it's a complex problem to have a simple UI. Add users that are not that aware of all that and you get interesting results.

The danger of allowing people to re-use custom data fields and change their properties is that they may unknowingly break a different form. Especially with the ability to edit in place, which gives the impression that you are only acting on something that will affect your survey.

Even allowing users to change the profile related settings could be confusing when it comes to analyzing the data. Suppose I create a field and I gave the cusotm data field the short name "needs_hotel_2011_conf" and then I create a profile and give the field the label "Needs Hotel 2011 Conf." I collect the data. Then, the next year, I create a new survey, re-use the field and change the label to "Needs Hotel Room 2012 conf". This is all great.... until someone in the organization sees the custom data field with the label needs_hotel_2011_conf and does a search on that field thinking they will find everyone who needed a hotel for the 2011 conference. But - instead, they get everyone who needed a hotel in either the 2011 or 2012 conference.

I think the safest (and most extreme) way would be to dis-allow the re-use of custom data fields, period. Every survey has it's own custom data fields. The advantage is no mistakes or surprises and it's very clear and logical and simple to the end user. The disadvantage is an explosion in custom data fields and some user frustration because they have to create the same field over and over again.

I think it's not an option, the back office interface is basically broken if you have a dozen custom groups, especially since the (bad) idea of loading some of it async using ajax.

And no matter the interface, having a gazillion custom table (one data set per survey) or a gazillion columns (re-using the same data set but creating new fields) are not going to work nicely.

As to take your example (that reminds me another one more colourfully worded I think that's a matter of training as the "proper" way is to have one "Needs Hotel  Room" field and change the label on the profile as you suggested

If a user finds they need to change the custom data field, it means that they should instead create a new one.

I feel like we'd be encouraging a bad practice. But finding the right explanation is really tricky

Perhaps, to ease the new custom data field creation process, a custom data set can be automatically created for the survey questions (an activity type custom data set). Every survey/petition has a one-to-one corresponding custom data set.

At least offer that as an option ("use an existing field", "create a new set for this survey", "associate this survey with an existing set") ?

That leaves one more piece: what if people want to ask questions that should be an extension of the contact (not an extension of the activity)? I think this is common for petitions (less so for surveys) - the page displays some questions about who you are (name, email, etc.) and then asks the particular questions for the survey/petition.

Well, some, eg age, gender or country fits naturally into a survey. I don't think we stretch it too far imagining custom fields for a contact/individual/subtype (ethical info, department they work on...)

For this purpose, for our first phase, I think it's acceptable to have people choose an existing profile (as the petition user interface already handles it). I imagine it would be quite common for people to re-use existing profiles for this purpose, so I think the flexible user interface need is less urgent for this functionality.

What about adding a field on an existing profile?


This being said, I think you identified a missing info: is the profile already used? is the field shared with another profile (when trying to modify the option)?

Those info would help taking the right decision

X+

[petednz]

I know this is Drupal-side discussion but I am finding myself leaning on using Webform more and more for surveys, data input, etc.

Clearly that isn't a solution for paid Events (yet or at least not without use of multiple modules resulting in ubercart for example handling the payment) and nor does it help get this functionality in to CiviCRM core - but it may be a useful alternative to use as a comparison of approaches for such a project.

[Donald Lobo]

a couple of random thoughts and comments on a late friday nite:

1. I do think that we need to make people aware of the implications of creating "custom" fields and groups. Making it too easy to do so, has the bad implication of basically having a bloated database with redundant data across various tables etc.

2. At some point in the future, we do need to have profile evolve to have more "webform" like capabilities across CMS'es. There have been a couple of inquiries on this and i hope we'll be able to add this to core in a future release

lobo

[xavier]

1. I do think that we need to make people aware of the implications of creating "custom" fields and groups. Making it too easy to do so, has the bad implication of basically having a bloated database with redundant data across various tables etc.

Agree. But a hard to use interface doesn't help preventing a bloated database. Right now, the UI expose the complexity, but doesn't explain it.

2. At some point in the future, we do need to have profile evolve to have more "webform" like capabilities across CMS'es. There have been a couple of inquiries on this and i hope we'll be able to add this to core in a future release

What do you think is missing on profiles to be more webform? Is this about the UI or feature?

[xavier]

Playing with webform civicrm integration (definitely really worthwhile using)

What is better than on the profile/field and we should emulate
1) being able to add several fields in one go
2) drag n' drop reordering
3) changing mandatory or not for several fields

I'm not clear of how it works re-using fields between webforms. is re-using an existing field a matter of creating the same column into a different table (one per webform)?

[Coleman Watts]

I'm not clear of how it works re-using fields between webforms. is re-using an existing field a matter of creating the same column into a different table (one per webform)?

It's completely different with the webform module. There's no ability to re-use option lists, although you could clone the field, or the entire form (node_clone module). This just creates a copy.
Webform has no equivalent to the civicrm_option_value table. Instead, options are stored with their component in a textfield, like

Code: [Select]

1|Female
2|Male
3|TransgenderThe webform_civicrm module hides this textfield for civicrm fields and replaces it with the drag-n-drop table to give a nicer user experience.

[xavier]

Thanks for the clarifications

Wondering if the "extra" step is the profile. Ie on an event/survey, instead of only the option to re-use an existing profile, you could start from scratch and add the fields directly. Webform and the option to add several fields seems to be a better solution to having to go through dozen of profiles that have non explicit names ("more about you"...), switch to the front office view and see it's not the right one

In the background, we still use profile, but add a "is_hidden" field, like for searches

Thoughts?

X+

[josue]

i just started using the civicrm webform module, and i must say i am quite impressed. the ability to add people and activities on the same screen is very handy. and when i add the arrange fields module, creating a form that looks good (via the UI) is then possible.

as for the survey interface:

phase 1, but should focus on the 80% and have a single page that allows to (without any refresh)

1) being able to change the texts (label/pre&post help)
2) being able to re-order
3) being able to change mandatory/not
4) being able to add an existing field
5) being able to modify/add one of the option in a custom field (radio/checkbox)

yes, yes, yes, yes, yes!

Perhaps, to ease the new custom data field creation process, a custom data set can be automatically created for the survey questions (an activity type custom data set). Every survey/petition has a one-to-one corresponding custom data set.

At least offer that as an option ("use an existing field", "create a new set for this survey", "associate this survey with an existing set") ?

i think that when you create a new survey you should get the option to create a new set or associate with an existing set. encouraging people to reuse an existing set and add questions to that set is one way of trying to minimize some of the bloat. it would be great if, when you choose an existing set, you could get all the fields in the set and be able to choose which ones you want to use. (the civicrm webform module does this and it is quite handy) and also be able to add fields to that set.

as for re-using fields, i don't think that changing the label is too bad, especially if we can teach people to search for that field AND the name of the survey they are interested in in order to get the proper results.

I think you identified a missing info: is the profile already used? is the field shared with another profile (when trying to modify the option)?

being able to show this info would be really valuable, since it is very challenging now to figure it out.

[xavier]

Ok, the more I think about it, the more I think the "extra" step that can be skipped IS NOT the custom field/set (that has a big potential impact) but the profile.

Having a option to create an "hidden" profile for the survey allows to focus on the fields (standard or custom) and having something like webform where you can add several fields in one go makes it easier.

I'm going to play with that idea an prototype an interface at the sprint in london and check with lobo.

X+