Form administration and Angular customization

[totten]

Context: We're increasingly using Angular for forms. If you follow Angular docs or tutorials, the form administration is pretty different from what developers normally do in Civi/Drupal. I want to verbalize a bit about the trade-offs in what we do -- to identify what we should carry-over to Angular.

Background: CiviCRM is a prebuilt application that covers a range of use-cases, business-contexts, and features. It comes bundled with hundreds of forms. Generally, there are a few "plays" you can make:

a. Use a prebuilt form
b. Alter a prebuilt form
c. Define a new form

Definition: OK, this is gonna sound pretty bland. Form administration is the workflow by which we maintain forms. Tada!

Form administration and CiviCRM: There are several different examples:

1. Profile Forms (define a new form; alter a prebuilt form):  An administrator creates a profile through the web UI. The profile system is strongly supported -- if you upgrade to a new version of CiviCRM, the profile will still work the same way in the next version. You can easily create as many profiles as you want, and it only requires moderate technical skill. In some cases, a profile form can be plugged into another form (e.g. on the event-registration form). However, upstream rarely provides proactive improvements to existing profiles, and the range of features supported by a profile are pretty limited.

2. Webform-CiviCRM (define a new form): This is very similar to Profile Forms. (An administrator creates a webform through the web UI. The form is strongly supported, and you can make as many as you want with moderate technical skill.)  It differs primarily in that (a) it provides a much richer set of form elements and (b) it has a hard dependency on Drupal 7 (incompatible with ~40% of install-base).

3. PHP/TPL Overrides (alter a prebuilt form): An administrator copies a Smarty template and/or PHP class... then adds or removes elements (to taste). This allows him to add/remove many nuanced features (incl details of HTML/CSS/JS). However, there are many implicit dependencies which are weakly supported -- if you upgrade to a new version of CiviCRM, there's a good chance that the overridden form will break or (best case) miss out on new features, so you must carefully revalidate and manually merge changes. In practice, you don't create multiple forms. This requires HTML+PHP+CSS+JS skill.

4. Git Forks (alter a prebuilt form): An administrator forks civicrm-core.git and makes changes in his own branch. This works out roughly the same as overrides (pretty flexible but w/many implicit dependencies). However, upgrading isn't quite as difficult -- simple changes (to a single file or HTML element) merge automatically, and you have a built-in log of changes (both for your fork and for upstream). Never-the-less, upgrades still present merge-conflicts (which must be investigated). This requires HTML+PHP+CSS+JS+git skill.

5. jQuery Alter (alter a prebuilt form): An administrator inspects an existing form and prepares a selector for the section which he wishes to change; then, he writes a hook (or extra.tpl) which applies a change on top of the form. When upgrading, this technique won't suffer upfront merge-conflicts (like in git forking)... in fact, it won't say anything about potential problems; you must diligently test. There are some implicit dependencies, but not as many. The main downside is that it's harder to get a broad snapshot of the components in page -- they page definition is split across more files. This requires HTML+PHP+CSS+JS skill.

6. Drupal Form Alter (alter a prebuilt form): In terms of form administration, this is basically the same as jQuery alter. The differences are in implementation (eg selector notation, component names, JS vs PHP, client-side vs server-side).

Form administration and Angular:

In following an Angular tutorial or book, forms are primarily defined using an HTML document and JS controller. The form administration is fairly simple: make your own forms from the ground-up. You can change whatever you want without any negotiation, but you have to design and implement every form -- there are no prebuilt forms.

This is not unique to Angular -- Angular, React, Ember, and Backbone are all the same (in terms of form administration). These libraries and frameworks are targeted at HTML+JS developers. Prebuilt forms are simply outside their scope, so there are no standard practices for altering or overriding prebuilt forms.

Do we need to alter or override prebuilt forms?

Many projects don't need support for altering or overriding prebuilt forms. JS MVC frameworks don't. Crypto libraries don't. Custom, in-house business applications don't. Twitter.com and google.com don't. And, for the majority of CiviCRM users and the majority of CiviCRM screens, users generally work with Civi's prebuilt forms. (Yay!)

But... CiviCRM is an open-source development community. We need to ensure that the developers and customizers and contributors who drive improvements can do their jobs without too much stress. We know that TPL overrides, forks, or alter-hooks can be a source of stress -- especially during upgrades. (Even if you don't write code yourself, you may use extensions or modules which rely on these techniques -- which means that your upgrade-experience depends on it.)

Candidates / General Concepts A few ways we might approach the problem of altering/overriding Angular forms in CiviCRM:

1. git fork: The simplest option -- just fork civicrm-core and hack. This is basically where it stands today, but it's monolithic. You can't easily mix-and-match modules (which each do separate alterations) and non-technical users can't configure it through the GUI.

2. Custom notation. Angular under-the-hood: Design a custom form notation (e.g. JSON or YAML or PHP arrays) -- this is the canonical representation of a form in Civi, and we expose it to an alter hook. Implement the runtime for it in Angular (or some other system). However, this loses some of the key benefits of choosing a popular third-party form system: we can't pass-through the documentation (where a developer reads about Angular and then gets a good understanding of Civi). Moreover, it doesn't work at all with our existing set of Angular forms.

3. Angular markup as hookable data: The canonical representation of a form is Angular-style HTML, but we provide an alter-hook. (This doesn't mean exposing a flat string -- I think http://code.google.com/p/phpquery might be a good interface.) I did some sketching of a hook last weekend ( https://gist.github.com/totten/acfca15a8df8ec921296 ). Generally, there is decent tooling on both client-side and server-side for manipulating HTML content (e.g. DOM, jQuery, phpQuery); if we focus on a subset of Angular, I think it would be reasonable to provide GUI editing. But all of this far out - I'm not nearly satisfied with specs yet.

Wrap up

As I kick it around, I'm growing more keen on Angular markup as hookable data-model. However, before taking a public deep-dive into that, I wanted to share these high-level thoughts. What do you think? Have I missed important goals/criteria/obstacles?

(If there aren't problems, I'll follow up with digging in on #3.)

(NOTE: Edited May 2016 to also reference webform_civicrm example.)

[Eileen]

I agree #3 seems worth looking into