Proposal: Installer Rewrite

[totten]

I'm starting to think that the installer is due for a rewrite/rethink. Specifically, by "installer", I'm referring to the code which creates database tables, config files, data folders. (This is separate from the process of downloading code.)

At a high-level, today's distribution-channel is different from 2005's -- hosting has moved into the cloud, a notable chunk of the market expects to have ownership over their deployment cycle, and Civi integrates with more CMS's.

However, I didn't come to this proposal from a high-level. Rather, there's a litany of issues in the current system:

 * In managing our test/demo/training infrastructure, I've basically had to rewrite the installer.
 * There are, in fact, several different installers for Civi:
   + "civicrm/install/*.php" (which provides a web-based UX for D6/D7/WP)
   + Joomla installer
   + D8 installer
   + drush installer
   + wp-cli installer
   + provision_civicrm
   + buildkit
   + setup.sh
 * There are arbitrary variations between the installers. For example:
   + "civirm/install/*.php" (D6/D7/WP), drush, wp-cli, and setup.sh require the user to enter a DSN (so that he has an opportunity to split CMS+CRM across diff DBs)
   + Joomla, D8, and provision_civicrm installers don't display a screen -- they reuse the CMS DB.
   + "civirm/install/*.php" (D6/D7/WP) allows one to optionally load sample data. setup.sh and buildkit require it. Other installers don't support it.
   + (IIRC) "civirm/install/*.php" and D8 support checking "requirements", but none of the others do.
 * In drush+wp-cli, the download logic is tightly coupled to the install logic. This creates a chicken/egg problem for new installations, and it doesn't work well for folks installing unofficial code (e.g. developers, forkers, testers).
 * The Joomla installer insists on moving code into the Joomla source tree. This works for typical users -- but sucks for development.
 * Some of the installers have... ugly code.
 * The install UX doesn't "feel right" to CMS aficionados.
 * SQL-files are pre-generated.
   + This relies on mysql_real_escape_string(), but that's deprecated because it doesn't handle connection-specific encoding issues. The "correct" solution is to generate SQL at runtime (with escaping rules suitable to the connection).
   + This makes the build/distribution process more complex for localization -- one must pre-generate .sql files for each combination of locale and Civi version.
 * Contemporary build + deployment tools (e.g. docker, vagrant, heroku, puppet, composer) require the system-integrator to script out installation. If I could give a grade to Civi's support for its scriptablity with these tools, it would be "B" -- but I'd like it to be an "A."
 * The file-format of civicrm.settings.php makes it hard to maintain during upgrades.
 * Cerberus: For core development, it's nice to have a three-headed system (D/J/WP) with one Civi codebase. This is kind of possible now -- but it's hard to install directly with a cerberus layout thus hard to automate/reproduce.

None of these issues is really fatal in itself, but I'm starting to think that the installer is the common thread in a batch of issues. My proposal is to redesign the installer -- instead of 8 installers, there should be 1 well-designed core installer with ~6 thin frontends.

Pseudocode: https://gist.github.com/totten/5f68b5b8d453be768625

(Note: Edited to add a couple more items to my laundry list.)

[Eileen]

I'm sure I don't need to suggest you include Torrance's d8 install class in your review...

[Erik Hommel]

Sounds like a good idea to me!

[Coleman Watts]

+1

[Michael Z Daryabeygi]

Sounds great.
Also if we could invent a better way to ensure that different upgrade paths are consistent.

[Eileen]

Yes, re upgrade paths - I know this is out of scope but just in case ... being able to add things like indexes in a more robust way. At the moment you can't move ownership of a table from extension to core as the create table sql will fail. More often the problem is around dropping or creating indexes crashing the upgrade script

[JonGold]

This sounds great!  The one issue I have might be a topic for another thread, which is Tim's feeling that a single db for Civi and the CMS is preferable.  I was wondering if this is how other devs are doing things these days as well?

I often am doing a Civi implementation while a colleague (or another shop) handles the CMS implementation, and being able to backup/restore Civi independently is helpful - and separate dbs makes that much simpler.  Likewise when I'm doing a test upgrade or import, and something might need to be rolled back.  Quickly restoring during test migrations is another use case.

Perhaps I just need to be cleverer about how I do my backups?  I'm wondering how other experienced devs are handling this?

[xavier]

great ideas,

kinda linked: it seems we can't integrate properly into the drupal automated module distribution because the licence is different.

What is that new install was under a gpl licence? would it allow to get civicrm install (the module) into drupal, and consider civicrm-core/package/other as "libraries" under our aGPL?

It wouldn't solve 100% of the problems to automate the install, but would help feeling more "native". I'm not sure about WP nor J!, but might be the same.

X+

[kreynen]

Using an installer module licensed as GPLv2 or later (GPLV2+) to load an APLv3 licensed PHP project that also contains several additional libraries using a handful of other licenses is really dancing around Drupal.org policy vs. GPL requirements.  The resulting combination of code would need to be licensed as GPLv3 and AGPLv3 if it resulting package of code was distributed. 

While I love the idea of improving CiviCRM's integration with Drupal's update status and usage tracking, you should weigh the effort this will take with how much longer these advantages will remain relevant in a D8/Backdrop/Composer/GitHub world.  While Drupal is making major investments in improving the taste of the "dog food" they force themselves to eat by using the Drupal CMS as an issue tracker with version control integration, every indication I've seen is that unless Drupal moves away from patches to a PR workflow, a growing number of existing developers and most new developers will continuing migrating to GitHub.  Backdrop is developing their update status for modules and themes using a combination GitHub service and a centralized directory which is very appealing.  While many Drupal projects periodically commit updates and package releases on Drupal.org to take advantage of update status and usage tracking, it is becoming increasingly common to find project pages on Drupal.org that just point to GitHub and developers who have abandoned Drupal.org completely.

LICENSING: While there isn't a lot of case law about this and IANAL, I have been working with the Drupal Association and the handful of community members who are bother active and well informed on licensing form a new Licensing Working Group (https://www.drupal.org/node/2387339) to address these very issues.  For many years, it has been unclear who even had the authority to make decisions about licensing policy or enforce it.

Drupal's current policy of requiring EVERYTHING (code, fonts, images, content, etc) that is committed to their repository be licensed with the same license as Drupal core and views all modules as derivatives of Drupal core.  These are not policies or views shared by other popular PHP CMSes like WordPress and Joomla.  While the legality of Drupal's interpretation of derivatives can be argued, the goal of the policies made sense in the early days of the project.

Anyone downloading code form Drupal.org could have safely assumed it was either directly licensed as GPLv2+ or used a lax license like MIT that allows re-licensing that code as GPLv2+. 

The problem is that the open source licensing landscape has changed since the Drupal project started while the Drupal.org policies have not and the community didn't scale to match the amount of new modules being committed and has failed to enforce the policies.   This leaves the Drupal Association exposed to some level of legal liability for "enticing" users to violation the terms of these licenses. 

If someone still believes that all code in the Drupal.org git repo follows the https://www.drupal.org/git-repository-usage-policy, they would be wrong.  There are hundreds of known, reported violations and likely thousands more if anyone wanted to look for them.  (FYI... no one does).

While most people agree that some Drupal.org licensing policy violations are obvious violations like committing PHP code licensed as Apache2, other violations aren't as clear and don't pass a common sense test.  The most common are fonts and images licensed with SIL OFL or a Creative Commons ShareAlike license.  The example I use to show how insane trying to enforce these policies has gotten is Drupal's documentation.  While someone has the foresight to make all code snippets included in a documentation wiki licensed as GPLv2 or later, the rest of the content is licensed with a Creative Commons ShareAlike 2.0 license.  According to the FSF, that CC BY-SA license is not compatible with a GPLv2 or later license (https://www.gnu.org/licenses/license-list.html#ccby).

So the act of copying text from a wiki and pasting it into the help function of a module to make it easier for users to use the module is a violation of Drupal.org's current Git Usage Policy and should result in hundreds of developers having their projects unpublished and being banned from future git commits.

The new LWG will be tasked with not only revising the Git Usage Policy to take an approach to non-code assets that passes a common sense test, but also the clarifying the policies around committing code to Drupal.org's git repo vs. committing packaging instructions like a .make file of a git submodule.   

My point in sharing all of this is that things are changing with both the licensing policies on Drupal.org and options for packaging modules.  These changes won't happen quickly, but I'd like to see CiviCRM pushing for changes to Drupal's policies that allow that best technical solution over developing a less than ideal solution to adhere to the current policies.

[totten]

I often am doing a Civi implementation while a colleague (or another shop) handles the CMS implementation, and being able to backup/restore Civi independently is helpful - and separate dbs makes that much simpler.  Likewise when I'm doing a test upgrade or import, and something might need to be rolled back.  Quickly restoring during test migrations is another use case.

These are good points. Perhaps deciding factors are the size of the DB and the engagement level of the technical team:
 - For a large DB, split DB configuration improves performance during upgrades/testing. For a small DB, the performance gain is a rounding-error.
 - For an engaged/experienced/professional/multi-person team, split DB allows more separation of responsibilities. For a novice/volunteer/part-time/solo admin, single DB allows simpler administration (eg unified backup, simpler install steps, no weird configuration for Views) and doesn't contradict our marketing ("Civi and CMS work together to make things easier").

My feeling is that there are two paths which lead to wanting a split DB:
 - The site starts out with an experienced/professional/multi-person team. These folks probably use drush, wp-cli, or some customized tooling. Performing a CLI install here shouldn't be a burden.
 - The site starts out with a novice/part-time/solo admin -- but then things change. (Perhaps the admin struggles? Perhaps he grows more experienced/professional? Perhaps the organization grows and brings in more people?) At this point, we'd need a migration function(e.g. "foreach $table: RENAME TABLE ${drupaldb}.${table} TO ${cividb}.${table}").

kinda linked: it seems we can't integrate properly into the drupal automated module distribution because the licence is different.

While I love the idea of improving CiviCRM's integration with Drupal's update status and usage tracking, you should weigh the effort this will take with how much longer these advantages will remain relevant in a D8/Backdrop/Composer/GitHub world.

First - thanks for the comments on Drupal's licensing situation. It's great that LWG investigating some kind of policy change.

I agree that that d.o's distribution channel might be nice -- *and* that we should be more concerned about future distribution channels. My main goal in this proposal is to improve the installer so that it's simpler/easier with *any* distribution channel (or no particular channel).  To wit: the topics of download ("drush dl", "wget", "composer") and install/configuration (DSNs, schema, civicrm.settings.php) should be separated so that we have flexibility on each.

[totten]

A couple minor thoughts related to installer:

1. Just had a bug where setup.sh and buildkit were working for 4.6 but the web-based installer wasn't --  because they use different installers. (Specifically, setup.sh uses "mysql" to execute .sql files, and web-based installer uses its own parser for .sql files.) I understand that the web-based installer needs a PHP parser for broader compat... setup.sh should use the same one.

2. On IRC a few days ago, someone (deweydb?) pointed out that migrating is... difficult ( http://wiki.civicrm.org/confluence/display/CRMDOC/Moving+an+Existing+Installation+to+a+New+Server+or+Location ). This isn't entirely the fault of the installer, but there's some overlap in the issues. Would be good to keep that in mind in designing a new installer.

[mathieu]

Kind of tangential, but I would really like to have an installer that supports i18n, i.e. so that the text of the installer itself (intro, requirements, etc) can be in the user's language.

The main issue for that is:
https://issues.civicrm.org/jira/browse/CRM-13584

I have a new patch against 4.6 here:
https://github.com/mlutfy/civicrm-core/commit/93974d6bd0d4b2ae7046f9ed1a99b3363179d415

If we could get that in 4.6, it would be really appreciated. To anyone not fluent in English, having the installer in English-only is a huge barrier to entry, and reflects badly on the quality of the software.

My next item on the wishlist, is making sure that once the installation is finished, it should set the default language to whatever was preseeded. Currently it fallsback to English, and the user has to go to "admin>system settings>localisation" to set it back to the seeded language. Another related issue, is that we should preseed the default currency, date formats, etc.

+1 on all the rest!

[JoeMurray]

+1 on doing this and general drift of comments.

Based on observations above I think it would make sense to optimize certain installation methods for certain classes of users. In particular, D8 web install should put CiviCRM into same db as Drupal, while drush would enable separate dbs through different parameters (perhaps defaulting to sharing db if no args provided to specify the CiviCRM db). That way, the initial and on-going process is simple for less technical users, while developers and more sophisticated shops can continue to separate the dbs.

[totten]

Thanks for the comments, Joe.

Just got reminded of something and didn't want to forget it...

Contemporary build + deployment tools (e.g. docker, vagrant, heroku, puppet, composer) require the system-integrator to script out installation. If I could give a grade to Civi's support for its scriptablity with these tools, it would be "B" -- but I'd like it to be an "A."

To add a bit about how this point fits into a bigger picture -- one of the issues with Civi extensions is that Civi doesn't have a dependency-manager. We could write one, but it's not cheap, and it still leaves us in a position where you have to follow several different playbooks for managing a build. Ideally, one dependency manager could work with CMS + CRM + extensions to both. Several pieces are already in-place -- e.g. composer has options for downloading Drupal/Joomla/WordPress extensions, and Civi can load extensions from composer's "vendor" folder. However, once you have this big source tree, there's still the problem of setting up DBs+config files. That's where it helps for Civi to include a scriptable/loosely-coupled installer.

[totten]

IRC convo with IRC reminded of two more things in this general realm:

1. Running tests. You have to basically install Civi in a headless DB to provide a baseline. This was scripted sorta sui generis for core but doesn't generalize well to systems with extensions. The "civix test" command does some workarounds, but it's not pretty.

2. When running tests in extensions with "civix test", the extensions have to be stored in the Civi source tree - they can't be loaded from the dynamically-configured extension dir. It's some kind of bootstrap/initialization race.