Author Topic: workflow dev/staging/live again (Read 2882 times)

johanv · January 22, 2013, 06:11:32 am

We plan to setup several development instances, a staging instance and a live instance of civicrm (on drupal). The source code of our project is stored in a git repository, which allows us to easily move new code from dev to staging to live.

The difficult part, is re-applying configuration changes (like e.g. creation of new contact types, changing the language,...) you do on dev, onto staging or live. One option is to write down all configuration changes you do on dev, and to manually re-apply them on staging or live. However, we want a solution that is less sensitive to errors.

To solve this problem in Drupal, the features module can be used (http://drupal.org/node/580026). Using features, drupal configuration settings can be exported to modules (i.e. source code), which we add to the git repository. This way, when we fix an issue in dev by configuring Drupal, this new configuration can easily be applied on the live site. (Just enabling the newly created feature-module)

The Drupal features module cannot be used for civicrm, and we did not find a civicrm extension with more or less the same functionality. So we plan to do the same on a lower level.

If we want to reconfigure the civicrm-site, we will write a drupal module that uses the API to do the actional configuration during the module installation. We will add the module to the git repository (similar as we do with features), and enabling the module on staging or live should do all the work.

What do you think? Is this a good workflow? Do you use something else? Any comment is greatly appreciated.

Donald Lobo · January 22, 2013, 07:17:44 am

any idea if:

1. The features modules allows other modules to "hook" into it (like views / webform). i suspect it has to

2. How hard would it be for you to integrate with the features module for at least the sub-set of config changes you need so it will be a base that other folks can build on?

thanx

lobo

johanv · January 24, 2013, 01:13:17 am

Quote from: Donald Lobo on January 22, 2013, 07:17:44 am

2. How hard would it be for you to integrate with the features module for at least the sub-set of config changes you need so it will be a base that other folks can build on?

Thanks for the hints. I will research the possibilities of integrating civicrm and features, and keep you updated.

totten · January 24, 2013, 02:41:20 am

I haven't really used features, but these are some things that might help with an integration:

* A lot of tables do include a 'name' column which is usually a stable/internal name.

* hook_civicrm_managed allows you to create entities declaratively. This should work with any entity that works with the API, although it's only been tested in a few limited use-cases.
- See http://wiki.civicrm.org/confluence/display/CRMDOC42/Hook+Reference

* A lot of CiviCRM's configuration data is stored in "civicrm_domain.config_backend" and "civicrm_setting.value". For the upcoming CiviCRM 4.3, Eileen has developed a unified API for managing settings in these two tables -- and she's produced metadata to describe (all?) the settings. (I'm sure some more stress-testing would be helpful on this.) Some links:
- http://svn.civicrm.org/civicrm/trunk/api/v3/Setting.php
- http://svn.civicrm.org/civicrm/trunk/api/v3/examples/ (see Setting**php)
- http://svn.civicrm.org/civicrm/trunk/settings/
- http://issues.civicrm.org/jira/browse/CRM-10040

johanv · January 25, 2013, 02:58:08 am

Thanks for all the links.

I will try to make civicrm settings 'features exportable'. My collegue passed me this link (http://treehouseagency.com/blog/thomas-wysocki/2010/10/28/making-your-module-features-exportable), which I still have to read.

The new configuration API for civi 4.3 would be very useful for this, so it could be worth an upgrade to 4.3 (4.3 will certainly be released at the time our civi installation will go into production).

We have also considered making the strongarm drupal module work with the civi configuration settings, but since I am new to all this, I it might be a better idea to try the easiest solution first (using the 4.3 api).

thegene · February 02, 2013, 12:02:09 pm

johanv,

One of the more frustrating things I am finding about civi is that it is not terribly good with multi stage deployment strategies. My workaround for the workflow issue has been using the backup migrate module (BAM). I create a profile in BAM which excludes all but a couple of tables and download them without encryption and without the trailing timestamp into my git repository. For instance, if I am changing the nav menu layout with some customized reports, I include only the navigation_menu and report_instance tables in the profile.

Granted, this is an unsatisfactory solution compared to a features-driven one but you might find it useful.

Would love to know where you land with the features integration.

Donald Lobo · February 02, 2013, 07:18:29 pm

This is definitely a solvable issue. Would be great if a few folks get together and figure out what a good cross-CMS solution is an help implement it

lobo

mallezie · April 22, 2013, 09:14:38 am

We started work for making CiviCrm compatible with drupal features.
You're very welcome to test code on drupal.org / sandbox / mallezie / 1955416 (Sorry, apparently i'm not allowed to post external links).

mallezie · May 02, 2013, 03:20:49 am

In the meantime we promoted the sandbox to a full module.
drupal.org / project / features_civicrm

mallezie · May 21, 2013, 02:58:25 am

No idea if it's a civi-bug or mine (i guess i'm doing something wrong, which makes it my bug). If i try to use the 'setting' api, i can find all data, only the create-function doesn"t seems to work, while i don't get any error.
(I'm testing with the lcMessage-setting).

I read http://wiki.civicrm.org/confluence/display/CRM/Settings+Metadata+and+Usage
which hints that perhaps not every setting is yet converted, but i'm not sure.

Greetings.

johanv · October 17, 2013, 04:40:12 am

Sorry for replying to this old post. But I want to close this discussion by mentioning how we handle the issue.

We don't use drupal features anymore. Instead we create a drupal module that performs all necessary configuration by calling the CiviCRM API.

This works for us.

Disadvantage compared to features, is that you cannot see which feature settings have changed. So we'll have to make sure that we do any configuration changes in code, and not by clicking around.

xavier · October 17, 2013, 04:42:00 am

assuming you have you module in git, you kinda can see the changes.

Could you share one of the config module you've done? would be interesting to see how readable it is.

X+

JoeMurray · October 18, 2013, 09:22:33 am

Eileen has done a lot of work on migration of CiviCRM data using Drupal migrate. There is now a fast moving group bringing Migrate into D8 core to facilitate direct upgrades from D6 to D8 https://groups.drupal.org/imp. I think we might want to leverage their work by writing contrib extensions for civicrm. Conceptually we should be able to use a D8 site to migrate data between two external CiviCRM databases, such as a staging and production CiviCRM instance integrated with WordPress.

johanv · October 18, 2013, 11:59:56 am

Quote from: xavier on October 17, 2013, 04:42:00 am

assuming you have you module in git, you kinda can see the changes.

It is. But if you use drupal features (https://drupal.org/project/features), and you change the configuration of the site by clicking around, you can see what you have changed. Since we don't use features, we should avoid doing configuration by clicking; we should just do it in code using the API.

Quote from: xavier on October 17, 2013, 04:42:00 am

Could you share one of the config module you've done? would be interesting to see how readable it is.

X+

Here is the .install file of one of our modules.

Code: [Select]

<?php

/**
 * Implements hook_install().
 */
function chirocivi_civicrm_base_install() {
  //create / change contact 1
  chirocivi_civicrm_helper_civicrm_api('Contact', 'create', array(
    'id' => 1,
    'contact_id' => 1,
    'contact_type' => 'Organization',
    'sort_name' => 'Chirojeugd Vlaanderen vzw',
    'display_name' => 'Chirojeugd Vlaanderen vzw',
    'do_not_email' => 0,
    'legal_name' => 'Chirojeugd Vlaanderen vzw',
    'organization_name' => 'Chirojeugd Vlaanderen vzw',
  ));

  //create / change domain settings
  chirocivi_civicrm_helper_civicrm_api('Domain', 'create', array(
    'id' => '1',
    'name' => 'Chirojeugd-Vlaanderen vzw',
    'description' => 'De grootste jeugdbeweging van Vlaanderen en Brussel.',
  ));
  
  //Create / change settings
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array(
    'countryLimit' => array(
      0 => 1020,
      1 => 1076,
      2 => 1152,
      3 => 1082,
  )));

  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('dateformatDatetime' => '%B %E%f, %Y %l:%M %P'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('dateformatFull' => '%B %E%f, %Y'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('dateformatPartial' => '%B %Y'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('defaultCurrency' => 'EUR'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('lcMessages' => 'nl_NL'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('monetaryDecimalPoint' => ','));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('monetaryThousandSeparator' => '.'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('moneyformat' => '%c %a'));
  chirocivi_civicrm_helper_civicrm_api('Setting', 'create', array('moneyvalueformat' => '%!i'));

  
  //edit settings in domain config backend.
  //do after settings, if possible through settings, that's easier
  $domain = chirocivi_civicrm_helper_civicrm_api('Domain', 'get');
  $unserialized_domain = unserialize($domain['values'][1]['config_backend']);
  $unserialized_domain['enableComponents'] = array();
  $unserialized_domain['enableComponentIDs'] = array();
  //create / change domain settings
  chirocivi_civicrm_helper_civicrm_api('Domain', 'create', array(
    'id' => '1',
    'config_backend' => serialize($unserialized_domain),
  ));
  
  
  //Remove default made relationship types. 
  $relationshiptypes = chirocivi_civicrm_helper_civicrm_api('RelationshipType', 'Get');
  foreach ($relationshiptypes['values'] as $relationshipType) {
    if ($relationshipType['is_reserved'] == 0) {
      chirocivi_civicrm_helper_civicrm_api('RelationshipType', 'Delete', array('id' => $relationshipType['id']));
    }
  }
}

This one changes some settings, sets the domain info, and deletes the relationship types we don't need. You can do similar things to create new relationships, custom field sets, etc.

As you can see, we created a simple wrapper around the API, so that we don't always have to specify version number, and so on.

JoeMurray · October 18, 2013, 01:09:14 pm

It might also be nice to be able to Create a Feature by exporting initially all CiviCRM Settings (allowing unneeded ones to be deleted manually from the file) and similarly for other configured objects like events, contribution pages, profiles, custom fieldsets, petitions, etc.

CiviCRM Community Forums (archive)

News:

Author Topic: workflow dev/staging/live again (Read 2882 times)

johanv

workflow dev/staging/live again

Donald Lobo

Re: workflow dev/staging/live again

johanv

Re: workflow dev/staging/live again

totten

Re: workflow dev/staging/live again

johanv

Re: workflow dev/staging/live again

thegene

Re: workflow dev/staging/live again

Donald Lobo

Re: workflow dev/staging/live again

mallezie

Re: workflow dev/staging/live again

mallezie

Re: workflow dev/staging/live again

mallezie

Re: workflow dev/staging/live again

johanv

Re: workflow dev/staging/live again

xavier

Re: workflow dev/staging/live again

JoeMurray

Re: workflow dev/staging/live again

johanv

Re: workflow dev/staging/live again

JoeMurray

Re: workflow dev/staging/live again