CiviCRM Community Forums (archive)

Now that we have CiviCRM in git it would be good to incorporate that into our deployment process. (actually I think 4.3 was already on git we skipped 4.3 - partly because of the deployment issue).

Background
Prior to 4.1 (which we also skipped) it was possible to get through a version without forking it because the policy on allowing fixes & improvements into the stable release was quite open. However, for stability reasons this changed in 4.1 & I don't think anyone managing more than multiple sites would consider doing it on 'stock' CiviCRM now.

Fuzion 4.2 deployment (our current version)
In 4.2 our approach was to check the unzipped files from the release into github as a branch. From this we removed the drupal folder & added all the l10n files & that was our base.

We then created a branch off that called 4.2-master and all the patches we did were added to that. We then added 2 more branches - 4.2-d6 & 4.2-d7 which were master + the drupal folders & there were our deployment branches.

When a new release came out we upgraded the base branch & rebased the 4.2-master over that & the drupal branches over that. Each release some commits needed tidying up & I tended to squash some as well. When other consultants wanted to do an LTS release of 4.2 I cherry-picked all my commits onto the LTS & the LTS became 4.2.9 + about 80 patches from our repo. (I only put patches already in 4.3 or 4.4 into the LTS).

Attempting to work with CiviCRM in git
My expectation was that when CiviCRM went onto git the CiviCRM version would become the base of our versions. However, having tried to make this work on 4.3 & the LTS and it has been very painful. Basically the problem is that the civicrm-core doesn't include important folders like packages DAO & does include deployment files which are not all in one directory.  This means building a base revision off the civi code is a PITA - but worse cherry-picking stuff across causes all kinds of problems because a patch to core might include changes to tests which have been removed in the base revision but cause a conflict or re-add a test as you try to pull across.

What about make files?
I had been hearing about a drupal-make approach perhaps being the answer - but talking to Chris today it still seems like you need to build the patches you will apply on top against a base revision - so the need for a base-revision appropriate to deployment remains.

So?
So, at this stage nothing seems to beat simply ignoring the fact CiviCRM is now in github & checking in the actual releases into our github. We can't do pulls directly against core but for some reason github seems to have removed the ability to turn an individual commit into a PR anyway (a cherry-pick PR) & there is no way doing a PR from a deployment release into core would work.

Overall I'm spending way too much time wrangling code between our deployment repo & submitting to Core & I have some personal strategies that I am contemplating - but I am aware that what would be good for me would not be good for the community - especially not if everyone adopted those sorts of strategies.

What is working for other people?

Hi,

I feel the pain, we have a workflow with a bit of rsync and a bit of git, but too hackish to be considered as a good solution.

May be the time has come to properly manage dependencies and use composer to "make" a civi in one step.

for our own need, we would "just" have to change the composer.json to cherry pick with version of civi-core and whatever else is non standard.


X+

I absolutely concur: the new git structure makes contributing to upstream very cumbersome and time-consuming.

This comes from the fact that the git repo is not directly 'deployable' (creating an operational Civi instance from git takes at minimum cloning of 3 repos, sometimes 4, and deleting some directories). So I, as Eileen and other implementers do, maintain a unique 'deployable' repository that is also our main development and testing repository. From this repository we can deploy to our customers with a single git pull.

But if we want to contribute our patches/enhancements to core, in addition to this repository we need to maintain public clones of the 6 LTS repositories, and public clones of the 6 core team repositories. So we jump from a single repository repository to 13! And that's not even counting the separate local clones, development/test environments, etc. Moreover, since our 'deployable' repository is a clone of neither, and the github UI is very particular about PRs, we have to manually copy/paste bit of code from our repo to the LTS clones and the core team clones (yes, that's twice), and create branches and issue PR's in each 'silo'.

Very cumbersome and time consuming. And I'm not event tasking about the frustration of having the core team refuse your patch for some good reason or another, and having to go through this whole process again!

I think there are multiple ways to solve this issue:

• go back to a 'deployable' repository structure such as we had in SVN
• use nested repositories with the git submodule feature
• automated builds with a dependencies manager such as composer
• automated builds with a scripting system such as make or phing

Whatever is chosen, the primary goal would be that maintainers of clones/forked version of CiviCRM can both:

• deploy to their customers with a single pull,
• and contribute back to upstream with the minimum amount of work - ideally just issuing a PR from the same repository.

I deploy with "Drush make":

* deploy from my git repos or from the civicrm standard ones (drush make example.make)
* run GenCode.php

Example makefile: https://gist.github.com/mlutfy/5310366

What do you mean by: "build the patches you will apply on top against a base revision"? Since you're cloning your own git repo (ex: fuzion, or github.com/mlutfy/civicrm-core), not from github.com/civicrm, you don't need to apply your own patches from the issue que, as we did when drupal.org was still on CVS.

It could be a nice improvement if we had DAOs in git directly. Although I wonder if a post-checkout hook would do the trick: http://nschoenmaker.nl/2013/07/composer-post-checkout-hook-in-git/?

But I agree that we should make the developer barrier simpler.

Eileen - I went through the IRC logs, a lot was discussed about deployment, but I think the issue is more on the development side. You and I deploy with a simple git pull, no need to change that. The issue on the development side is that it is cumbersome and time consuming to contribute back patches, in great part due to these split repos.

On your comments:

• composer could help by managing the 'packages' directory. You list all dependencies in a text file and they get downloaded automatically by composer. It would be very useful as well in understanding which packages are original vs modified, as Kevin pointed out
• on the extra files, I agree with you we should not leave all these dev/test/packaging scripts in live instances, they represent a huge potential security risk
• we definitely should include the DAO's in core. I would also advocate to push all development tools, makefiles, scripts AND xml and source sql files in a 'development' repository. This way we have clean separation between the 'runtime', ie. deployable environment that people can just git clone and run, and the 'development' that includes all stuff used by the developers.
• all solutions based on drush, aegir, etc are Drupal-only and do not apply to the other CMS'es (or Standalone for that matter). They should not be part of our brainstorming since we also want to include the non-Drupal developers in our community, and therefore make it easy for them to contribute (right ?)
• I love your #7 - this is the ideal world for me as well: paste a link to your commit, you've just contributed to core! But I do like the PR process for the peer review a well. What angers me in the PR proces today is that it is linked to a specific branch: I would like to be able to submit a single PR, and that then the core team can merge it in 4.4, 4.3 and 4.4, or LTS, 4.3 and 4.4. This would combine the best of the patch file process with the best of the PR process.

NB -

I am quite new to CiviCRM and I am surprised about the difficulty of version management which does not seem the case with other open source software (Such as Contao and Drupal).
I don't know how true that is - I suspect a very small subset of drupal users submit patches for core. Patches to modules are more common - but they tend to languish & it also seems painful.

A lot of the difficulty is around the pushing upstream process.

@bgm, thanks for the example drush-make file!

@krenyen, I agree that the current management of civicrm-packages is scary/cumbersome. Would greatly prefer to kill that repo and use something like composer or drush-make to handle those dependencies. (This would require us to untangle some of CiviCRM's adhoc hacks/forks inside packages and align better with upstream... but that's good!)

@krenyen, what were the issues that prevented you from using drush-make to prepare a distribution from source on github.com? @bgm's example makes sense to me, but maybe there's something more - something we can fix?

@nganivet, I like your list of four options. I feel like options #2, #3, and #4 all share a basic architecture: civicrm.org provides components in a way that can be remixed, and then the system-integrator/implementer does the remixing with whichever tool he likes (drush-make, composer, git-submodules, git-subtree, phing, bash, etc). The outlier is #1. In approach #1, everyone needs to agree on what the "deployable" file-structure is. Will try to comment more on #1 later (because eileen digs into that one). But I really believe the architecture of #2/#3/#4 gives integrators the most benefit. (My main disappointment is that I haven't had time to bring this approach all the way to fruition.)

@eileen, a key pain-point that struck me in your comments in the IRC log -- it seems that (a) you submit every change to three repos (fuzion+LTS+official) and (b) you commit to *all* repos *before* code-review. That sounds painful. Could we simplify without making any technical changes -- i.e. make some low-tech changes to the workflow:

a. Don't do the gitwork for LTS yourself -- instead, send a 2-line email to someone else on the LTS team with a link to your civicrm-core PR. They can backport it. When the LTS team formed in April/May, they wanted to build on your work -- but didn't want to impose on you to do everything. They were ready/willing to help. Delegate! They're capable!

b. Develop patches as PRs to civicrm-core first. Get feedback and let Jenkins run the tests. After QA, backport the final fix to fuzion's git and notify LTS (if you feel like it). This avoids the problem of re-submitting intermediate versions to multiple places.

1) Several people have mentioned composer. I'm not sure I understand how that would help - could someone explain? It seems to me composer is the holy grail in the same way that that make scripts are - if you have an appropriate base which seems to be the sticking point

I think the origin of it is this: your current architecture requires maintaining a long-running, deployable fork (ie one in which you've manually merged changes to CiviCRM, Drupal, and all the other modules/extensions you use). That's painful no matter how you slice it, and the best alternative is to make it unnecessary by using a good build tool. Composer and drush-make are cool, but there are many, many others.

But listening to your workflow, I agree that there's an issue with "appropriate base" - or, put another, "serving too many masters". It sounds frustrating. That workflow puts you in the position of submitting code to too many places in a weird order. I think the low-tech solution (above) is probably simplest, but there are other solutions:

- Setup a "merge-forward" workflow (we have scripts to do this for 4.3=>4.4). Since you mostly work on 4.2, you would submit changes for review/testing to 4.2-LTS, and then all changes to 4.2-LTS would "merge-forward" en masse to 4.3/4.4. This in turn requires some work (e.g. upfront work with "git graft/git replace" to reconcile the repos, and then on-going work to resolve merge issues). I don't know who would do that work. (Maybe someone from LTS team?) But it's another possible solution.

- (There was another approach... but I forgot it... it's taking way too long for me to reply to this thread...  )

2) yes, the svn repository DID include the packages dir & the packages dir is a notable problem

As @krenyen pointed out, the packages dir has a... quixotic management practice. It needs to die. If we merge it into the main repo, then it will not die nicely -- it will haunt us forever, floating around with its 20 revisions of selenium.jar clogging up everyone's harddrive.

3) the svn repository also included the CMS directories. It did not include drupal 6 which could not co-exist with drupal7. However, I don't think that will apply to drupal 8 for CMS's other than drupal 6 having the CMS files in the main repo works fine.

We can only speculate about whether D8/Backdrop/D7 will coexist (because they're moving targets). We know that D6/D7 cannot, and we know that the old svn structure was uninstallable for Joomla and WordPress. The old svn structure really only worked well for one group at any point in time. (For 4.2/4.1/4.0, the anointed group was D7 developers. For 2.x-3.x, it was D6 developers.)

No one has put any time into it yet, but I believe that splitting out the CMS modules gives us much better prospects for creating easy deployment solutions that work on virgin copies of all CMS's (WP/J/D6/D7/D8/Backdrop).

4) we've talked previously about including DAOs in the repo & it has been agreed they should be included (although it hasn't yet happened)

+1, #AOL, YES! +1 ++Good

5) However, the problem is not just missing files but additional files scatterered around the place. How big a problem is this? It's quite a big problem in terms of managing getting them out of a deployable git. Do they need to be removed? I don't know. I have been hesitant to leave them in if others are not doing so in case there is a security problem with them. I don't have any other problems with leaving them in & if others were making the same choice I probably would. Removing the tests dir is a real pain because about 80% of my commits involve a change to tests as well as core & I either have to commit the 2 separately or deal with cherry-picking problems

I've missed something. What files are scattered? Why are you removing the tests dir? Why would you want to do development-activities (like cherry-picking and merging) on a source tree that doesn't have any tests -- how can one QA the result?

If it's a security a concern, then one can easily block access at the httpd level, or one can script it. Trivially:

#!/bin/bash
git checkout master
rm -rf tools tests xml
# Note: Each time you run, the unwanted files will get recreated ('git checkout') and redestroyed ('rm').
# The files endure for ... half a second? But who cares? If that's a problem, spend 5 minutes
# writing a better bash script.


7) Alternatively, in my perfect world, you would be able to paste a link to a proposed commit into JIRA & there would be an automated process that a) tests it - like with drupal & b)generates a PR based on it - if it will apply smoothly, In this flow it wouldn't matter if you didn't use civicrm-core as your base as you would be effectively submitting a patch (although you would need to be careful to patch with the core structure in mind)

That sounds cool. Some tools that integrate JIRA/Github/tests into a nicer workflow could be quite handy.

if multiple people are, like us, coming to the conclusion "I came to the conclusion that we couldn't use Github as a source and had to use the packages created by the scripts in https://github.com/civicrm/civicrm-core/tree/master/distmaker.  "

If folks have issues/pain-points with checking out code from github.com (like DAOs and unwanted tests), then let's work through them. I think we can find solutions that address those issues, that work with multiple CMSs, and that are compatible with the powerful build tools written outside our community.

Why are you removing the tests dir? Why would you want to do development-activities (like cherry-picking and merging) on a source tree that doesn't have any tests -- how can one QA the result?
I missed this one - I do development / bug-fixes on our development copy of a customer's site - on the relevant server - unless it's api related in which case I usually do it against core & backport it  (I generally write the tests first when I do api fixes so I generally do api work 'headless'). I do git work related to our repo on that site & then switch back to my local machine to do any work on putting the code into core. (I run my lts repo entirely headless).

I don't consider the sample dataset/ config to be complex enough for much of the work I do.