CiviCRM Community Forums (archive)

Thinking back to why I originally abandoned the submodules approach (& whether I should go back to it) - apart from the quirks submodules throws up - I hit a couple things, 1) I found it easier to have a d6 & a d7 deployment branch since I needed different versions - at that point it seemed easier just to commit the directory. The other was the 'cleaning' of patches I tend to do - ie. rebasing against new releases - however, looking at github I see only 3 patches on packages & only a handful on https://github.com/fuzionnz/civicrm-drupal7/commits/master -  probably already in core. So, perhaps I need to try the submodules approach again for 4.4.

I haven't used git-submodules much -- only a little bit when using "puppet" (because that's common in the puppet community). Their directory structure is a little different from the (typical/traditional) CiviCRM directory structure -- and I think their structure would be a bit easier for managing long-term forks (due to cleaner "separation of concerns").

Quick comparison. If you wanted a git submodule setup for a "full build", you might try "Example A" in https://docs.google.com/spreadsheet/ccc?key=0AujctkQuBowkdG1QbEpSZFU3N0p4NGRaYTh6VE91TFE&usp=sharing .

In this setup, https://github.com/myuser/civicrm-core.git#4.3 (aka origin/4.3) serves multiple purposes -- it holds the core code; it holds all your patches to core code; AND it holds metadata about each submodule's revision. Note that your long-running fork branch (e.g. origin/4.3) includes submodules for drupal & packages, but the official branch (e.g. upstream/4.3) does not. Now suppose you want to begin work on a new patch (ie make a new branch for future PR). One must choose a base-branch of either origin/4.3 or upstream/4.3), but that's an impossible choice: if you use origin/4.3, then your working copy will include the submodules (making a full/usable source-tree) but will also include all your unrelated/proprietary patches (which prevents you from sending a clean PR). If you use "upstream/4.3", then it can merge cleanly anywhere, but you can't run it locally because it's missing the submodules.

If we emulated the typical usage of "git submodules" from the puppet community (see "Example B" in https://docs.google.com/spreadsheet/ccc?key=0AujctkQuBowkdG1QbEpSZFU3N0p4NGRaYTh6VE91TFE&usp=sharing ), you wouldn't face this impossible choice because the build data is separate from the code patches. I did a little experimenting on my local machine, and I don't think it would be too hard to make "Example B" work. (It requires changes to civicrm.settings.php, civicrm.config.php, and any code that references "packages". If you need the web-based installer, then that might also require patches.)

So what would the day-to-day steps look like?

• A. Pick a feature or bug to work on
  
• A.1. Create branch "origin/my-fix" based on "upstream/4.3"
  
• A.2. Write code and commit to "origin/my-fix"
  
• A.3. Merge "origin/my-fix" into any long-running branches -- switch the order/timing as desired:
  
• A.3.a. Merge "origin/my-fix" into "upstream/4.3" by submitting a pull-request
  
• A.3.b. (Special) Merge "origin/my-fix" into "origin/4.3" via CLI, pull-request, or whatever your group wants.
  B. (Special) Periodically, to make sure that your long-running fork stays aligned with the official code, merge "upstream/4.3" into "origin/4.3".
  

My strong suspicion is that most of the steps are the same regardless of whether you work with civicrm-core.git, civicrm-drupal.git, civicrm-packages.git, or a hypothetical civicrm-everything.git -- and regardless of whether you have a long-running fork. The major variations are A.3.b and B:

• If (like most folks) you don't have a long-running fork, then skip A.3.b + B.
• If you have a long-running fork, then do the git merge for A.3.b + B.
• If you have a long-running fork based on git-submodules, then you also need to update the parent project so that the submodule ref points to the latest HEAD of origin/4.3.
• If you have a long-running fork based on drush-make or composer, then updating the build document (make-file/composer.json/composer.lock) is optional. It depends on whether you choose to include branch-names or commit-names in the build document.
  

TLDR

• The basic mechanics of making a patch/PR should be the same for everyone.
• Branches should be based on the lowest-common-denominator (e.g. upstream/4.3). If you base patches on something else, then you cannot use git merge or PRs!. The only option is cherry-picking in multiple places.
• git-submodules requires a little overhead compared to drush-make/composer
• Some patches (relatively small/managaeble) may be required to core to work more flexibly with different build practices.

ie. - to put this really simply - if the CMS & packages folders were not inside the civicrm folder then the problem would go away?

Yes (give-or-take bug-fixes).

And by extension tools & tests out of the main repo as well.

Yes, I can see how it appears that way with your stated goal, and (from a "build-process" perspective) that's true. I could go along with moving "tools" to its own repo but disagree with moving "tests" (or "xml" -- which is also omitted from tarballs used for deployment). A few key points:

• "tests" and "xml" are tightly coupled to the core code-base, and working on those is integral to the development process. It should be normal, accepted, and expected that a patch touches both the "CRM" or "api" code as well as the "tests" or "xml" code. Splitting those out into separate repos would incur the overhead of coordinating separate patches to separate repos on a day-to-day basis. (By contrast, the CMS and packages directories are things that need coordinated patches for maybe one developer each week, so the overhead is much lower; and the "packages" aspect will resolve itself when we replace "packages" with a sane build-document.)
• On a production deployment, it's pretty easy to remove (or restrict access to) unwanted files with a config-file or script. It might take 3 lines or 30 lines (depending on the exact requirements/goals/process), but either way it's pretty small.
• Looking at other systems like Drupal contrib and Symfony, it's common for unit-tests and schema documents (like "tests"/"xml") to live in the same repo as the corresponding code -- and even get deployed! Why are these problematic for CiviCRM -- and no one else?

I agree with Eileen from a security perspective we should not put 'tests', 'packaging' or any other utility scripts/code in a live, Internet-facing site. However I also see Tim's point about convenience for the developer, and we want to make it very easy to write units tests!

The logical way to solve this is to have a higher-level directory that contains a civicrm (executable code), tests, doc and other such folders. This is common and a lot of components are packaged like this today. You can then point your web root at the civicrm directory and then have both Tim and Eileen happy.

In our case, 'pointing the web root' might mean creating a symlink from the CMS module/plugin/extension folder, or extracting the tarball and copying only the civicrm directory to the CMS directory tree.

So the install instructions for civicrm might be a little more complicated than they are today, but by pushing this theory a little further I think we might even be able to release just one tarball for civicrm, with all the CMS bits included, and then have a couple of 'cp' or 'ln -s' for the install process. This could simplify the GIT repos, development and release processes significantly.

In the node.js world, you can define a prod and test environment that can fetch different extra packages.

Isn't it the same with composer?

A) @xavier and @jaapjansama -- I *think* the analog to Xavier's node.js "environment" would be to differentiate "require" and "require-dev" -- e.g. http://getcomposer.org/doc/04-schema.md#require-dev . A typical example would be the "phpunit" or "Selenium" executable -- you need to install it on a development machine but not on a production machine. When calling composer, you can pass "--dev" or "--no-dev" to indicate whether you want to install dev tools. (The phrase "prod and test environment" might be confusing because refers to something else in Symfony's jargon -- e.g. it refers to having multiple databases running off the same codebase.)

B) There's been a confluence of issues which led me to create a github repo called "civicrm-drupal-project." This is a very small repo that you can clone/fork which defines how to build a CiviCRM environment. The architecture is inspired by "Symfony Standard Edition" (which provides a thin top-level project for downstream developers to clone/fork). The issues motivating this are:

 - The discussion here about managing deployments.
 - A discussion with erawat about how difficult it is to setup testing.
 - A discussion with other folks about setting up their git forks.
 - A pull request from Peter H with a new way to run tests. (His approach downloads+configures Drupal, spawns mysqld on a ramdisk, creates config files, etc.)
 - A discussion with some Windows users about how hard it is to setup civix in their environment.

The "civicrm-drupal-project" basically downloads all things we recommend for Civi development. The difficulty of installation should be roughly on par with installation of civix, but it gets a lot more done (so that the overall process is easier/less frustrating). It's still early/WIP:

 - Notes: https://pad.riseup.net/p/u6aiEzRqP4YN
 - Web page: https://github.com/civicrm/civicrm-drupal-project

Macs run bash.

I don't really use Windows enough to say what would work well for new developers there (and I'm skeptical of whether anything can work well there). Windows devs would still need to setup bash+git+drush+drushmake+AMP (e.g. maybe with Vagrant/VMs, Cygwin, or Acquia Dev Desktop?). But hopefully this would be a net-improvement (e.g. reducing a 20 step process to a 10 step process).