Case for an Official Minimized Version of CiviCRM for Packaging on Drupal.org

[kreynen]

In March, the team that manages Drupal.org announced that packaged distributions (Commerce, Commons, OpenAtrium, OpenPublish, OpenChurch, etc) could now included external packages in their downloads.  These packages had to meet certain criteria like being a GPL compatible license and being < 10MB.

http://drupal.org/project/drupalorg_whitelist

CiviCRM 4.12 is currently at 11MB.  To show why the CiviCRM team should invest the additional time it would take to maintain a download of CiviCRM that excludes some of the libraries that can be added separately with Drupal, I created a mirror of 4.12 on GitHub and requested the mirror be whitelisted.

http://drupal.org/node/1588192

I then included that version of CiviCRM with the collection of other modules as moderate level of a Community Media Starter Kit.

http://drupal.org/project/cm_starterkit_moderate

I'm still waiting on several of the packages I removed to be whitelisted so I can re-add them.  Once the URL for downloading a package is approved, it can be added to the drupal-org.make....

http://drupalcode.org/project/cm_starterkit_moderate.git/blob/refs/heads/7.x-1.x:/drupal-org.make#l5

I haven't tried using a directory within another download as the target for a download, so I'm not sure if the packages will be able to be placed into modules/civicrm/packages or I'll have to use sites/all/libraries and process a patch in the packaging.

This is still a really just a proof of concept to show the types of distributions possible with a CiviCRM download that is < 10MB.  If the CiviCRM team decides to support this, there are obviously several ways to get a download < 10MB.

[Hershel]

Have you see http://forum.civicrm.org/index.php?topic=23535.0 ?

[xavier]

Is agpl (civi licence) ok?

[kreynen]

@xavier Yes.  The only issue is the size.  If you download http://drupal.org/project/cm_starterkit_moderate you'd see it includes both the most recent core of Drupal and CiviCRM from the minimized mirror.

@Hershel I didn't see @donquixote's post, but I'll cross post there too.

This is really an alternative to achieving the same thing @donquixote was after.  For a drush user, the process of installing Drupal and CiviCRM with the community media modules is...

git clone --recursive --branch 7.x-1.x http://git.drupal.org/project/cm_starterkit_moderate.git
cd cm_starterkit_moderate
drush make cm_starterkit_moderate

You have to use git or some other process to get the .make file, but once you have that drush downloads everything else from Drupal.org or the download location of the the third party package.  While the whitelist requires downloads to be < 10MB, drush make does not.

[kreynen]

A minimized download that is <10MB does NOT solve the problem @donquixote was trying to address of wanting to add CiviCRM to an existing Drupal install with drush, but I don't see how moving most of civicrm to sites/all/libraries would accomplish that either.

I can see the avantages of moving the Drupal modules in CiviCRM back to a Drupal.org project, but that is probably a discussion best left for after CiviCRM's move to git.  Having both Drupal and CiviCRM in git would make it MUCH easier to maintain a variation of CiviCRM as 2 downloads (a library download from CiviCRM and modules available on Drupal.org). The modules could work like so many other Drupal modules with dependancies on 3rd party libraries. If you simply drush dl civicrm/drush en civicrm, you would see a warning if civicrm wasn't found in your sites libraries.  If you updated the modules independently of the CiviCRM library to the point they were no longer compatible, the module could display a warning with instructions for updating the library.

While this approach has many advantages including allowing CiviCRM to take advantage of Drupal's update status checks AND allowing drush users take advantage of everything drush offers, it would require more work from the CiviCRM team to maintain issues in both Jira and Drupal.org.  Honestly, I don't see that happening any time soon.

What I'm proposing instead is to have an official, minimized version of CiviCRM that excludes libraries that can be re-added using drush make.  Moving these packages to Drupal's libraries would require a patch to CiviCRM.  That patch could be applied during the drush make processing.  3rd party libraries CAN be patched with a .patch from Drupal.org.  This example shows how CKEditor's examples are removed by several distributions by adding libraries[ckeditor][patch][] = "http://drupal.org/files/1337004-ckeditor-remove-samples-3.patch" to the make file.  Drush not only downloads, ckeditor... it changes it in both the distribution downloads AND for users running drush make locally.

[xavier]

So to solve the size issue, the solution is to split it and put the libraries (the package folder) into separate libraries/dependencies and it would solve the problem?

Obviously, civi is tested with specific versions of each of these, is this possible for instance to specify which version of jquery & jquery plugins is required? beside, there is a bit of changes so that's loaded in a separate jquery (noconflict) with a different name (cj instead of $). Is this something that can be done?

Not familiar with these packaged distributions. If the aim is to have a bundle of drupal+"all the good stuff ngos should have") wouldn't it make sense to have one that includes webform_civicrm for instance?

otherwise, would it be already possible to have a drush make that downloads from sf (and so bypass the 10Mb limit)?

donquixote reason to split into core & library was mostly to work around the licence issue (ie. to have a focussed drupal module on drupal.org and the rest as a separate "library" in agpl). The CMS specific parts (either D6/D7/J! or WP) are anyway tiny compared to the CMS agnostic part, that can be seen as an external library from the CMS point of view.

X+
X+

[kreynen]

would it be already possible to have a drush make that downloads from sf (and so bypass the 10Mb limit)?

Yes.  Pulling libraries from external sources has been supported since Drush make was written.

wouldn't it make sense to have one that includes webform_civicrm for instance?

The modules included in the moderate starter kit have been defined by the community media stations themselves.  Many of these groups are my clients, but not all. 

http://groups.drupal.org/node/214333

If someone requests civicrm_webform and the other stations think it would be useful and not confusing, I'd gladly add it.  I've added other civi related modules we've developed to improve the integration between Drupal and CiviCRM for anyone using this distribution.

The Community Media Starter Kit is NOT a basic Drupal + CiviCRM configuration.  It is just the first to include CiviCRM in the download.  I'm doing this to scratch my own itch, but now that the minimized version of CiviCRM has been whitelisted it can be included in ANY distribution.  You could add feature requests to any of the distributions listed here...

http://drupal.org/project/distributions

Open Outreach, OpenAid, and Open Church seem obvious, but my guess is that the maintain will reject that request because of the size and complexity of CiviCRM.  We didn't even include it in http://drupal.org/project/cm_starterkit_easy.

*Someone* could create a basic Drupal + CiviCRM distribution, but CIVICM IS STILL BROKEN IN THE COMMUNITY MEDIA STARTER KIT.

Before starting another distribution that uses the minimized mirror, I want to have this discussion and see if maintaining an official minimized download was something the CiviCRM team would consider.

So to solve the size issue, the solution is to split it and put the libraries (the package folder) into separate libraries/dependencies and it would solve the problem?

You can see what I removed to get < 10MB on GitHub https://github.com/kreynen/civicrm-min/commits/master

Unfortunately TCPDF is also > 10MB before the removing the doc, examples, images and most of fonts like CiviCRM does so the request to whitelist that package was rejected.

http://drupal.org/node/1589206

Moving ALL the packages to a sperate download WOULD solve the 10MB packaging issue and maybe that is the best option, but I was looking at specific packages that were both large, could be whitelisted on their own, AND were used by other Drupal modules... so there would be a reason to move them into sites/all/libraries and patch CiviCRM to use that version.  It never occurred to be that CiviCRM itself was basically a library.

[xavier]

Yes.  Pulling libraries from external sources has been supported since Drush make was written.

Thx...and yet another stuff to study to add to my todo

The Community Media Starter Kit is NOT a basic Drupal + CiviCRM configuration.  It is just the first to include CiviCRM in the download.  I'm doing this to scratch my own itch, but now that the minimized version of CiviCRM has been whitelisted it can be included in ANY distribution.  You could add feature requests to any of the distributions listed here...

Thx for the clarification

Before starting another distribution that uses the minimized mirror, I want to have this discussion and see if maintaining an official minimized download was something the CiviCRM team would consider.

If you look at the distmaker, you have the scripts to generate the various distributions (D6/D7/J!/WP). Could you make the minification work part of it and generated automatically at the same time as the others or do you think it would need manual work at every release?

It never occurred to be that CiviCRM itself was basically a library.

Matter or point of view, everything is relative
If I understand the drupal structure properly, "libraries" is basically what isn't a drupal module but used by it.

[kreynen]

If I understand the drupal structure properly, "libraries" is basically what isn't a drupal module but used by it.

 

Libraries in modules have always been an issue.  I argued for weeks to get permission to include the TinyMCE javascript with module back in 2007 (http://lists.drupal.org/pipermail/development/2007-May/023997.html).  The version control team stuck to their policies in that case... and ultimately it was the right call, but it took years to get to this point.  The library issue really came to a head with JQuery UI plugins.  If a module depended on a specific version of a plugin, it could be included... but when several modules added different versions of the same plugin they conflicted.

The Libraries API module (http://drupal.org/project/libraries) has resolved most of this, but there are still issues modules that require users to download a package and place it within the module.

One easy reason this is wrong is that if you download a new version of the module, you either need to copy the library over form the previous version, download (and likely uncompress) it again, or replace only the files within the module. 

If you look at the distmaker, you have the scripts to generate the various distributions

Looking at http://svn.civicrm.org/civicrm/trunk/distmaker/dists/drupal_php5.sh

If someone with permission and more shell scripting experience than I have added something like drupal_min_php5.sh with just

# delete packages that distributions on Drupal.org repalce if present

rm -rf $TRG/packages/dompdf
rm -rf $TRG/packages/IDS
rm -rf $TRG/packages/jquery
rm -rf $TRG/packages/ckeditor
rm -rf $TRG/packages/tinymce

That should be enough to get < 10MB and all of those can already (or will shortly) be able to be readded either back to civicrm/packages OR to sites/all/libraries.

Not sure which direction to go with that, but I'm leaning towards putting CiviCRM back together in civicrm/packages initially. 

Getting an official minimized version would be a HUGE step forward, but would require addressing this...

http://drupal.org/node/1588192#comment-6013358

[Donald Lobo]

We can potentially do this for the D7 version of 4.2

Ideally we can push to sourceforge but not publish it to the enduser (i.e. u can get it if u know the URL) at least initially. Reasoning being, downloading the package by itself  will not work unless done in conjunction with other stuff. Not sure if this is possible via SF today.

lobo

[kreynen]

Thanks!  That's all we'd need.  I'll continue updating the mirror until there is an official option.  I'll also continue getting the packages that have been removed, whitelisted and re-added using http://drupal.org/project/cm_starterkit_moderate as test site.

Is creating a "standard" Drupal + CiviCRM distribution something the CiviCRM team wants to maintain and/or support on Drupal.org?

[Donald Lobo]

Is creating a "standard" Drupal + CiviCRM distribution something the CiviCRM team wants to maintain and/or support on Drupal.org?

Yes definitely. Ideally we'd like someone (like you!) to maintain/support it on drupal.org and we can do the part of hosting and deploying minified versions to make the task easier?

Would you be interested in doing the above? maybe a drupal + civi + views + webform + webform_civicrm module?

lobo

[kreynen]

Started.  http://drupal.org/project/civicrm_starterkit

I'll commit in a few hours.  Initially the download will only include the .make and .profile files.  After the packaging script runs again, it will process the drupal-org.make the same way a user running drush make would locally.  The packaging script can take as long as 24 hours to update the downloadable dev snapshot.

I've added lobo and dggreenberg as maintainers as well.  I know you guys aren't very active on the Drupal side, but in case I get hit by a bus... someone else has the keys. 

When the official, minimized download is available I'll make the request to change the whitelist record.  Once that is done, I'll update all the distributions I'm involved in.

[Donald Lobo]

Kevin:

can u please check and review this issue:

http://issues.civicrm.org/jira/browse/CRM-10299

I've committed code which matches the issue. Let me know if we need to make any changes to it

lobo

[kreynen]

The Drupal.org whitelist use regular expressions to limit which urls a file can be downloaded from.  I don't think the infrastructure team will whitelist anything on http://downloads.civicrm.org/ so the whitelist record would limited to http://downloads.civicrm.org/civicrm-***.starterkit.tgz

So as long as you follow that pattern on the CiviCRM side, everything should work for future releases.