PHP A/B testing framework

[wtheaker]

Has anyone used a PHP A/B testing framework on a contribution page? I was hoping to start out with some basic things, like testing various images, and tracking the results with Piwik (a free replacement for Google Analytics).

I looked into using phpA/B http://phpabtest.com/quick-start, but had issues using PHP in a smarty template. Is it possible to embed PHP code in smarty templates?

I've also heard good things about http://sixpack.seatgeek.com/ since it's language agnostic, but haven't used it.

[Michael McAndrew]

Is it possible to embed PHP code in smarty templates?

Yes. I can't remember if CiviCRM disables it by default, but it is possible.  It's not a brilliant idea though and smarty is designed to stop you from doing that for a few reasons (security, seperation of different layers etc.)

Looking at that phpA/B site, my first thought was is there a smarty integration with phpA/B.  I then thought that it would be cool to have a more generic way to integrate that with CiviCRM since it seems like quite a powerful tool.
i.e. I reckon there is a way to integrate it, but including it at the smarty level is probably not the best way to do it.  Maybe via a call to a hook when the page/form is loading.

I suspect there are other ways to do AB testing on contribution pages that people have experimented with. I'll mention it to people and see if they have ideas to share.

[Owen]

>Has anyone used a PHP A/B testing framework on a contribution page? I was hoping to start out with some basic things, like testing various images, and tracking the results with Piwik (a free replacement for Google Analytics).

Does it need to be into Piwik and not Google Analytics?

Otherwise we use this and I can recommend it, can drop a JS tag in the them and then make tests using wysywyg GUI and not get down into code level.

https://vwo.com/?v=3

Alternatively could you take a step back and randomise the URL to two separate Civi donation forms and track success in Civi?

[JoeMurray]

Important topic. What we really need to support is the reporting, and the feedback into action. Ideally, you want to continue A/B testing until there is statistically significant difference in results as defined by a particular level of confidence, and then switch to the better alternative automatically.

Owen, is there a reasonable way to poll Google Analytics, or do a callback from it?

[Hershel]

Yes, of course Google Analytics provides an API:

https://developers.google.com/analytics/devguides/reporting/

However I can also understand that not everyone wants to use GA (the original poster possibly being such). There are JS-based tools also such as

http://blog.alexmaccaw.com/abba
http://www.optimisationbeacon.com/testing/using-cohorts-js-an-open-source-javascript-ab-mvt-library/

I haven't examined these in enough detail to determine how easy they would be to integrate with CiviCRM, but at least they are pure JS solutions so they might be easier than a PHP-based tool.

[Owen]

>Important topic. What we really need to support is the reporting, and the feedback into action. Ideally, you want to continue A/B testing until there is statistically significant difference in results as defined by a particular level of confidence, and then switch to the better alternative automatically.

Agreed (VWO the tool I posted below uses significance testing https://vwo.com/knowledge/calculating-ctbo)

>Owen, is there a reasonable way to poll Google Analytics, or do a callback from it?

It's got a good API, would just need to make sure the ecommerce tracking data went over in the right way in the first place. There's enough space for variables in the GA system for it but for example you'd likely want Campaign, Contribution form ID, a test ID and a variation ID.

>However I can also understand that not everyone wants to use GA (the original poster possibly being such). There are JS-based tools also such as

GA has understandably come up against some resistance from sectors of Civi community. JS/PHP and built into Civi would be the ultimate but that seems like more dev?


A split URL and a results page in civi compare #of donations or total amount donated seems to me to strike a balance between privacy and ease of implementation.

example.com/civicrm/donateredirect

Sends 50% to /civicrm/contribute/transact?reset=1&id=TEST1
the other to /civicrm/contribute/transact?reset=1&id=TEST2

Within Civi is a page that compares TEST1 to TEST2 and changes the redirect to one of them once significance on pre established criteria has been reached.

[JoeMurray]

Of course Google provides an API. But looking at it a bit suggests there is no callback mechanism, so it would unfortunately have to be polled.

One thing I would expect they could do but haven't checked for lately is statistical significance on an A/B test. This page suggests they don't even have that: http://designmodo.com/ab-testing-google-analytics/

Optimize.ly is a paid service that seems to be a leader in this space.

Would be a great addition to CiviCRM to see this. A/B testing for CiviMail is expected in 4.6. It came out of a GSoC project, and Tim is reworking extensively I believe.

[JoeMurray]

@Owen: I like your thinking on how this could be implemented in terms of a URL.

Perhaps a UX element like "Make an A/B test for this Contribution/Event... Page" could create a copy of the existing page using existing code, and set up other stuff behind the scenes as required.

[Owen]

>Perhaps a UX element like "Make an A/B test for this Contribution/Event... Page" could create a copy of the existing page using existing code, and set up other stuff behind the scenes as required.

I'd normally avoid doing split URL as it's not great for web analytics but in the case of Civi forms it makes a bit more sense and would ensure portability across CMS. Might be able to fit into GSoC code as well?