GSOC - Visualization for Mailing stats and A/B testing

[aditya.nambiar]

Hi, I'm Aditya Nambiar a second year undergraduate student studying CSE at IIT Bombay . There was a seminar held at   IITB regarding GSOC. I came across CiviCRM and heard that they have a core team based in Bombay. I met Kurund Jalmi and Donald Lobo core team members of CiviCRM to discuss about Visualization for Mailing stats and A/B testing for CiviMail.

CiviMail is an extremely important component of CiviCRM. A/B testing is a relatively recent, highly effective, cost-efficient strategy for improving the conversion rate, and since CiviCRM is specifically designed for the needs of NGO and advocacy groups, and serves as an association management system it is also a must for it.
CiviCRM lacks this feature and it is essential for any mail blasting application. Also it is a commonly requested feature on the community.

Here is my plan:

Week 1-2: Research weeks:
1: Get up to speed on d3 ( and may be filterjs )
2: Get up to speed on civi mail functionality
3: Learn civicrm REST API.

Week 3: Design and Usability mock ups
1.Create wire frame for A/B testing and reporting.
2.Create specifications at a high level.

Week 4-6:
1. Blog post on civicrm.org summarizing findings of research and  the specification
2. Work on reporting.
3. Incorporate D3 (and FilterJS) into Civi reporting framework.
   
Week 7-9:
1. Bug fixes and improvements for reporting
2. Deploy reporting on demo instance
3. A blog post with details of the new version and requests of feedback from the community.
4. Implement an A/B testing screens and functionality.

Week 10:
1. Document work done.
2. Integrate with CiviMail-blast UI project.
3. Incorporate visualization in other CiviCRM reports.

Week 11:
1. Testing and bug fixes.
2. Merge into CiviCRM master.

Week 12:
1. Project wrap up and buffer week.
2. Final blog post

Any feedback and advice would be highly appreciated.

Thanks
Aditya

[xavier]

Hi,

Awesome! Some comments:

1) A/B testing isn't only useful/needed for mailing but can come super handy for pages as well. eg what's the impact of suggesting 5 10 and 20 USD instead of 3 7 and 15 in a contribution page? smaller % of conversion, but higher average amount?

It would be great if some of your work could be reused in both A/B testing for mailing and pages.

2) there is already a dataviz framework that is probably a better starting point than reports (it can benefit from them too to fetch data if needed)
http://github.com/TechToThePeople/civisualize
it's using dc, a mix of d3 and crossfilter that is amazingly cool

3) A/B testing on mailing is a slightly different beast than the "normal" A/B testing on visitors, because you know already the complete population (eg all the subscribers to the newsletter).

You'll probably have to take samples (eg out of the 10k subscribers to the newsletter, I'm going to sent version A to 1k and version B to a different 1k), avoiding sampling bias might be non trivial.

4) You reports should help explaining the math being and avoid incorrect readings (our users are probably way less into stats than you'd hope)

ie. you should be clear that a difference between A at 45 and B at 53 shouldn't be that B is x% better, but that the result is meaningless because the sample size is too small and give the confidence interval.

and the mandatory dilbert. http://dilbert.com/fast/2014-03-08/

5) To be helpful, the results should be actionable. Eg. once you have showed that the newsletter in blue get more clicks/better opening rate than the red version, it should be easy to send it to all of the subscribers that weren't already in the test groups.

6) It could be great to be able to choose what you want to optimise in the A/B testing: opening rate? click through? number of contributions generated through the mailing?

7) if you want to optimise for complex actions (eg does it generates a bigger amount of contributions or more registration to the event) it could be interesting to make it easy to integrate with google analytics rather than re-inventing the tracking wheel (and google might like it too .

Ok, f**king long list, not everything can be tackled in a single GSoC, take all of the points as suggestions.

This project could be split into at least two loosely correlated parts. Do you know another student that might be interested, you split the work between you and go for two projects?

and here please allow me to one project I'd like to mentor but that is looking for a student
http://wiki.civicrm.org/confluence/display/CRM/Google+Summer+of+Code+-+2014#GoogleSummerofCode-2014-Datavizframework,Geotoolsandchoropleths

Having a proper dataviz is a project by itself. Eg it would be awesome to not only visualise if A works better than B, but be able to have a map to see that A works better for contacts on the east cost, with a pie chart to see it B is more successful among the women and a line chart that shows that A was more successful just after the mailing, but then B took over after 24 hours.

And they are tons of other nice things to visualise better beside A/B testing obviously.

X+

[Donald Lobo]

to clarify:

a. the focus of the project is A/B testing on mailings only (this will be combined with the new mail blast UI)

b. adding data visualization to make the mailing reports more visual and hence easily comparable across multiple axis is a smaller sub-project within the main project

if we are done ahead of schedule, we can always increase scope.Its nice to have a target that can be met

yes, would be great if aditya and sid can recruit a few more CS students from the school

lobo

[Owen]

Would be great if transactional emails could also be considered (Thank you emails, receipts etc..)

I'd be happy to co mentor, it fits my background quite nicely.

[Eileen]

How do you intend to link the data from the CiviMail to the contributions/ other actions? There is an extension here - https://civicrm.org/extensions/extended-mailing-stats that simply looks at contributions made within a certain time period by mail recipients after the mailing & you should probably take a look at that in your R&D phase.

A-B testing has been a heavily requested component. The guys at Wikimedia foundation have done extension A-B testing on their banners & the results are somewhere in the public domain - not sure if that would be a useful thing to link to in your intro?

[xavier]

It's most probably too complex for a GSoC, but

when you click on the mailing, you can set a cookie (session) and track it comes from the mailing. you can then check that at contrib confirmation.

For the google analytics way, you can put on the url two params (so that would be altering the way the tracking url is redirected to) that are tracked by google analytics by the visits script, and set a goal to the thank you url.

X+

[Owen]

Integration with Google Analytics might be an idea to explore.

Google Analytics has an API which could pull back into a mailing reports. Mailchimp has a nice auto append tracking on links feature.

Actions and goals that are in Google Analytics could be used for reporting and/or the measurement for an A/B test. In it's simplest form this could has number setting up a goal in GA like :

"Reached a transaction thankyou page" and has come from a link from the email which contains URL tracking [linktodonationpage]?utm_Source=Civimail&utm_medium=email&utm_campaign=Marchnewsletter

If an org has GA ecommerce tracking implemented this could go down to the sum total of contributions, not sure how many orgs have implemented this though?

N.B Many orgs using Civi will be using Google Analytics, some will be opposed to it in principle. If you consider it a good idea and if it's in scope having this switchable would be a good idea.

[Owen]

It's most probably too complex for a GSoC, but

when you click on the mailing, you can set a cookie (session) and track it comes from the mailing. you can then check that at contrib confirmation.

For the google analytics way, you can put on the url two params (so that would be altering the way the tracking url is redirected to) that are tracked by google analytics by the visits script, and set a goal to the thank you url.

X+

Ha! you wrote that before I finished drafting mine.

[xavier]

Instead of something switchable, get that as an extension and hook it in (or inject the dependence like the cool kinds seems to do these days) the rewrite url function when they are clicked (track url feature).

@Aditya don't panic, if it goes into every direction, it just means you have a lot in the community that are really liking your project. Stay focused, ignore most and you should be fine

[Eileen]

I mentioned this in the other GSOC project. I would be careful about specifying integration with another project in your project. There is risk that one doesn't go ahead or that if they both go ahead one hits unforseeable circumstances. You might mention that you hope to integrate with it in your proposal but I think you need to be careful that it doesn't look like it has to be both projects or neither in case google only wants one to go ahead.

[jaapjansma]

Wow this really sounds awesome! And is probably something we also need for the donor journeys we are setting up with CiviCRM.

About the discussion of integrating with Google Analytics and mailings also take a look at an open source website analytic tool PiWik: http://piwik.org/

[Kurund Jalmi]

wow, so many great ideas.

As of now we are planning to start with minimum scope i.e. implementing A/B testing and improving mailing stats with some visualization. If we are ahead of schedule we can incorporate some idea mentioned here.

Eileen, this project and Mailing UI Blast project are two separate projects and they need not depend on each other. But it's good to clarify and mention each other. For eg:, A/B testing and mailing stats improvements can be done in the current CiviMail workflow, but if change both projects get approved Aditya will have integrate his work with new workflow.

Kurund

[aditya.nambiar]

Hi, please find below my current proposal. Feedback regarding the proposal would be greatly appreciated.

Thanks
Aditya Nambiar


Application for Google Summer Of Code 2014: CIVICRM –MailBlasting

Personal Information

Name : Aditya Nambiar
Email : aditya.nambiar007@gmail.com
University : IIT Bombay
Graduation Date : 2016
Major : Computer Science & Engineering
Location : Mumbai
Timezone : +5:30 GMT
Drupal.org Profile : aditya.nambiar
IRC :
Resume : http://www.cse.iitb.ac.in/~adityan/Resume.pdf
Open Source Experience: I have made a few chrome-plugins.

I am an undergraduate student studying Computer  Science & Engineering at the Indian Institute Of Technology , Bombay. I have been programming since I was 10 years old and have a passion for programming. I am an expert C++ programmer and I am very active in competitive programming. I have participated in several Hackathon competitions held on campus such as those conducted by Yahoo , Microsoft and also  won an Hackathon! I have been involved in the development of several websites such as those of Techfest IIT Bombay (http://www.techfest.org/), which we created using Code Igniter, a PHP framework.
During my time at IIT I have learned several programming languages such as Scheme, Python, Java and Bash and have also done several course related projects the details of which are available in my resume.
Apart from programming, I have a very good academic standing. I have secured an All India Rank of 97 among half a million candidates (top .02%) in IIT-JEE entrance exam to get into the department of my choice. I have received several scholarships such as the KVPY and NTSE from the Govt. Of India. I have also secured several top ranks in several competitive examination  (All India rank 27 in National Cyber Olympiad, International Rank 5 in Maths Olympiad etc).






GSOC Information

If you have participated in Google Summer of Code in the past, please describe your participation:
None.

Have you applied to GSoC in the past and are you applying to any other organizations this year? If so, please explain.
None.

How many hours will you devote to your GSoC project each week? What are your other summer plans (vacation, this time of year isn't your summer and you are in class, etc)?
I am very enthusiastic about my GSOC project. I am ready to devote at least 40 hours a week. I do not have any other summer vacation plans and would like to invest all my time in GSOC.

CIVICRM INFORMATION

Have you registered an account at CiviCRM.org?
Yes, my user name is – aditya.nambiar

Have you even built a site with Drupal, WordPress, or Joomla?  (Please provide details)
No

Have you ever built a CiviCRM site or helped on a CiviCRM project?
No

Have you ever posted a questions to the CiviCRM Forums, JIRA, or GitHub?
Yes, http://forum.civicrm.org/index.php/topic,31984.0.html

Have you ever contributed code to CiviCRM?
No

If you have not previously contributed to any open source projects, can you provide example code from school?

Developed a windows app, which can seamlessly convert normal text into mathematical formulas and enable users to create graphics using Paint like app on the go. I wrote the code to render a canvas, which is a paint like application in the notepad enabling the user to create creative graphics in javascript . This Windows app secured the first place at the Microsoft Hackathon. Github link-  https://github.com/Manishearth/Kapi

Technical Information

Have you ever utilized IRC?
Yes

Have you ever worked with Git?

Yes, I have learnt SVN and Git for version control. I have used Git  for several projects some of them include the classroom notepad app and also a project on creating a complex machine using BOX2D a physics simulation engine.

Question based on knowledge of git:
You just committed some code (and not pushed yet), but you realized there is a typo in commit message. How would you change it (please explain each step of the solution)?

 git  commit –-append –m  “New commit Message”

Question based on given 'almighty_function':
The function will always return “FAIL” since for it to return “Success!” x must be equal to y and x must be equal to z by transitivity y would be equal to z . However in the “if condition” y also must not be equal to z!
One of the 3 conditions will have to be removed for any possibility of returning “Success!”

PROJECT INFORMATION

Which project sparks your interest and why?

A/B testing and visual reporting for CiviMail

About The Project

A/B testing is a form of testing that allows an organization to determine which mail will be more effective when sent to their mailing list. This involves sending the email to smaller subsets and using data and statistics to pick the winning email which will be sent to the rest of the group. A/B testing is standard and recommended practice to improve your results from any mass mailing and has been used by Wikimedia, Obama campaign and others very effectively.

This project aims at creating an A/B testing tool for CiviMail so all users of CiviMail can improve their response rate for all their mailings. To make this easier on the end user this project will also incorporate graphical reporting (such as using charts, graphs etc.) of the mailing statistics. This will allow the admin to interpret and monitor the results of their mailing campaign and hence make a decision based on data.

I plan to use AngularJS, D3JS and CrossFilter ( may be ) in implementing the project.

The admin will have the option of which part of the mail he/she wishes to test: Subject, Date/Time of sending ,or the mailing body. We will also set smart defaults for A/B testing.

Admin will analyze the results and making a decision based on his choice. The interface would be intuitive and easy to use. Also results shall include confidence intervals of how good is the result and how much it could be trusted. This can be computed by using sample size and response rate.


If time permits
•   The app will have the option of choosing what the admin would like to optimize. Options such as Open Rate, Number of clicks on a particular link , number of contributions generated through the mailing etc. could be provided.
•   I would also like to integrate it with Google Analytics so as to make the results more informative and helpful.
•   Two options will be provided for A/B testing, automated and manual. In the automated option the mail which proved to be better among the two sets (using an admin chosen criteria) shall be automatically sent to the remainder set.

Benefits

CiviMail is an extremely important component of CiviCRM.
A/B testing is a highly effective, cost-efficient strategy for improving the conversion rate, and since CiviCRM is specifically designed for the needs of NGO and advocacy groups, and serves as an association management system, A/B testing is a must for it.
CiviCRM lacks this feature and it is essential for any mail blasting application. Also it is a commonly requested feature from the community.

Expected Deliverables:

•   Blog Post on week 4 and 9
•   Setup public demo in week 8
•   Final code on github

Week 1-2: Research weeks
1.   Get up to speed on AngularJS, D3 and CrossFilterJS
2.   Get up to speed on CiviMail functionality
3.   Learn CiviCRM REST API.
4.   Research competitors design and usability

Week 3: Design and Usability mock-ups
1.   Create wire frame for A/B testing and reporting.
2.   Create specifications at a high level.

Week 4-6:
1.   Blog post on civicrm.org summarizing findings of research and also the specifications
2.   Work on reporting.
3.   Incorporate D3JS and CrossFilterJS into Civi reporting framework.
   
Week 7-9:
1.   Bug fixes and improvements for reporting
2.   Deploy reporting on demo instance
3.   A blog post with details of the new version and requests of feedback from the community.
4.   Implement an A/B testing screens and functionality.

Week 10:
1.   Document work done.
2.   Integrate with CiviMail-blast UI project.
3.   Incorporate visualization in other CiviCRM reports.

Week 11:
1.   Testing and bug fixes.
2.   Merge into CiviCRM master.

Week 12:
1.   Project wrap up and buffer week.
2.   Final blog post

Potential Mentors: Kurund Jalmi

Which aspect project idea do you see as the most difficult?

Getting up to speed with AngularJS for completion of a complex project. Getting up to speed with CiviCRM, CiviCRM API.

Which aspect project idea do you see as the easiest?

Weeks 1/2 on Research and UI mockups