CiviCRM Community Forums (archive)

Hello Developers,

I am Siddharth Gupta, a 3rd year undergraduate student at IIIT Delhi, India. I am interested in working for Data Visualization for civiCRM which will help organization in better analysis of data. I have a small experience with D3 which is a data visualization javascript library.

I have previously worked with a student organization and have an idea about data visualization requirements. At that time, I was not very well aware of civiCRM but I think that knowledge will definitely help me in choosing generic dimensions for visualization.

I primarily want to work on the following two Ready to Use Data Visualizations:

• Event Registration Visualization: Data Visualization of Registrations for events. Some features I can think of right now are:
  
  • Being able to see how registration evolve over time.
  • Gender based data of the people who register.
  • Beign able to see which agegroup register faster for the event. etc.
• Contribution to Visualization of Donors Trend: There is already some work done on donor trends  at https://github.com/leez/org.eff.donortrends. I would like to work further on it and will be visualizing the same for better planning for the organizations.

I will be requiring the community's help and feedback for these prototype. Please suggest anything else I should work upon and include here. I am really interested t know anything you came across which need a better ready to use visualization for better analysis.

Thanks and Regards,
Siddharth Gupta
New Delhi, India

Hi Siddharth,

This sounds like an excellent project.

My advice is, if you build something that's quite a generic system then more people are likely to use it, polish it, and adapt it.

For example, you could build something that gets a count of distinct values for Individuals' data, the count of contacts with this, and display it as a pie chart. Allow the user to select which dataset is chosen. Then someone else might add something to that, that custom fields can also be considered.

You will also find that with a less specific approach, more developers are likely to want to join you...

my 2 cents:

a. considering the deadline for projects is fast approaching, i would do a first quick draft of your project and post it here (similar to siddanth and aditya)

b. I like your starting points, i'd also add xavier's contact history over time and use these 3 as examples to help build a potentially generic and useful platform for other stuff within civicrm (aditya's project includes using d3/crossfilter for mailing reports)

c. I would keep things simple and concrete and avoid making things too generic. I'd let the examples drive the framework once you have a better idea of how to interact with the DB etc.

lobo

Thanks a lot for your feedback. This is the initial plan I came up with for the project. This takes care of making everything generic so that users can modify it as and when required. Also, I tried my best to incorporate community feedback and suggestions all through the time-line.

• Week 1-2: Research and Learning weeks:
  1: Work on DC and get a good idea about it.
  2: Use civisualize and look for what data needs visualization.
  3: Learn different ways of extracting data using REST API and/or Direct Database Queries
  
• Week 3: Data Design and Extraction:
  1. Finalize on High Level Specific Data.
  2. Create module to extract data.
  3. Get feedback from the community on data specialization.
  
• Week 4-6: Data Visualization
  1. Blog post on civicrm.org summarizing the findings of research and the specification.
  2. Incorporate DC (http://nickqizhu.github.io/dc.js/) used in civisualize for data visualization.
  3. Take feedback.
  4. Bug fixes and improvement according to feedback.
  
• Week 7-9: Integration with already available extensions.
  1. Integrate with civisualize.
  2. Incorporate visualization with other Donor Trends (https://github.com/leez/org.eff.donortrends)
  3. Provide customizable modules for advanced users.
  
• Week 10: Integration with civiCRM
  1. Documentation.
  2. Integrate with civiCRM to make it Ready-To-Use.
  3. A blog post with details of Integration.
  
• Week 11: Testing and Deploying
  1. Testing and bug fixes.
  2. Feedback from community for further work.
  3. Merge with CiviCRM master.
  
• Week 12: Project Wrap up
  1. Project wrap up and buffer week.
  2. Final blog post which will include further possibilities.
  

Please provide your valuable feedback and comments about the same.

Hi Siddarth,

Great project.

Gender based data of the people who register.

A demographic report that that could be used for any group contacts (rather than just event participants) might be a nice project, looking at gender, age and location, donation averages etc...

Event Registration Visualization

Have a look at  this http://demo.webdetails.org:8080/pentaho/content/pentaho-cdf-dd/Render?solution=Leukaemia&path=&file=leukaemia.wcdf

It uses a different platform (Pentaho & CDE) but there's ideas you could look to implement. This version is using dummy data but I've mostly implemented this at LLR and can share the code and DB queries.

Contribution to Visualization of Donors Trend:

This would be great.

I've also done reports (in the extended report extension) that will create what is called pivot charts in MS Excel - ie, you can create a table like

Event	Male	Female	Bit-of-both
Swimming Relay	45	80	1
Catch up at Cafe	3	12	8

There are a bunch of variants on that - you basically choose a row header & a column header & whatever filters you want. I had a go with D3 using the report.getrows api and turned out a lovely gender by case type report. However, when I switched the columns to ethnicity it was pretty much unreadable - you can see the pics attached (data is dummy data). Also I made it such that some metadata is available via the api - but not enough I suspect. The api changes I made are all in 4.4.5- & the tpl is here https://gist.github.com/eileenmcnaughton/8660920 (with dataviz) (note hard-coded report instances & rest is generic).

This was all by nature of tinkering the customer mostly wanted the tables & didn't require the graphs & at this point the conversation always turns to X's feelings about pie charts.

However, I'm not too sure if all of that will or won't be terribly helpful. I definitely would love to see d3 as the report engine for the CiviCRM reports but my understanding is that the strength of Dataviz is when it works with raw data & allows you to dynamically filter - as opposed to the sort of pre-filtered pre-prepared data sets I was using.

In terms of integrating d3 into existing reports there are a number of challenges - how do you avoid the report specific hard-coding currently in the open flash chart implementation, how to you make the appearance of the report work with multiple data sets (as per my pic), how do axes get labeled, what configurability do you expose to the report builder, how do you make it work in dashlets. I am not sure if that will be a focus of this project or a different approach will be used.

I hope the output of the GSOC will be a little more than your above link....