Data Visualization Day by Day

[s0014]

Hi, I will be using this thread to post updates regarding my GSOC project Ready to Use Data Visualization. I would like the community to give suggestions and advice so that the project can go into the right direction.

I would like to start by mentioning that I used civicrm-buildkit to create development environment. After facing few issues with php and drupal configurations, I am finally done with that.

[Erik Hommel]

Cool Sid, I will be following the thread :-)

[s0014]

The first day of GSOC coding went into what to start with. The first chat with xavier began with trying out things. I installed civisualize and went through some visualization xavier has already done on civisualize and try to work around some SQL queries using CRMSQL.

The next thing was to playa little with crossfilter and dc. I hope I will be done with the first trial dataviz tomorrow.

[s0014]

The first trial of Contact Dashboard is done. Currently it has two pie charts for type and gender and one chart on date the contact was modified last. Giving a fair detail about how contacts are updated in the recent past. A screenshot for the same can be seen at

http://tinypic.com/r/2m3mq8j/8

Thanks Xavier for the help with the queries. Please suggest what all can be added to the contact visualization.

[xavier]

Hi sid,

that's a good start, but they are obviously plenty of tuning and adjustment between this and the final version.
Could you clarify what are the points we discussed about and you want to focus on tomorrow?

@everyone:

If you want to test, you can download the bleeding edge version on sid's fork at
https://github.com/siddharthg/civisualize
We'll probably will change a fair bit and please assume we'll change the urls and whatever you expect not to change. you've been warned

as you can see, the demo data isn't good enough (eg everything was created at the same date).
Do you have any real database that is "small enough" (eg between 2 and 10k contacts) and that use civievent, civicontribute, civimember and civimail?

Most of my customers have either bigger or don't use enough features. It would be great if you could dump a version, anonymise the data (there is already a program for that) and give it to Sid?

I can help you if needed.

X+

[s0014]

HI Xavier,

Thanks for asking for the dump. I myself planned to ask for one today. Today I went through the database structure and tried to understand the things around civicrm_contact. Most of the things in my dump were null but again things like civievent, civicontribute, civimail and civicrm_contact can together create a nice platform for visualization of data which is really helpful for the users. As discussed the points I will be focusing for the tomorrow are

• A cleaner code with better color codes
• Elastic Graphs
• As discussed we need more filters for the contact, a dump will be helpful
• Getting data from event and contribution

Today, I added one more graph on modification date and converted the existing lineChart to elastic.

@all
I am trying to understand where all contacts are used, if there is something that we really need for visualization. If you have any suggestions regarding any such attribute that can help distinguish contacts. It will be really appreciated. One thing I will be including is preferred language.

[Owen]

Hey Sid,

>I am trying to understand where all contacts are used, if there is something that we really need for visualization. If you have any suggestions regarding any such attribute that can help distinguish contacts. It will be really appreciated. One thing I will be including is preferred language.


"Contact source" might be an interesting one to throw in.

Cheers,

Owen

[s0014]

Hi, most of the time today was spent in reading about crossfilter for dc.js. I wanted to learn how group works and how you can change dimensions label etc.

One thing I found today was that my code gives this JavaScript error in the browser console was

Code: [Select]

Error: Invalid value for <g> attribute transform="null"
This was also present in the Contribute Page of civisualize. I tried fiddling with the code allowed and all I could understand was its an error with pie-charts. I am still unsure what it is about and there is no reference to a particular line. Also, even after this error the code doesn't break anywhere and everything works pretty fine.

Also I looked at the flash graphs on the dashboard. I think replacing them with dc.js graphs is a pretty nice idea as I couldnt notice them earlier in Firefox where Flash is not supported. A lot of people don't like Flash at all.
@Owen
Added Contact Source as suggested. Thanks. You can try installing the extension from https://github.com/siddharthg/civisualize.

[Owen]

@sid

• Have you considered filters? Filtering by group would be make any viz far more powerful.
• Have you considered comparative metrics? As an analyst some of the most  interesting things I can provide is comparing two groups of supporters.
• Replacing openflash would be awesome, it's bad.
• Will definitely install soon, will wait till it's a bit further along as I need a dev to stick it on the test server (Xav, has civivisualize code changed or can I just throw the d3 code into the sub-folder?)

Also I can skype/hangout at some point if you'd think it'd be useful. I'm not a developer but have built quite a few dashboards for Civi data viz stuff in Pentaho that might be useful.

[s0014]

Hi Owen,

Almost all the attributes we currently have in the contact visualization can be filtered. I am reading more about comparative metrics, can you please suggest what specifically are we looking for. As suggested by xavier I will look at OpenFlash Charts once I gather some more information about visualization. What you can help me with is what organisations look for in contacts. I understand source and joining date matters a lot, but is their anything which we can include in demographics that would be interesting.

@All.

After yesterdays update I found the issue that was causing the error. I have updated the d3.v3.js and everything seems to work nicely now. A database dump which uses a lot of features around civicrm_contact will really help in developing the dashboard further.

[xavier]

Have you considered filters? Filtering by group would be make any viz far more powerful.

They are two "filtering" that we can do:
- as a dimension, so you could filter within the page, that what is done on gender for instance, where you can click and see only when the women contact were added and mix it with the other metrics
- as a filter to create a dataviz taking into account only a subset of all the contacts

the first one means you need to group by (dimension 1, dimension 2...) so adding the groups is likely to increase the number of rows by several order of magnitude, so probably not an option.

However, the second one would we useful as well, eg. to analyse only the contacts that "subscribed to the newsletter" or those that are "big donors".

@owen, were you talking about the second type?

@Sid, for the later, there is an id parameter that you can add, eg example.org/civicrm/dataviz/contacts/42 will set the id variable=42 in the template
It needs more work to make it completely generic, Let's chat about it directly.

Have you considered comparative metrics? As an analyst some of the most  interesting things I can provide is comparing two groups of supporters.

having multidimensional charts is a powerful tool to compare and explore visually differences.
However, it doesn't by default compare, just provides you the tool to make it fun and easy to do that comparison.

Moreover (and it might well be outside the scope of the GSoC project) it would be super useful to indicate if a difference is statistically significant.

For instance, on sid's contact dashboard you have a pie of the split by gender and you see they are 53% of women, you can then select to see only the contacts that have been created last year, it adjusts the gender graph and you can see that you have 55% of women.

What is doesn't say is if this 2% is significant or if it's random changes. what it doesn't do either is to make it easy to compare the contacts created this year (48% of women) vs. last year 55%.

I don't have a smart solution to solve that, yet.

X+

[s0014]

Hi xavier,

I understand the second type of visualization. What I am concerned is even in the second type of filtering we need to predefine the filters and their corresponding ids to get that working. Don't you think that will not be generic enough. But if we are planning to go ahead with it, we need to define these subsets which are useful for people.
@Owen, am I on the right track?

[xavier]

Hi,

I think as an initial release to offer to filter by group would be enough.

@owen, what you had in mind?

[Eileen]

I feel like for the second type of filters you might look at the CiviReport data as source data - as that already has filters etc - but, agree with X that you might want to focus on the initial release before thinking too far on that.

[Owen]

What you can help me with is what organisations look for in contacts. I understand source and joining date matters a lot, but is their anything which we can include in demographics that would be interesting.

• Geography is something that would be interesting with age/gender/contact type. Counting the contacts per city or county (though maybe country for some orgs).
• Population pyramids are a nifty way of displaying more information in the same amount of space as it allows you to see the relationship between age and gender rather than viewing each in isolation, but not sure how this would work with crossfilter.

@owen, what you had in mind?

• Yep the 2nd, filtering subsets using groups. Filtering by dimension would be more useful in contribution/event reports.

having multidimensional charts is a powerful tool to compare and explore visually differences.
However, it doesn't by default compare, just provides you the tool to make it fun and easy to do that comparison.

• The two ways I've considered this previously (but never built). Would be to have a dashboard with down the middle and two filters, allowing you to see both side by side or plotting two groups on the same charts.
• For more time series based viz then the ability to compare different periods E.G Year on year