CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Google Summer of Code »
  • Data Visualization Day by Day
Pages: 1 [2] 3

Author Topic: Data Visualization Day by Day  (Read 5162 times)

Owen

  • I post occasionally
  • **
  • Posts: 83
  • Karma: 2
    • Leukaemia & Lymphoma Research
  • CiviCRM version: 4.3.4
  • CMS version: Drupal 7
  • MySQL version: -
  • PHP version: -
Re: Data Visualization Day by Day
May 25, 2014, 07:35:46 am
Quote
Moreover (and it might well be outside the scope of the GSoC project) it would be super useful to indicate if a difference is statistically significant.

For instance, on sid's contact dashboard you have a pie of the split by gender and you see they are 53% of women, you can then select to see only the contacts that have been created last year, it adjusts the gender graph and you can see that you have 55% of women.

What is doesn't say is if this 2% is significant or if it's random changes. what it doesn't do either is to make it easy to compare the contacts created this year (48% of women) vs. last year 55%.

I don't have a smart solution to solve that, yet.

Might require two defined groups rather more open exploratory crossfilter. Z-scores are interesting for subsets and groups too.

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
May 25, 2014, 02:06:03 pm
Quote from: Owen on May 25, 2014, 07:31:58 am
  • For more time series based viz then the ability to compare different periods E.G Year on year

I will surely try adding that. Though a timeline for the same solves the problem as well? Though fails to give a comparison. With the current data I have I won't be able to try it actually.

@all
Update: I sent a PR to the main civisualize code and now you can try the contacts dashboard at https://github.com/TechToThePeople/civisualize.
  • The updates with that is I changed the source pie-chart to a Bar Graph.
  • Added a overview of how many contacts are you considering in current visualization out of the total

@Eileen
I think (as read in Civisualize readme) that can be done using {crmReport ...}. Is this available in the developers version? I will surely look into what all we can take from the report.
Sid
DataViz Project, GSoC 2014

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: Data Visualization Day by Day
May 25, 2014, 02:52:03 pm
I theory you can render any civireport using the  {crmReport ...} - here's an old attempt of mine - it uses {crmAPI} but I think {crmReport} wraps that https://gist.github.com/eileenmcnaughton/8660920

The difficulty is that there are loads of reports available in CiviCRM. I tried using output from https://github.com/eileenmcnaughton/net.ourpowerbase.reports.advancedfundraising & also from a report in https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport

I'm not sure how to best encourage you here. I would love to see people able to render existing CiviCRM reports using d3 graphs on dashlets BUT I'm not sure to what extent that's a diversion. I would encourage you to start by just understanding a bit about the existing reports in CiviCRM (possibly just explore them through the UI at this stage).
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
May 26, 2014, 11:46:41 am
HI Eileen, today I spent most of the time in understanding how extensions work so that I could have a better understanding of extracting data. I read the whole code of civisualize and tried to gather as much understanding of the code as possible. This was basically to understand how SQL queries were being executed, API was called and templates were rendered.

I tried the gist you have given but wasn't able to run it properly. I will try that again though.

Also as suggested by you I tried looking at the reports. What I could get was most of these reports are  just there in tabular data or openflashcharts. If we even plan on working (rendering reports in dashlet using d3/dc) we would be needing to work with the crm code instead of the extension. Am I right about it? If thats the case, I think a better way for me to go ahead would be to learn more aspects of civicrm just apart from contacts(which I am currently doing) and them move to the core code.
Sid
DataViz Project, GSoC 2014

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: Data Visualization Day by Day
May 26, 2014, 01:59:25 pm
Sid, It's certainly the intention to replace the Open Flash charts in the built in reports (& extensions based on them) with d3 charts & obviously whatever you can contribute to that is great. However, I think that is not the full potential of d3 charts - so it might not be your focus.

Yes, I agree that using d3 with the existing reporting framework involves patching core rather than adding code to CiviVisualise.

Re the gist - it is hard-coded to pull from reports 51, 52 etc - these reports are not terribly typical in that they return a table that is already aggregated. However, if you create other report instances you should be able to load the data from them in that way
{assign var='instance_ids' value=','|explode:"51,52,59,54,55"}

But yes, if you think you should learn the other aspects of CiviCRM go ahead - we are pasting ideas on this thread but you should only take them on board as much as they are helpful to your project because you are the one who is responsible for pulling it together into something coherent
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

Erik Hommel

  • Forum Godess / God
  • I live on this forum
  • *****
  • Posts: 1773
  • Karma: 59
    • EE-atWork
  • CiviCRM version: all sorts
  • CMS version: Drupal
  • MySQL version: Ubuntu's latest LTS version
  • PHP version: Ubuntu's latest LTS version
Re: Data Visualization Day by Day
May 27, 2014, 09:28:08 am
Webinar recording to be found here : https://www.youtube.com/watch?v=76NfSoQHDgM
Consultant/project manager at EEatWork and CiviCooP (http://www.civicoop.org/)

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
May 27, 2014, 09:34:11 am
Thanks Erik :)
Sid
DataViz Project, GSoC 2014

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
May 28, 2014, 01:49:20 pm
Today the work was primarily focused on events dataviz. We are trying to build up the events page in such a way that a lot of things involving events can be achieved just using the events page and an event page. See the difference would be the scope. Most probably events will have all the events together and event will be specific to an event and its participants.

for events page the SQL query I could come up with currently was :-
Code: [Select]
SELECT event.id, event.title, COUNT(*),participant.status_id, fee_amount, SUM(fee_amount) from civicrm_event as event INNER JOIN civicrm_participant as participant ON event.id=participant.event_id group by event.id, status_id;
We will use more attributes as and when required to add graphs and charts. Anything you guys want to suggest regarding the events would be great. I couldn't do much today but lets hope that we will have a page ready by tomorrow.

Thanks. Lots of appreciation for xavier and erik for the webinar. Surely boosted my motivation a lot. 
Sid
DataViz Project, GSoC 2014

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: Data Visualization Day by Day
May 28, 2014, 03:02:22 pm
Looking at your query highlights that you are
1) not currently excluding deleted contacts &
2) not applying ACLs to what is viewed - doing #2 is likely to add the sql for #1

Not sure you should tackle adding that sql just yet - maybe just make a note of it & ping me later when you are up to looking at that
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
May 29, 2014, 02:03:44 pm
Hi Eileen,

I plan to apply ACLs at a later stage, once I get proper visualizations. Can you please suggest me what is ideally the number of events an average organization has?

At the same time, what do you think is more important information, number of events of particular kind or number of participants for event of a particular type. I can even put both.

Also, do you have any dataset which uses a lot of features and is almost complete and can be encrypted and given to me. I am really looking for such kind of a dump so as to get a better idea than what the current sample-data is giving me. :(

Thanks
Sid
DataViz Project, GSoC 2014

Owen

  • I post occasionally
  • **
  • Posts: 83
  • Karma: 2
    • Leukaemia & Lymphoma Research
  • CiviCRM version: 4.3.4
  • CMS version: Drupal 7
  • MySQL version: -
  • PHP version: -
Re: Data Visualization Day by Day
May 30, 2014, 03:15:11 am
Quote
Can you please suggest me what is ideally the number of events an average organization has?

This will vary quite a bit, your query brings back 2303 events for our org. I suspect we'd be toward the higher end as we do a lots of sports events. You might want to consider filtering by event date and/or event name.

Quote
At the same time, what do you think is more important information, number of events of particular kind or number of participants for event of a particular type. I can even put both.

For us participants, reporting on events is often for us seeing how an event is doing to compared to the same event last year.

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Data Visualization Day by Day
May 30, 2014, 08:43:01 am
How do you define the same event last year? Is this based on the type? or simply being able to choose two different event ids?
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Owen

  • I post occasionally
  • **
  • Posts: 83
  • Karma: 2
    • Leukaemia & Lymphoma Research
  • CiviCRM version: 4.3.4
  • CMS version: Drupal 7
  • MySQL version: -
  • PHP version: -
Re: Data Visualization Day by Day
May 30, 2014, 09:00:15 am
Quote
or simply being able to choose two different event ids?

Two different event ids.

@Sid

Here's one of our event dashboards, the production version has a few tweaks but this is what we use. A lot of this is specific to our event but these are the kind of metrics and dimensions that we look at for a fundraising event (NB lots of orgs events won't be fundraising events)

http://demo.webdetails.org:8080/pentaho/content/pentaho-cdf-dd/Render?solution=Leukaemia&path=&file=leukaemia.wcdf

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
May 30, 2014, 01:40:17 pm
Hi Owen,

Thanks for the info. I think comparing two events is a great idea and I will definitely try to include it in some way or the other.

The reason I wanted the number of average events is I have decided certain parameters for events if we look at organization's all the event.
  • Per Event Type Contribution
  • Contribution Depending of the duration of event
  • Event Over the years. A kind of graph showing the contribution over the year collectively. This can be based on contribution date or event dat. People can see in a bar graph that this year events in may raised this much of money.
  • Few other filters which will end up with a list of 5-6 filters

Further filtering and visualization can be decided once this list of 5-6 events is achieved.
Sid
DataViz Project, GSoC 2014

s0014

  • I post occasionally
  • **
  • Posts: 56
  • Karma: 2
  • CiviCRM version: 4.4.5
  • CMS version: Drupal 7.28
  • MySQL version: 5.5.37
  • PHP version: 5.5.9
Re: Data Visualization Day by Day
June 02, 2014, 09:06:43 am
Hi,

The events page now has event count based on type, duration of the event, and contribution. I have also added the number of participants as one filter. I am trying to make a pie chart where there can be slabs like 100-200, 200-300 etc. I am not sure whether this will really help as these slabs will be dynamic. Tell me if we can define such number or should i use a division model, where i divide the maximum number and make ranges.

Also, xavier can a database be arranged. My dataset has just three events and I am not able to get a proper visualization of whatever I am trying.
Sid
DataViz Project, GSoC 2014

Pages: 1 [2] 3
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Google Summer of Code »
  • Data Visualization Day by Day

This forum was archived on 2017-11-26.