CiviCRM Community Forums (archive)

Hi guys,

As promised a second post - basically to update you on what I'm doing that could advance stuff for us all.

At our unconference Erik presented some issues in standard civiCRM concerning automating donor journeys. I' list them again for you:
* Not possible to search on 'not member of group' in Advanced Search
* Group search in Search Builder does not work as expected, when combining 'in group' and 'not in group’
* Scheduled Reminders can only be used for mails
* Using activities to distinguish results in a huge list
* No UI to set up Donor Journey
* Households vs. Individuals conflict with workflow for payment driven communication
* Automatic group movement?

I was thinking that several of these issues are covered with functionalities in RapidMiner, data data analytics / data mining software I use, for sure in it's server edition. (have a look at www.rapidminer.com. it is at least to a certain degree open source as well)
* RM has large functionalities to filter records, do advanced calculations to create new attribtution, do some modelling on data... that could be an answer to some of the search issues.
* RM server has the functionality to set up an analytical process (for example defining in what stage of the donor journey a donor is) and set cron jobs to run this process on a regular basis.
*  RM server has the capacity to import data from a CRM system and to send results back to the CRM system, or to send (email) a results list to somebody (a TM-agency for example)
* RM has a nice user interface allowing non programmer like my to design an analytical script by just drag and drop (but it does take some time, and you have some learning time needed)

So at Amnesty Flanders I made or famous IT-guy George install the RM server software and connect it to our civi-data. I'm now test how they both communicate and what in reality I could do with it.

Soon I'll have more contact with civicoop and the people of RM do go through this all in detail. If it all works, the RM server could play a role as the engine to enable automated donor journeys. 

I'll keep you posted.

In attach a drawing of mine on how I see civiCRM and RM server working together in automated donor journeys...

Greetings,
Ilja

Just to update on this:

I finished the biggest part of testing and what I initially had in mind seems working:
I can pull out data from civi, do some fancy geeky dataporn with it and send the result back to civi (for example as a new group or tag). Sending back to civi does still need some optimisation but it works.

So I now start building real analytic processes supporting donor journeys. Exiting ;-)

Greetings,
Ilja

At the moment Ilja is sending it directly to the database, which IMO is not the best option. I would prefer Ilja to use the REST API but we need to check if RapidMiner allows that.

Thanks for the feedback JEdward!

John,

great if you want to contribute!! The best API documentation is on the Wiki: http://wiki.civicrm.org/confluence/display/CRMDOC/Using+the+API
The idea is that the data analysis is done in RapidMiner and donors are then placed in CiviCRM groups which deal with the donor journeys. So the API's to have a go with are the GroupContact API's. Probably this is stuff you know but just in case:
Use <your civicrm site>/api/doc to get to documentation on what parameters to use
Use <your civicrm site>/api/explorer to try the specific API call you need and see what it should be in code

Hi Xavier,

Node.JS is great, I'm just starting to learn JQuery to help improve my skills, but with this it would all be done with no coding needed; it would all be written directly as a RapidMiner processes (each API call would be a separate process) which can then be called from other processes and fired by the RapidMiner server in batches or as a webservice. 
It's something I've done many times already to integrate various ESPs, Facebook & Twitter data into RM. 

It's possible to code extensions & scripts for RapidMiner when needed, but I don't think it's necessary here.  Particularly as, should the API ever change, there are more people who know how to read a RapidMiner process and debug it to help keep the CiviCRM-RM process updated than there are people who can keep the code of a RapidMiner extension updated. 

Erik,

What I'll do (as I'm pretty pressed with the day job) is build the basics alongside instructions for use and how to build more calls and then post it either directly here or via a file sharing link which you can then download and import onto your test system and once you're happy and comfortable with it, you can then integrate it into Amnesty's RM processes. 

However, I've not got access to a CiviCRM implementation and will build this on my laptop.  It should be possible from the briefly reading the documentation to build it blind by using the example outputs, etc, but I would prefer to have something I can test.
Do you or anyone else have an API key and server details for a demo or unused installation that I can build the process on for testing? 

Best,
John.

Really cool and exciting news Ilja!

Yes there has been progress, but Ilja should tell you more about this. I know he has succesfully linked RapidMiner with the CiviCRM database, and is about to use CiviRules to setup his donor journeys. We will be at CiviCon London but not sure if that will be in your region....
Check out this blog post too: https://civicrm.org/blogs/erikhommel/extension-civirules-now-avaible-download-live-showcase-maf-norge