OSDI - common standard for interoperating with other CRMs

[jamie]

Hi all - I was wondering if anyone in Civiland has OSDI on their radar?

The web site is here:

http://opensupporter.org/

It seems like something we may want to support. However, I imagine others may have some experience with this initiative and a  more informed opinion.

[JoeMurray]

I think we should support given the list of supporters. We should aim to get our logo in there. Can you take the lead, Jamie?

[Michael Z Daryabeygi]

TOtten and I discussed it back in July.

I haven't found the NOI folks very approachable, they always seem to busy for me.
We are trying to engage with them about doing trainings for them.
I know people that know people at NOI if you need help opening the discussion. I hope you have better luck than I did.

Here is Tim's reply and my email on my cursory look at OSDI:

Tim said:
(I’ll put more high-level discussion in this thread — and more technical points in the other thread.)

Random thoughts in no particular order..

1. I haven’t heard anything about it, but if someone approached us, they’d probably go to Lobo or Dave.

2. My first impulse was to wonder about how much risk/value/benefit would come to Civi from supporting a standard like that. There may be opportunities for Civi to integrate more easily with other apps by supporting a standard, and there may be risks of being coopted by other vendors. I wish there were a way to make those risks/benefits more concrete.

3. The authors list includes a number of recognizable companies. My questions would be — why are they supporting it, and what level of buy-in do they have for the concept (eg buy-in at the junior-developer level? product-manager level? C-level)? Clearly they don’t want to publish that on their website, but… one must wonder.

4. I unthinkingly accepted the “open-standards" framing. You may be right not to accept their framing - that framing doesn’t play to our strength. Open standards are a form of openness — but not as open as FOSS. There are downsides to open-standards (ie the only thing slower than open-source development is open-standard development; and if the standard is truly multivendor, then there’s likely some impedance mismatch and/or proprietary extensions). If someone really prioritizes openness/freedom and efficiency, then maybe FOSS is the better way. (Of course, FOSS and open-standards aren’t mutually exclusive… in fact, thinking about successful open-standards, the poster-child for the standard is usually a FOSS implementation.)

5. If OSDI wrote and published a comprehensive test-suite for their standard, then we could discover that supporting an open-standard is easier than building a new API of our own (because it cuts down on testing costs).

On Jul 27, 2014, at 11:36 AM, Michael Daryabeygi <daryabeygi@ginkgostreet.com> wrote:

> Was Civi ever approached about or is otherwise aware of this project?
>
> It is an attempt to create a standard api and data adapter for some of the popular tools used by political organizers.
> "The Open Supporter Data Interface (OSDI) effort seeks to define an API and data structures for interoperability among products in the cause-based, campaign and non-profit marketplace. The existence of a common API will reduce customer costs related to moving data between different systems, lower integration costs and enhance the ability of innovators to create products for the marketplace."
>
> http://opensupporter.org/
> http://opensupporter.org/whitepaper/
> https://github.com/opensupporter/osdi-docs/blob/master/scenarios/scenarios.md
>
> ... which is how I stumbled on JSON+HAL
>
> They haven't posted the code itself on github.
>
> I met someone in the code for progress training program who is being mentored by NOI to work on the project.
>
> I'll be working on finding about more about the project.
>
> I'm currently feeling a little baffled. NOI does great things, but why aren't they embracing Civi?
> Why are they trying to have the benefits of Open Source while promoting proprietary systems?
> Supposedly, they will open source the code once it is built. Makes me wonder if the partners are not fully supportive of open sourcing it.
> There is a footnote to the authors list: "This effort is currently in an exploratory phase to determine if consensus on a common API can be achieved. The involvement of a person or company does not reflect a commitment to implement this API."
>
>

[JoeMurray]

Is there a tech standard at the moment for OSDI?

[adixon]

Looking at that list of supporters and the ambitiousness of the proposal, I'd guess this is likely to go nowhere useful in any short term, and maybe never. We don't want to say no to motherhood, but if it all turns into a fiasco, do we really want to spend the energy on it? I wonder if anyone else has a more realistic proposal? What do you think are going to be the real drivers of this project [e.g. large projects trying to cement their market position by making an api that only they could possibly implement completely?]

Here's the thing: a CRM in the abstract is hopelessly complex - you can start and think it's all codeable, but people are way to complicated and changing in how they interact with each other and organizations and the rest of the world, they don't abstract nicely. Start by thinking through all the imports from other crm's you've done ...

[andrewhunt]

So what's on my mind is the use cases: why does a standard matter, and for whom?

• integration for non-integrable tools: people without integrated systems like CiviCRM can connect their event forms and donor database
• easy migration: old system X can respond to OSDI API queries, so new system Y can automatically move people
• one-time list-share: you set up an ACL in organization A's system to allow a subset to be shared, and partner organization B can fetch the folks who are being shared
• federated systems: the central organization has the big list, and chapters with independent systems can sync their names using the OSDI API
• coalition clearinghouse: a coalition has a central system and shares names (within certain restrictions) freely to and from it
• vendor append: a vendor does social media profile or voter file appends has a single API to pull names and then update them with the data that's bough

The first case just makes other systems more like CiviCRM.  Not much interest there from me (except to watch what happens).

The next two cases don't really need OSDI: it just makes things faster.  The second case could be solved with the typical import techniques we know and love (and ideally with robust APIs on both sides).  The third is just as easily done with a spreadsheet.  The fourth makes sense, but it would be easier from the organization's perspective to just implement a big single federated database and train everyone on it.  The fifth  is the clearest use case, but the least-common in real life.

It's the last case that seems like the real goal of this, and you can see it in the list of supporters.  You could do one-time or biannual updates, or you could potentially have a service like Attentive.ly that scrapes posts on social media accounts of your supporters and updates them in your database based upon keywords.  It could run continuously, and using OSDI, they've only got one API to keep up-to-date.

The opportunity for us as a community is more integration points for CiviCRM: something that only works with NGPVAN or Salsa or something could start being available for CiviCRM.  This may be, at long last, the solution for the Congressional action alert problem: an OSDI-compatible vendor could handle it seamlessly from CiviCRM.  There's also some value in having CiviCRM's name out there on the OSDI site.

A risk would be that more integration standards make it easier to work using a series of fragmented services.  The CiviCRM "jack of all trades, master of none" approach would suddenly have as a realistic competitor a series of best-in-class systems strung together using OSDI.  That said, there's no reason CiviCRM wouldn't be the best hub for that, and I have faith that over time, CiviCRM will become top-of-the-line in each of the categories.

Anyway, those are some thoughts.  I think that it's probably worth being involved, and it probably could be simply a bunch of wrappers on the API.  There's a bunch of examples and documentation here: https://github.com/opensupporter/osdi-docs/blob/gh-pages/README.md

That said, Alan's points just now are really apt, and I don't know how much effort should be spent unless this looks like it'll be used in real life.

[xavier]

Hi,

Github contains a more detailed spec: https://github.com/opensupporter/osdi-docs/blob/gh-pages/README.md

It seems to be interesting and might be a good GSoC project, both as a consumer (eg. integrate with a petition tool) and as a producer (to be seen how the mapping goes)

@jamie and all, someone willing to mentor as a google summer of code project?
X+

[Michael Z Daryabeygi]

I would be willing to mentor a GSoC for this.

[totten]

I agree that this sounds like a good size/scope for a GSOC project.

[xavier]

Awesome!

Could you add yourself and the project to the wiki?

http://wiki.civicrm.org/confluence/display/CRM/Google+Summer+of+Code+-+2015

[Michael Z Daryabeygi]

and now this:
http://www.buzzfeed.com/evanmcsan/new-organizing-institute-implodes#.xhBmMXXpmZ

I don't know if Tim Anderegg is among the departing staff.

[joemcl]

Hey there. Jamie put me on to this thread, I just learned it was going on.

Let me try to make the case for why Civi should get in on OSDI. I e-mailed Lobo and Michael about OSDI awhile ago, and Michael levelled me up to be able to do a blog post to pitch OSDI, which I haven't, shame on me.

I'm the data guy for Citizen Action of New York, www.citizenactionny.org, and we're on Civi, we have been for about 3 1/2 years. I came on staff about two years ago. We have also joined up as a supporter of OSDI. With the help of Eileen, Peter Davis and other folks at Fuzion, we're using Civi as our organizing database, though we're not yet where we want to be yet on getting Civi to be, in my opinion, a true political/community organizing CRM/database.

We like Civi partially because it's open source, and we can tweak it as we want, with the help of Fuzion and the Civi community. We can't do what we want with what I see as Civi's for-profit and proprietary competitors in the CRM space for political and community organizing - like NationBuilder, Salesforce, etc. 

We are also customers of NGP VAN, Salsa, Attentive.ly and a couple of other vendors. We use Salsa for e-mail blasts. We have access to VAN voter data, as well as data from Catalist. We use VAN for voter and civic engagement. VAN and Salsa are not (in my opinion) really full blown and customizeable CRMs. They have certain valuable functions and data - whitelisting and updated legislative contacts for e-mail blast targeting in Salsa's case, and voter data, modeling, turf building and other election campaign tools in VAN's case. So we'll continue to use them. 

What we really want to do, at minimum, is sync up contact data between CiviCRM, VAN, Catalist, Salsa etc and hope to use OSDI to to do that, in real time, without constant imports and exports. While VAN and Salsa have APIs, they are not "open". OSDI allows us, I hope, and other (admittedly, paying) users/customers of those and other vendors to do that syncing. But the API can be adopted by any vendor, for-profit or not.

There are other political and community organizations in the US which are using CiviCRM and are also customers of VAN, Salsa, etc. or have access to those vendors services and data through relationships. I think there are probably many. Some of them I know are members/customers of the Progressive Technology Project and use Powerbase, a customized "flavor" of Civi.

Even with what looks like the collapse of NOI, as it had existed, there are many other vendors and user orgs like mine committed to OSDI. 

Joining up with OSDI and making it easy for Civi user orgs like mine to use the API can go quite a way to making Civi a better choice than NationBuilder, Salesforce etc. I can't speak for them, but I believe that some of the vendors supporting OSDI would be smart to help pay for the development work to connect Civi to OSDI. Citizen Action can also throw in some resources for that. I believe other US Civi user orgs - political, civic and community organizations in particular - would also benefit and could/should contribute.

We don't want to wait for OSDI to be a possible Google Summer of Code project. A better candidate for a Civi GSoC project would be hooking up Civi to the Google Civic Infomation API (see  https://developers.google.com/civic-information/ ) as well as the Google Maps API, both of which first require work to connect to Civi via the Google APIs Console.

[JoeMurray]

JMA Consulting would be interested in participating in developing real-time two-way data synchronization between CiviCRM and OSDI.

I also wouldn't wait for a GSoC project to come around. As I understand it, little code from the projects was production ready and merged into master before the end of the students' time, though with additional work from others their work has indeed pushed forward core and the development of a Voice Broadcast extension.

There is a bit of integration between CiviCRM and Google Maps: it is generally used to geocode addresses (ie look up their latitude and longitude) and geocoded addresses for contacts and event locations can be displayed on maps fairly easily. A variety of extensions have provided geolocation to elected representative lookups over the years. It would be good to see deeper integration in the form of turf-cutting.

I like the idea of integrating with Google Civic Information API, especially if they put together a Canadian version.

[joemcl]

Joe, great to read that JMA is interested in OSDI API connection to Civi. I'll set up a Hangout with Josh Cohen, chair of OSDI, to dive in more about OSDI,  why it's worth getting the connection to Civi going, and some of the specs. If other folks want to join that, please reply here or send me a private message via this forum.  For more background on OSDI, see www.opensupporter.org and https://github.com/opensupporter/osdi-docs/blob/gh-pages/README.md .