rest.php interface: a few questions/rant

[xavier]

Hi,

Got a "few" issues with the rest interface:

1) it isn't a rest interface, as you have to call the login service, and keep both the session and a key it returns to be able to use any other method. REST is supposed to be about independant methods you can call the way you want in the order you want

What about doing something like google calendar, where each user got a secret GUID (a long non guessable string), that can be used instead of the login+password, and can be added on every request, so it's properly REST ?

2) What's the point of that key anyway ?
Right now, the only thing that stop us using the rest api to make really funky web2ish interface (with jquery+json+PURE, that's be really a nice fit), it's that bl*** mandatory key.

I don't understand what's the added value of this key: if you have a session, that's because you logged in. If your account has been compromised and your pwd stolen, nothing stops the attacker to use the rest login method anyway to get the key. In a nutshell, that's annoying, and IMO it doesn't increase the security.

3) If you want proper authentication :OAuth (IMO, overkill)

4) right now, the "standard" is having a REST method (eg. civicrm/contact/search) having calling contact_search in  api/v2/contact.php with each get variable transformed into a key into a big params associative array
(beside the return modification I made)

Works, but knowing what key are accepted, and what value is a bit of a guess. I don't have any idea how to improve it, just complaining
Any self documentation/ explaination mode/ debug mode idea ?

In conclusion, that rest api needs some love, I've been able to use it to interface it with an external CMS (spip), works, but should be made easier to use and better promoted, and used as well from within the civicrm (eg for the autocomplete features, what's the point of having separated json method, one size fits all
X+

[xavier]

The interface doesn't provide error codes. Just an error flag and a text. I'm missing an error code to make the difference between "I don't return the contacts because your session is not valid anymore" or "I don't return contacts because you didn't put the right params" or ...

On the top of the error codes, it'd might be useful to have "warnings". eg. you've asked to search contacts according to a criteria that doesn't exist, or required to return a field that doesn't exist, or... Basically, at least in developpement, to provide some additionnal clues about why it returns an empty list/doesn't take into account a criteria.

X+

[Donald Lobo]

Would be great for you to step up and come with a new revised REST interface that is more useful, generic, secure etc. You might also want to chat with Aaron Crossman about your plans and ideas. We are open to ideas/suggestions/alternatives for a better implementation (along with complete code

Each contact (and hence user) has a 32 character random md5 hash in the record, that you can potentially use as a GUID

ping us on irc and we can chat more

lobo

[xavier]

Hi,

If Aaron could jump on the thread, that'd be great indeed.

Anyway, I think one of the big/important feature is to have CiviCRM using that interface for CiviCRM.

There isn't any reason the various autocomplete and other webtwoize can't be done using a regular json interface, and if CiviCRM does need a method X, there isn't any reason to let this server side accessible by another client than CiviCRM.

It'd provides several benefits:
1) the rest/api becomes more central, that's never good to be at the fringe.
2) we got good examples of how to use the rest interface
3) it might make it easier to change the main interface if the api is versatile enough.
4) we got at least one "real" client

Don't know where you are on the various ajax improvements you've discussed here and there, but could it be plugged and using that rest interface ?

X+

[acrosman]

Happy to jump in...sorry, been asleep at the wheel on threads about REST lately.

I agree that the key is a pain without good purpose, having to keep both the session AND maintain a key in the URL just duplicates effort.  If the attacker is good enough to hijack a session, they can beat a GET parameter without issue.  I would expect if they can pick off the URL,they can manage to hijack the session as well.  Even with that resolved, NEVER use this API without SSL.

I posted on the blog (http://civicrm.org/node/412) a couple months ago that we're working on getting a PHP library written to go against the REST API, and that our vendor has agreed to allow us to open source the library once it's completed.  I got an update from them today, and they are still squashing the last few bugs.  I've been holding my tongue about changes that would need to affect the database until I have the PHP library in hand and until 2.1 was out the door.  I didn't want to cause problems for the 2.1 process, and frankly I didn't want our vendor having to deal with a moving target.

That said, there are a couple of things that would be nice to clean up.  First and foremost, getting those API keys saved outside of the session.  It would be nice it this were more like many of the existing public APIs, where I could request a key from the system to use going forward.  Alternatively, even it's felt that we need to require the remote system to login every time it does something, at least save the key in the database and NOT the session, it could be timestamped easily enough. Either way it would make life easier.

As soon as I have the library in hand, I'll work out a way to get it shared and provide documentation for it.  I wrote up some detailed docs for using the API that I gave our vendor to start out the project.  I'd be happy to share those now if it would be helpful to others.

[xavier]

I agree that the key is a pain without good purpose, having to keep both the session AND maintain a key in the URL just duplicates effort.  If the attacker is good enough to hijack a session, they can beat a GET parameter without issue.  I would expect if they can pick off the URL,they can manage to hijack the session as well.  Even with that resolved, NEVER use this API without SSL.

Why would it make it more risky to get this hijacked than a regular usage on the the normal interface ?
Anyway, here is a patch to get rid of the key (makes it much easier to set the interface from the comfort of your browser)
http://issues.civicrm.org/jira/browse/CRM-3832

I posted on the blog (http://civicrm.org/node/412) a couple months ago that we're working on getting a PHP library written to go against the REST API, and that our vendor has agreed to allow us to open source the library once it's completed.

Yes, thanks, got me started, and i've added an api http://issues.civicrm.org/jira/browse/CRM-3823

As for the doc, the only api I used is the contact search, and what I'm missing is the list of the parameters that you can have. For instance, how can I return the first name and last name as separate fields ?

I've found that the easiest is to use the advanced search, and pick up the name of the fields... Surely, there is something better.

As soon as I have the library in hand, I'll work out a way to get it shared and provide documentation for it.  I wrote up some detailed docs for using the API that I gave our vendor to start out the project.  I'd be happy to share those now if it would be helpful to others.

YES !!!!!

[xavier]

I did start a few weeks ago a page on the wiki:

http://wiki.civicrm.org/confluence/display/CRM/REST+interface

If you could dump your knowledge into it, it'd be great.

[Donald Lobo]

Ideally we can get most of this into 2.2. The current code freeze target is dec 1, so if we can get all code changes in prior to that, it would be awesome

thanx for combining efforts and working on this. Considering the timing etc, we'll probably start using the rest api for ajax calls in 2.3

lobo

[acrosman]

I'll work on getting the stuff I wrote into the wiki later today. 

I'm happy to put some work into getting the code for the REST interface updated to store the keys someplace other than in the session, but what do we need to do to create that place.

Someone needs to decide if the keys are static, and generated through the web interface and copied into the app, or if the keys are generated by a login at the start of the process.  I think either is acceptable, although static keys certainly make some applications much easier to develop.  Lobo are you that person?  How do we get that part handled?

[xavier]

I'll work on getting the stuff I wrote into the wiki later today.

Great !

I'm happy to put some work into getting the code for the REST interface updated to store the keys someplace other than in the session, but what do we need to do to create that place.

Someone needs to decide if the keys are static, and generated through the web interface and copied into the app, or if the keys are generated by a login at the start of the process. 

If they aren't static (ie. per session), then let's keep what we have now and add &PHPSESSID=<whatever is generated by /civicrm/login> on the requests


If they are static Lobo mentioned a MD5 (civicrm_contact.hash most probably) per contact, but I'm not convinced that's a good idea. Rationale:
1) It must be possible to reset it whenever we want, and be sure that it only affects the REST interface
2) That's, basically, a cleartext login+password, should be fairly hard to find or oversee.
3) Only a handful of users should be able to have access via that "non session based" REST interface

The easiest IMO is to have a rest/authorized_keys.php that contains:
<?
$civicrm_rest_keys = array ();
// contains the user authorised to access the rest interface statically, the format is
// $civicrm_rest_keys ["<key>"] = "<username>";
// example:
// $civicrm_rest_keys ["b5c3e8087731aac3bb790ff22744b33098ff070c"] = xavier.dutoit
// generate by hand now, suggestion: use sha or mD5 on a random string to generate the key. The key has to be long and difficult to guess


?>

and includes in the rest.php

Issues:
1) Haven't been able to find a method that allows to login without providing the password
2) I haven't been able to avoid starting a session

[acrosman]

I've added significantly to the wiki documentation of the REST interface. 

There was one difference from what I get from my system and what you've been noting.  I'm not getting the PHPSESSIONID back as part of the XML response from CiviCRM, I just get the error flag and the key.  Are you seeing something different, or just documenting what you'd like to get?

I wondering if allowing either instead of requiring both might be the way to go.  For the AJAX interfaces to come, sessions might be easier since it could be maintained by the browser, and not require the coder to do much in the JavaScript.  However, when doing server-to-server work (which is what I'm interested in at the moment), the session just adds a layer of complexity.  I'm okay with the login requirement for server-to-server work, but having to maintain both the session cookie and the key is just redundant.  If the session allowed AJAX develops not to have to authenticate again (since the user has already done that) it should make that easier as well.

Oh, and as far as I'm concerned, it's okay to establish the session, as long as I don't have to maintain it.

[acrosman]

I finally read the other thread from October about this issue (http://forum.civicrm.org/index.php/topic,5230.0.html).  Now I understand where the PHPSESSIONID came in, and why I'm not seeing it (I'm currently running against the latest stable 2.1, not development versions).

[xavier]

Hi,

The patch should be fairly easy to apply to your version (one line to change) try the patch I submitted.
http://issues.civicrm.org/jira/browse/CRM-3832

Could you apply and confirm it works as advertised on the wiki ?

As for the complexity, it's very easy: call the login, get the PHPSESSID, and add &PHPSESSID=<returned value> on all your queries on machine to machine interfaces. To test from the browser: look ma, works directly, without any extra param ! (all set by the cookie)

You don't have to worry about session anymore, the session ID is passed on get instead of cookie.

I've modified the doc accordingly (many thanks for all the detailed explanations).

Do you think you'll be able to share the new api you created before the code freeze end of the month ?

X+

[xavier]

I wondering if allowing either instead of requiring both might be the way to go.  For the AJAX interfaces to come, sessions might be easier since it could be maintained by the browser, and not require the coder to do much in the JavaScript.  However, when doing server-to-server work (which is what I'm interested in at the moment), the session just adds a layer of complexity.  I'm okay with the login requirement for server-to-server work, but having to maintain both the session cookie and the key is just redundant.  If the session allowed AJAX develops not to have to authenticate again (since the user has already done that) it should make that easier as well.

Just to be sure we understand each other: with my patches, you replace &key=<bla> by &PHPSESSID=<bla> and it works in a machine to machine, no need to bother about session.

The ultimate goal is to be able to have a permanent key, that you can put on any query and it just works.

@Lobo, I can't find a way of setting the active user without providing the pwd to the authenticate method. Does it exist ? (would be magic on the cli, more improvements on that area before the code freeze).

Additionnal question: is it possible to avoid the (albeit small) overhead of starting a session and still being authenticated the duration of the script ?

[Donald Lobo]

@Lobo, I can't find a way of setting the active user without providing the pwd to the authenticate method. Does it exist ? (would be magic on the cli, more improvements on that area before the code freeze).

You will need to write another authenticate function and call that when you are sending in a "pass-phrase" unique to each user and do the work there. (i.e. find the userID / contactID and set that in the session)

ajax requests definitely would not want to reauthenticate and send in hash key etc

lobo