CiviCRM Community Forums (archive)

[11:00] <xavier_d> eileen, that's way more than profile config that you are listing
[11:00] <xavier_d> (eg import/export mapping)
[11:00] <eileen> yep - not sure that's required
[11:01] <eileen> I didn't add an action for that
[11:02] <eileen> (the ones @ the top I added actions for - the others are hopefully either covered by the actions or not required
[11:02] <xavier_d> but good to see what's missing, no matter what
[11:02] <xavier_d> did you already think about the name/syntax for the missing apis?
[11:03] <eileen> nope - took quite a while just to get that much order into it all
[11:03] <eileen> I went throught the UI one day & couldn't face it again for several days :-)
[11:04] <xavier_d> id/task id, that's something internal for you or is this meaning something for everyone?
[11:04] <eileen> yep - those are mostly about sorting in Excel :-)
[11:05] <eileen> (& making sure if I sorted it wrong I could get the original order back )
[11:06] <eileen> So, I'm thinking stage 1 would be to add the settings api & a way to load it from xml - with probably domain + loc_block + chain too to test the xml syntax
[11:06] <eileen> not sure the xml would be part of API or just a drush command
[11:06] <xavier_d> for the syntax it works already on json (including chaining)
[11:07] <eileen> hmm - is json used to ship config in the way xml is? I guess Drupal would just do arrays
[11:08] <xavier_d> not sure json is the right format indeed, just mentioning it. IMO the syntax is part or drush command, not the api
[11:08] <eileen> OK - seems a reasonable position
[11:08] <eileen> So, how on earth does one try to come up with any time estimate!
[11:10] <xavier_d> yes, quite a few moving parts. did you see if everything stored at the same place (eg. civicrm_setting) has the same BAO?
[11:10] <xavier_d> ie. not too hard to have something generic?
[11:11] <xavier_d> (this being said, enabling a component or list of available countries aren't going to have the same syntax at all)
[11:11] <eileen> well - I made some assumptions about which would be in settings table - I didn't check every value - some will be wrong
[11:13] <eileen> I think most settings will are serialized - so it will need to know if it is receiving an integer or string or array
[11:14] <eileen> So, maybe need to set up a way of describing settings (would probably include info currently stored in templates too)
[11:14] <eileen> ie. an xml for them
[11:16] <xavier_d> so starting with an settings -> xml converter?
[11:17] <eileen> Well, I was thinking about how the schema xml is stored & thinking about having one for the settings table maybe- so it's possible for the API (& other code) to understand the data-type of each setting
[11:18] <eileen> (of course maybe that aready exists in some way - haven't looked @ what the BAO does)
[11:21] <eileen> Well - I think I have to go to bed - will flick out an email encouraging others to look
[11:21] <xavier_d> actually, wondering if some of these "settings" should be exposed directly or is this more an implementation detail that should be abstracted for the api users. eg that the default country or the list of enabled country is stored in the settings is an implementation choice and could be moved to civicrm_country without breaking all the config scripts
[11:22] <xavier_d> eg. civicrm_api ('country','create', iso='us', is_default=>1)
[11:22] <xavier_d> and this does store it into settings
[11:23] <xavier_d> on the chaining, we might need different "actions" (eg create_if_not_exists)
[11:23] <xavier_d> or having options=>on_error_update
[11:23] <eileen> oh yeah - that's probably needed on chaining
[11:24] <eileen> the issue with stuff being moved is kind of about testing - ie. we need enough 'static' tests that we pick up changes & adapt the api to cope with them
[11:25] <eileen> (which will mean over time the API becomes increasingly cluttered & we need to start looking for excuses to move to v4 & ditch the clutter :-)
[11:25] <xavier_d> yeap, that's a problem with kind of have already with the constant api
[11:25] <eileen> Actually with 4.2 coming out I know of at least one schema change which we possibly should add api handling-cruft for - the change from civicrm_event.payment_processor_id to civicrm_event.payment_processor
[11:26] <xavier_d> where some stuff you can read from there but need to use optionvalue to modify
[11:26] <xavier_d> anyway, great work, will think about it

(10:20:54 AM) eileen: So, I chatted with Joe & his prioriteis are 1 - stuff in civicrm_settings, 2 some thing still in domain->config & 3) contribution page
(10:21:31 AM) eileen: I'm guessing that the things in domain->config (e.g. date format) are probably intended to be moved into settings before too long
(10:21:46 AM) totten: yeah. i think it would be better for us/him to push that along
(10:22:06 AM) totten: unless it's really important to support old versions of civi
(10:22:53 AM) eileen: Re the civicrm_settings table - It makes sense to me to create a civicrm_api('settings','create' ) action (& a 'real' update for when you don't have id)
(10:23:46 AM) eileen: but where I see an issue is that the value in civicrm_settings table is stored as a serialized array & I *think* that knowledge of what the variable is @ the moment is in the form layer
(10:24:10 AM) totten: right. it should be in some xml metadata -- like we do with sql schema
(10:24:42 AM) eileen: OK - so that was what I was trying to say in the emails
(10:24:57 AM) totten: right. i agree.
(10:25:22 AM) eileen: (did you understand that was what I was trying to say or did you just separately come to that conclusion?)
(10:26:00 AM) xavier_d1: beside, would help to move away the big serialise blob if we want to store on separate records
(10:26:29 AM) totten: eileen: both
(10:26:32 AM) eileen: xavier_d1 - are you refering to the $domain->config blob
(10:26:53 AM) eileen: totten - yay - I wasn't clear if my mention of 2 sorts of xml had hopelessly muddied the picture
(10:27:26 AM) xavier_d1: eileen: made sense to use xml to define the structure
(10:27:48 AM) totten: xavier_d1: i think eileen was saying that the payload of some civicrm_setting rows contains serialized blobs
(10:28:17 AM) totten: actually, i guess all of them do...
(10:28:26 AM) eileen: So, I'm thinking it would be a case of 1) coming up with some sort of xml structure, 2) prototyping a settings form to run directly off it (with less in form layer) & 3 write an API for settings that uses the same for validation when settings
(10:29:13 AM) eileen: also, need to look @ what function does the equivalent of sql REPLACE INTO (ie. create or update as appropriate)
(10:29:48 AM) xavier_d1: totten: all the 39 rows contains serialise blob I think
(10:30:37 AM) eileen: If you look @ drupal config architecture (http://groups.drupal.org/node/191283 - which I believe was the thinking behind Civi)
(10:30:52 AM) eileen: they also allow you to store config on disk as xml
(10:31:38 AM) eileen: I would imagine that would be really cool, done @ the setting BAO level - but probably not phase 1 unless already in place.
(10:32:37 AM) eileen: then the next challenge would be tracking changes to the xml. ie. ideally whenever either the schema xml or settings xml changed there would be a ticket logged to assess the impact the impact of the change on the api

(10:33:32 AM) eileen: & as the settings & schema morphed over time we'd build up yucky long _civicrm_api_contact_create_legacyhandling() functions until we found an excuse to drop API v3 & all the legacy cruft

(10:35:25 AM) totten: eileen: building up legacy adapters for changes in settings xml doesn't sound appealing
(10:35:43 AM) eileen: nope - but, any other options as it changes over time?
(10:35:48 AM) totten: i'd lean towards a "fail-fast, fail-hard" approach
(10:36:10 AM) eileen: so force people to adapt - I guess that's OK for settings
(10:36:58 AM) eileen: as in it's used for site set up - it's the ones like the change to event schema from processor_id to processor that we really should handle as a legacy setting
(10:37:15 AM) totten: right. the approach isn't appropriate for every piece of data, but it feels acceptable to me for the multi-site autoconfiguration stuff
(10:37:16 AM) eileen: otherwise - where is the claim that the api is more future proof than sql
(10:37:51 AM) xavier_d1: doesn't sound too weird that if the setting changes between civi versions you need to adjust the code
(10:38:16 AM) xavier_d1: eileen: is there already a way to know civicrm's version via the api?
(10:38:40 AM) xavier_d1: that's probably needed to know what setting can/can't be done
(10:38:46 AM) eileen: nope - that makes sense, so basically we equip the settings api (via xml descriptor) to do reasonable validation & then do that before anything get's written or roll back
(10:39:34 AM) eileen: version should be available from domain - as long as upgrade has been run
(10:40:31 AM) xavier_d1: well, kind of: config_backend":"a:81:{s:15:\"civiRelativeURL\";s:17:\"\/civicrm_staging\/\";s:10:\"lcMessages\";s:5:\"fr_FR\";s:25:\"monetaryThousandSeparator\";s:1:\"'\";s:20:\"monetaryDecimalPoint\";s:1:\".\";s:11:\"moneyformat\";s:6:\" %a %c\";s:16:\"moneyvalueformat\";s:3:\"%!i\";s:12:\"countryLimit\";a:229:{i:0;s:4:\"1228\";i:1;s:4:\"1070\";i:2;s:4:\"1068\";i:3;s:4:\"1064\";i:4;s:4:\"1240\";i:5;s:4:\"1239\";i:6;s:4:\"1237\";i:7;s:4:\"1235\"
(10:40:43 AM) eileen: there's a version field!

(10:41:03 AM) totten: you can also get version another way

(10:41:28 AM) totten: ie CRM_Utils_System::version(); -- which reads from a file
(10:42:01 AM) grobot: db version is db version, for upgrade you need to compare them
(10:42:22 AM) totten: right. for the uprade you compare them
(10:42:45 AM) eileen: So, one question there is whether we should think about making some util functions available via the API - e.g. civicrm_api('utils','getversion',$params); - & if we did that would it be possible to keep them sane

(10:42:59 AM) totten: but in the context of, say, an install profile -- would the two versions agree?
(10:44:15 AM) eileen: NB - I don't know if there are any potential positives from this but in drupal config it looks like drupal will read the config (d8) if in a dir called xml
(10:44:18 AM) eileen: called config
(10:45:00 AM) totten: wouldn't it be a bad idea to run any api call (settings or otherwise) when the db and code disagree
(10:45:12 AM) xavier_d1: civicrm_api ("settings","get", return= array("code_version","db_version"))
(10:45:21 AM) eileen: totten - that seems like a reasonable point of view
(10:45:34 AM) xavier_d1: totten: why would we be more restrictive than the UI?
(10:45:53 AM) eileen: OK - so code_version & db_version are not actually in the settings table though are they (xavier_d1)?
(10:46:18 AM) eileen: or are you saying they should be
(10:46:41 AM) xavier_d1: eileen: or that the api does fetch the info at the right place if not in setting
(10:47:18 AM) xavier_d1: ie. where this is stored shouldn't be the concern of the api user
(10:47:40 AM) eileen: hmm - maybe - but then we're starting off with something that almost feels like a hack to the new api
(10:48:27 AM) totten: ok, so xavier_d1 is saying that the db_version should be reported to downstream devs the same way as settings

(10:48:43 AM) totten: if we want that, then i think we should move that data from the domain backend to civicrm_setting
(10:49:08 AM) xavier_d1: totten: does it make sense to have it in domain anyway?
(10:49:38 AM) eileen: for 3.4 / 4.0 yes, after that I doubt it
(10:49:39 AM) xavier_d1: as in could it be different between domains of the same instance
(10:49:59 AM) eileen: (ie. I have sites with d6 & d7 plugged into the same civi - mostly as a transition)
(10:50:16 AM) eileen: but that's not really relevant as we're not expecting that to occur again
(10:50:34 AM) eileen: so, OK - settings table makes sense to me
(10:51:14 AM) totten: settings makes more sense to me; having multiple values for db_version (for domains) would be problematic
(10:51:39 AM) totten: the db_version is used to drive DB updates -- e.g. schema changes which apply to any/all domains that share a DB
(10:51:49 AM) xavier_d1: anyway, in the long term, most of config_backend in domain will migrate to settings, isn't it?
(10:52:12 AM) eileen: OK so I feel like we are fairly in agreement on the api side of it & the xml, what about testing - do we have any specific ideas
(10:52:44 AM) eileen: I guess we need to have an automated test that chugs through setting & getting the settings

(10:52:58 AM) xavier_d1: api: what's your feeling of "perverting/extending" getfields to expose the xml ?
(10:53:23 AM) xavier_d1: ie. show each setting as a separate attribute of the setting entity
(10:53:26 AM) eileen: I was thinking about a new function
(10:53:32 AM) eileen: getoptions
(10:53:35 AM) eileen: perhaps

(10:54:11 AM) xavier_d1: we agree that "usual getfields" doesn't make sense for settings, right?

(10:54:51 AM) totten: from the perspective of downstream dev,  it would be nice sugar to treat each setting as a different attribute on the singleton setting entity
(10:54:54 AM) eileen: no, you've possibly noticed that some fields now return options (e.g. activity.priority_id) but that doesn't make sense either I don't think
(10:55:11 AM) eileen: totten - say that again - in 3 sentences
(10:55:15 AM) xavier_d1: I think it will make it easier to discover if we use the same actions than usual
(10:55:19 AM) eileen: with more words
(10:55:23 AM) totten: e.g. civicrm_api('setting', 'set', array('mail_server'=>…, 'mail_port'=>…,'mail_user'.=>…))
(10:55:55 AM) xavier_d1: eileen: that would make sense as well on settings.getfields I think

(10:56:23 AM) xavier_d1: eg. outgoing mail is smtp, null, mail(), sendmail
(10:56:37 AM) totten: right. you would make it appear that way for all parts of the api -- CRUD and getfields
(10:56:55 AM) eileen: OK - so, the point there is that there is more than one field in that table - & that format doesn't cover them
(10:57:20 AM) eileen: are any of them important?
(10:57:58 AM) eileen: possibly domain_id & contact_id are -the rest can be derived from the XML I think
(10:57:59 AM) xavier_d1: totten: yes, but as it's a singleton no id

(10:59:16 AM) xavier_d1: eileen: the only one that has a contact_id is the navigation menu
(10:59:55 AM) totten: so would one use this api to set options in a multi-domain context?
(10:59:57 AM) eileen: yes, that seems to be the case. In which case it may make sense for a separate api for navigation menu when it's for a contact?
(10:59:59 AM) xavier_d1: totten: actually, might be that the id is the domain id and that it isn't (always) a singleton
(11:00:07 AM) totten: i.e. domain_id=2 has one value for mail-server and domain_id=3 has a different value
(11:00:17 AM) eileen: totten - yes -
(11:00:29 AM) xavier_d1: eileen: does it work?
(11:00:40 AM) eileen: or a better example is domain_group_id
(11:01:05 AM) eileen: xavier_d1 - yes, I think that would be OK - would it confuse people who are used to our current usage?
(11:01:26 AM) eileen: It would mean you could do a get & get the different domains indexed by their ids too

(11:02:09 AM) eileen: NB - are you thinking we still call the actions 'create', & 'get' & 'getfields' but just massage the table interpretation?
(11:02:40 AM) xavier_d1: eileen: yes, and I don't think I twist them too much
(11:03:02 AM) eileen: OK, I'm feeling OK with that -
(11:04:00 AM) eileen: nb - did you look @ the drupal architecture link?
(11:04:13 AM) totten: so do we need to be concerned that hte list of of per-domain and per-contact settings are probably different?
(11:04:39 AM) eileen: totten - yes for per-domain - although we could treat it as an id per x
(11:04:59 AM) eileen: for per-contact - not sure - if it's just navigation menu perhaps we should treat that differently
(11:05:53 AM) xavier_d1: is this really possible to have different navigation menues between users?

(11:06:09 AM) xavier_d1: (beside hiding due to permissions obviously)
(11:06:16 AM) eileen: esp as navigation menu doesn't seem to store the data in the same way - it feels like a shotgun marriage between settings data & navigation

(11:07:27 AM) eileen: like, only navigation uses contact_id & navigation doesn't use value but the were already co-habitating in the old preferences table so someone said they couldn't separate
(11:08:14 AM) eileen: So, I'm going to vote that the limit of the settings api is anything that is per-contact & that gets it's own api
(11:08:37 AM) xavier_d1: eileen +1
(11:10:06 AM) xavier_d1: and anyway, if we can manipulate the navigation via the api, that's hopefully going to be something different than value= 22k of serialized php that contains html that contains javascript
(11:10:28 AM) totten: eileen and xavier_d1 : just asked dlobo about why the two types of settings are stored in the same table
(11:10:31 AM) dlobo:  yes, but u dont want to recompute that 22K on every page load
(11:10:48 AM) dlobo: since navigation is displayed pretty much on all pages
(11:11:03 AM) xavier_d1: sure, It's for the setting, for the storing make sense
(11:11:14 AM) eileen: nope - I'm lost
(11:11:23 AM) totten: the basic design issue is that the programmer often doesn't care about whether settings are "per-domain", "per-contact', "per-group", etc
(11:11:56 AM) totten: the programmer wants to say "give me the active timezone" -- but the timezone might be a per-user preference or a per-domain default
(11:12:43 AM) totten: so you want one interface -- eg CRM_Core_BAO_Setting::getItem() -- which looks up the setting regardless of how the policies are designed
(11:14:00 AM) eileen: OK - so the options are to either have to set your contact & domain first & it will only let you get them in a call
(11:14:22 AM) xavier_d1: totten: 99% of the case that's single domain and navigation is the only case with contact id set
(11:14:22 AM) eileen: or, you do as Xavier suggested (for domain) & optionally set the domain a s 'id'
(11:14:57 AM) eileen: yes, that's the thing - navigation also appears not to use the value field - so it's very different (either that or the upgrade script mulched it)
(11:15:18 AM) eileen: but, do we forsee other settings that might involve contact
(11:15:47 AM) xavier_d1: so if civicm_api ("setting","get", array(return="timezone", id=1))
(11:16:15 AM) dlobo left the room (quit: Quit: dlobo).
(11:16:25 AM) xavier_d1: id can actually be optional if single domain or if we can guess the domain (eg via rest/ajax)
(11:16:35 AM) eileen: yep
(11:16:52 AM) eileen: & people dealing with multidomains understand it's a multi
(11:17:19 AM) xavier_d1: and instead of trying to make something complicated to cover things that don't exist, we decide it won't be dealt through this setting api if contact_id
(11:17:22 AM) eileen: but we'd want to support 'is_current_domain'
(11:17:38 AM) xavier_d1: eileen: if not set it's implicit
(11:17:48 AM) xavier_d1: (id not set I mean)
(11:17:51 AM) eileen: how do you return all then?