CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Scalability (Moderator: Donald Lobo) »
  • Making Memcache support more configurable
Pages: [1]

Author Topic: Making Memcache support more configurable  (Read 7826 times)

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Making Memcache support more configurable
February 11, 2011, 12:05:42 am
I have a client with several different CiviCRM set-ups on a single box.  Increasingly, I've needed to use memcached to get good performance.  While it's easy to set up memcache for CiviCRM -- you just add this to civicrm.settings.php...
Code: [Select]
define( 'CIVICRM_USE_MEMCACHE', 1);

it looks like there isn't a good way to keep two separate instances of CiviCRM from clobbering each other in cache, since the way the cache configures itself will end up sending multiple copies of CiviCRM into the same memcache buckets:
Code: [Select]
    // CRM/Utils/Cache.php, near line 61 in CiviCRM 3.2.3:
    /**
     * singleton function used to manage this object
     *
     * @param string  $host      the memcached server host
     * @param int     $port      the memcached server port
     * @param int     $timeout   the default timeout
     *
     * @return object
     * @static
     *
     */
    static function &singleton( $host      = 'localhost',
                                $port      = 11211,
                                $timeout   = 3600 ) {
        if (self::$_singleton === null ) {
            if ( defined( 'CIVICRM_USE_MEMCACHE' ) && CIVICRM_USE_MEMCACHE) {
                require_once 'CRM/Utils/Cache/Memcache.php';
                self::$_singleton = new CRM_Utils_Cache_Memcache( $host, $port, $timeout );
            } else {
                self::$_singleton = new CRM_Utils_Cache( );
            }
        }
        return self::$_singleton;
    }

In practice, the call to get the cache singleton always uses the defaults.  If I could be certain I was the *first* caller, I could call singleton() with a different port, which would get around this problem.  Assuming it was not too early to do so, I might do this immediately after I defined  :
Code: [Select]
define( 'CIVICRM_USE_MEMCACHE', 1);
$my _mc_host = 10.0.0.25; //default is 'localhost'
$my_mc_port = 11222; //default is 11211
require_once('CRM/Utils/Cache.php');
//initialize the cache...
CRM_Utils_Cache::singleton($my_mc_host, $my_mc_port);

I'm not sure if this would work or not.  But even if it would:  is there a better way to do this?

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Making Memcache support more configurable
February 11, 2011, 07:05:59 am

There are two potential options and might be good to do both:

1. Implement the host, port and timeout as config variables, so different installs can connect to a different memcache server

2. Add a new "prefix" to the above config, so all "keys" for that memcache are uniquely prefixed. This will allow the same memcache to be used by different installs. You can potentially do this automatically by md5'ing the dsn (or base url or both)

wanna submit a patch to do the above. We should use the preferences table for the config (and not the Config serialized arrray, moving away from that!)

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Making Memcache support more configurable
February 12, 2011, 01:00:12 pm
It looks straightforward enough.  I've factored the Cache code very slightly so that instead of taking the parameters from arguments, it calls a static routine that fetches the prefs.  The prefix is trivial (add it to the constructor, and make sure that get, set and delete use it).

One thing I'd don't understand yet is how to use the preferences table.  This is the table you're talking about, I'm guessing:
Code: [Select]
mysql> desc civicrm_preferences;
+----------------------------------+------------------+------+-----+---------+----------------+
| Field                            | Type             | Null | Key | Default | Extra          |
+----------------------------------+------------------+------+-----+---------+----------------+
| id                               | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| domain_id                        | int(10) unsigned | NO   | MUL | NULL    |                |
| contact_id                       | int(10) unsigned | YES  | MUL | NULL    |                |
| is_domain                        | tinyint(4)       | YES  |     | NULL    |                |
| contact_view_options             | varchar(128)     | YES  | MUL | NULL    |                |
| contact_edit_options             | varchar(128)     | YES  | MUL | NULL    |                |
| advanced_search_options          | varchar(128)     | YES  | MUL | NULL    |                |
| user_dashboard_options           | varchar(128)     | YES  | MUL | NULL    |                |
| address_options                  | varchar(128)     | YES  | MUL | NULL    |                |
| address_format                   | text             | YES  |     | NULL    |                |
| mailing_format                   | text             | YES  |     | NULL    |                |
| display_name_format              | text             | YES  |     | NULL    |                |
| sort_name_format                 | text             | YES  |     | NULL    |                |
| address_standardization_provider | varchar(64)      | YES  |     | NULL    |                |
| address_standardization_userid   | varchar(64)      | YES  |     | NULL    |                |
| address_standardization_url      | varchar(255)     | YES  |     | NULL    |                |
| editor_id                        | int(10) unsigned | YES  |     | NULL    |                |
| mailing_backend                  | text             | YES  |     | NULL    |                |
| navigation                       | text             | YES  |     | NULL    |                |
| contact_autocomplete_options     | varchar(255)     | YES  |     | NULL    |                |
+----------------------------------+------------------+------+-----+---------+----------------+
20 rows in set (0.00 sec)


If I'm understanding you correctly, you'd want me to add another slot and handle it similar to mailing_backend, i.e., serialize the memcache related cache params and add the needed code to CRM_Core_BAO_Preferences?

UI would be under Global Settings;  I'm inclined to call it "Cache" since if you don't know what a cache is, you won't want to worry about these settings.

Default settings should behave exactly as things have since 1.7, i.e., if somebody sets the define in civicrm.settings.php, they will get the same "set it and forget it" behavior.  If they want the new behavior, they'll need to either use the Settings UI or munge the domain pref record accordingly.

AFAICT, if I access the prefs via the BAO interface, it's never cached, so I can't get into a chicken and egg situation.

That about cover it?

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Making Memcache support more configurable
February 12, 2011, 10:47:41 pm
Quote
1. Implement the host, port and timeout as config variables, so different installs can connect to a different memcache server

2. Add a new "prefix" to the above config, so all "keys" for that memcache are uniquely prefixed. This will allow the same memcache to be used by different installs. You can potentially do this automatically by md5'ing the dsn (or base url or both)

There turns out to be a subtlety here, and now that I've done a first pass, I see it  ::)

In order to get at the database, CRM_Core_Config::_initialize() needs to get called.  But the Config object may be cached, so as currently written, you can't access the settings for the cache out of the database.

There may be a way around this, but it'll take some thought.  It isn't clear to me why the Config object needs to actually initialize the database, even if the database settings are managed through the Config object.  But Lobo may know different.

If this isn't readily solvable, then of course we can put the settings into civicrm.settings.php.  But I'm not willing to admit defeat just yet.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Making Memcache support more configurable
February 13, 2011, 01:36:17 pm

a lot of the config is stored in the database and hence the need to have a valid DB object. In the memcache scenario, we basically store the config object in memcache

I suspect for the short term (at least) using the settings file is probably the easiest way out of this

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Making Memcache support more configurable
February 13, 2011, 01:41:12 pm
Yeah,  the setting file is the way to do this.  When the cache comes up, it's too early in Civi-Time to hit the database.  The only way around I could see would be to have the cache open the database independently of the DAO framework.  Definitely killing a mosquito with a heat seeking missile.  Slight overkill, you know.

I'll have a patch up based upon using the setting file today.  While I did have fun putting together an admin interface for this, well, chalk that off to future knowledge of the platform :-)

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Making Memcache support more configurable
February 15, 2011, 04:46:56 pm
Patch is posted to queue as CRM-7552: Make Memcached support configurable.

Should be backward compatible; documentation will be in civicrm.settings.php.

Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Scalability (Moderator: Donald Lobo) »
  • Making Memcache support more configurable

This forum was archived on 2017-11-26.