Smarty caching efficiency

[ChrisChinchilla]

Hey everyone,

We keep hitting a major issue with smarty when it brings our servers to a grinding halt on cache writes (as it writes to the file system), we have quite a high traffic site, but I would think others have experienced this, over the next few days we are looking into some ideas to solve this, but i had a few questions:

• Is this a problem with Smarty of CiviCRM's implementation of Smarty?
• For our own usage, we're thinking of trying to redirect the file write to write to Redis instead, has anyone else tried something like this?
• Any other thoughts or ideas?

Anything we come up with will be contributed back if it can be.

Thanks, Chris

[totten]

1. It's not supposed to write frequently. It's only supposed to write when you first use a (new/modified) *.tpl file. It sounds like maybe the system is generating false cache-misses on a systemic basis?

2. Storing in a redis is an interesting idea. I don't have any experience with redis+php, but my gut says you'd get the same benefit by just turning off Smarty caching entirely. The general goal of Smarty's caching system is to compile the *.tpl down to *.php and then allow the PHP runtime to apply furter optimizations (eg XCache, APC). If you have Smarty compile the *.tpl down to *.php code and then store the *.php code in Redis, then you would still need to have the PHP runtime evaluate the code. Short of writing a PECL extension or a core PHP patch, I think you'd wind up with something like this pseudocode:

Code: [Select]

$cachedTpl = $redis->get('the-compiled-template');
if (!$cachedTpl)
  $cachedTpl = $smarty->compile('the-raw-template');
  $redis->put('the-compiled-template', $cachedTpl);
eval("$cachedTpl");

Running through eval() is probably going to nullify much of the benefit of Smarty's caching architecture.

If the redis arrangement does help, it's probably because you're skipping the disk IO during false-cache-misses. But disabling caching entirely would produce the same benefit.

3. Can you determine if it's writing all templates -- or specific templates?

4. There are a few possibilities that come to mind:

4a. If there's some other process (eg a cron job or custom module) which flushes the cache frequently, then that would undermine the Smarty caching. You might investigate this adding some logging/monitoring to CRM_Core_Config::cleanup().

4b. A year or so back, I heard about an interesting case with CiviMail -- if you enabled Smarty support, it would compile a slightly different Smarty template for each outbound email. Micah was working on it at the time; I don't know if it was resolved.

4c. I think Smarty relies of file timestamps to assist in deciding whether to use a cached *.php or recompile the *.tpl to *.php. If there's something weird with the clock or filesystem, it could theoretically break the caching behavior.

[ChrisChinchilla]

I may have worded my original post slightly poorly, it doesn't write all the time / frequently, but when it does it hits server performance massively. So our main reasoning is stopping it writing cache to disk completely, like many other cache systems and have it write to memory instead.

This is sort of what you've touched upon in your very useful comments. i guess in summary our desire is to make the caching work better at the scale we're running things, maybe our initial thoughts are not the best way to achieve this, but I will investigate further and keep you posted.

Thanks, Chris

[Donald Lobo]

i think micah and the eff guys put the smarty template cache directory in a memory file system.

no mods of code required and u get a fair speedup

lobo

[ChrisChinchilla]

Cool! You know if they documented it anywhere?

[herb]

Totten, you mention ChrisChinchilla may be better off just turning of Smarty caching entirely. Just how would this be done? I'm trying to debug an issue and would like to see how it runs if the templates aren't compiled.

Thanks!

[totten]

If you're talking about my point #2... that was just a general idea. Never tried it and don't know steps. In skimming Smarty.class.php, the closest option I can see is "$force_compile" -- which doesn't sound quite like what I had in mind, but maybe it's what you had in mind. One would generally change/override options in Smarty by patching CRM/Core/Smarty.php or using hook_civicrm_config.

If you're talking about point #4b, IIRC CiviMail+Smarty is disabled by default. It's controlled by a define() in civicrm.settings.php.