Large database – omit elements to cache

Anonymous

#194338

We have a large database of products (about 1 million records in post_meta).

Once the database is cached, S&F works well. But it takes a really long time to build the cache.

We noticed in the S&F DB table that there are elements cached that we don’t need, such as ‘shipping class’, description and some variations we created ourselves.

Is it possible to use a hook or another way to remove elements from the cache so the indexing will go quicker?

Ross Moderator

#194339

Hi Stijn

There might be a way to help with this (but nothing official). Just so you know S&F will only cache taxonomies to the post types you are using.

Before I dig that out, can I ask, why you are finding yourself rebuilding the whole DB?

When new posts are added, it should add the cache for those automatically.

Thanks

Anonymous

#194340

Hi Ross,

We are setting up a wholesale Woocommerce store.

Once per season (2 times a year) all products needs to be imported (we are using WP All Import for this task). Because the products are imported the ‘save post’-hook is not fired, so S&F does not detect that there is a new post.

Thats why we manually need to trigger the indexing.

Ross Moderator

#194428

Hi Stijn

So, I dug out a bit of code that should help.

Just to note, this is pretty crude, and undocumented, and likely to change in v3 (if we remove it, we will replace it with another way of doing this).

Anyway, for now, this seems to work:

add_filter('search_filter_post_cache_insert_data', 'search_filter_post_cache_insert_data', 100, 3); //

function search_filter_post_cache_insert_data($insert_data, $post_id, $type){
	
	if(isset($insert_data['_sft_taxonomy_name'])){
		unset($insert_data['_sft_taxonomy_name']);
	}
	
	return $insert_data;
}

You can replace _sft_taxonomy_name with any field name, for meta it will be _sfm_something but assuming its those automatically included taxonomies you’re not using, which you want to remove.

If you want to find out exactly what is being cached when it runs, you can do something like:

add_filter('search_filter_post_cache_insert_data', 'search_filter_post_cache_insert_data', 100, 3); //

function search_filter_post_cache_insert_data($insert_data, $post_id, $type){
	
	if($type=="taxonomy"){
		error_log( print_r( $insert_data, true ) );
	}
	
	return $insert_data;
}

With WP_DEBUG and WP_DEBUG_LOG enabled, you’ll find the output of this in wp-content/debug.log

Just remember when rebuilding the cache, this filter runs for every post, so its going to add a an entry per post to your debug which will get big fast – you can test what data is being added on a single post, by going to a product and just clicking “update”.

I hope that helps.

Thanks

Anonymous

#194435

Thank you very much for this input.

We’ll give it a go tomorrow.

We are also looking into filtering only on the parent products and not the variations since the variations make the DB really big. (and save some extra meta in the parent product)

We’ll let you know if we have any additional questions

Thanks!

[Author]

Posts