Forums › Forums › Search & Filter Pro › Gracefully import 30000 posts
Tagged: cache, Import, performance
- This topic has 26 replies, 4 voices, and was last updated 7 years, 11 months ago by
Trevor.
-
Anonymous(Private) September 15, 2016 at 5:42 pm #58217
Hello,
We are trying to import about 30 000 blog posts to our WordPress blog from a legacy system. Our ETL process inserts posts and all related metadata directly into WordPress Database. However we are running into issues with Search and Filter plugin handling these newly imported posts.
Our understanding is that in order for the new posts to be searchable and filterable they need to be cached by the plugin. We tried 2 approaches to achieve this and both didn’t succeed.
APPROACH #1
1. Run ETL process.
2. After all the posts and related metadata are inserted – go to Search And Filter Settings in Admin and click Rebuild Cache (or wait till triggered automatically).
ISSUES:
– Rebuild cache process was taking very long and got stuck on 77% after running for over 20 hoursAPPROACH #2
1. Run ETL and after each inserted post programmatically add it to the search and filter cache as follows:
$postCache = @new Search_Filter_Post_Cache();
$postCache->setup_cached_search_forms();
$postCache->update_post_cache($postid);
ISSUES:
– After the Process completed performance on the blog was severely impacted. Each call to the blog page with search and filter widget was timing out with the following errors:
PHP Fatal error: Maximum execution time of 120 seconds exceeded in …wp-content\\plugins\\search-filter-pro\\public\\includes\\class-search-filter-cache.php on line 1149
PHP Fatal error: Maximum execution time of 120 seconds exceeded in …wp-content\\plugins\\search-filter-pro\\public\\includes\\class-search-filter-cache.php on line 1161
– After opening search and filter setting admin page rebuild cache process got triggered on it’s own which deleted all the records in wp_search_filter_cache table and restarted caching process from scratchPlease advice on how to efficiently cache imported posts and make sure all of them are searchable\filterable without performance impact.
Thank you
Trevor(Private) September 15, 2016 at 7:18 pm #58259Please accept some caveats before I reply:
It is late here (2 hours past closing) and one should never reply when tired 😉
30,000 blog posts is enormous. I doubt that we have tested it under that sort of load.
Any import process effectively runs a loop. If you have access to the code, adding a small block of code at the end of each loop might help. See here.The resources you have available will be key. A PHP timeout error such as you see means that PHP is set to a fairly low and safe limit. This is so with PHP out of the box, because most hosts like it that way to preserve balance on their servers. A blog with that many posts ought to be on some sort of dedicated resource system. I am unsure how big though.
What hosting is it on right now? Shared, reseller, VPS, Cloud, Dedicated ….. (the last three will have contractually available resource lists, so if it is one of them, what do you get for your money?
Anonymous(Private) September 15, 2016 at 8:04 pm #58276Hi Trevor,
Thanks for responding after hours.
Regarding your suggestion to add individual post to the cache after each iteration: that seems like exactly what we did in APPROACH#2. Please refer my original post to see issues that we ran into with that.
Our blog is hosted in AWS cloud on a t2.small instance.
If we potentially switch to more resourceful server, which PHP settings do you recommend to change?
Trevor(Private) September 16, 2016 at 9:45 am #58360I am in the UK, so unless you are too, telephone may not be cost effective! I am not quite sure what you meant about Skype? You can’t use it outside of the work network, or can’t use it on the corporate network? Why not from home (or on a tablet with mic/headset in a Starbucks)?
Anonymous(Private) September 21, 2016 at 10:19 pm #59386Hello again,
So we were able to significantly lower the page load time by disabling Auto Count setting in S&F.
For 30 000 posts the load time of a page with S&F widget went down from 6 minutes (!!) to 7 seconds.As per our our conversation, we are considering chunking our posts in smaller groups. We ran some performance tests for 5 000 posts with Auto Count disabled. Here are the results:
Load page with S&F widget ~3.6 sec
Perform a search ~27 secondsWhile this timings are looking much better, 27 seconds is way too slow for searching 5 thousand posts.
Is there anything else we are missing?Some info about our environment:
Physical server: AWS t2.small; 2 GB memory; CPU 1 core up to 3.3 GHz
OS: Windows Server 2012
Webserver: IIS 8.5
PHP 5.6.22
Wordpress 4.6.1
S&F Pro 4.6.1
MySQL 5.6.27php max_memory_limit 256M;
WP_MEMORY_LIMIT 256M;
WP_MAX_MEMORY_LIMIT 256M;Snippet from S&F System Status:
Database
Tablewp_search_filter_cacheExists: ✔ 204875 Rows Found
Tablewp_search_filter_term_resultsExists: ✔ 9916 Rows Found
Total Number of Fields Cached: 4Anonymous(Private) September 22, 2016 at 2:02 am #59404UPDATE:
We use “Search Everything” plugin to control which content should be searchable. Looking closer at its configuration we disabled “Search every custom field” setting because our custom fields are already searchable (thanks to S&F?). That decreased the search time drastically.5000 posts
Load page with S&F widget: ~4 sec
Perform search: ~8 sec30000 posts
Load page with S&F widget: ~7 sec
Perform search: ~18 secOur blog is usable again, although these timings are still far from a great user experience. Any ideas on how to further improve search performance are much appreciated.
Ross Moderator(Private) September 22, 2016 at 1:53 pm #59534Hey Rob
I’m looking at performance at the moment. You’re definitely at the higher end of usage for the plugins capabilites – I’ve tested regularly with 4k posts & autocount enabled, but speed is often affected by how many fields & options you have.
Do you have any fields with a considerable amount of options (+100)?
Thanks
-
AuthorPosts
