Support Forums

The forums are closed and will be removed when we launch our new site.

Looking for support? You can access the support system via your account.

Forums › Forums › Search & Filter Pro › Gracefully import 30000 posts

Tagged: , ,

Viewing 10 posts - 1 through 10 (of 27 total)
1 2 3
  • Grosvenor
    (Private)  September 15, 2016 at 5:42 pm #58217

    Hello,

    We are trying to import about 30 000 blog posts to our WordPress blog from a legacy system. Our ETL process inserts posts and all related metadata directly into WordPress Database. However we are running into issues with Search and Filter plugin handling these newly imported posts.
    Our understanding is that in order for the new posts to be searchable and filterable they need to be cached by the plugin. We tried 2 approaches to achieve this and both didn’t succeed.
    APPROACH #1
    1. Run ETL process.
    2. After all the posts and related metadata are inserted – go to Search And Filter Settings in Admin and click Rebuild Cache (or wait till triggered automatically).
    ISSUES:
    – Rebuild cache process was taking very long and got stuck on 77% after running for over 20 hours

    APPROACH #2
    1. Run ETL and after each inserted post programmatically add it to the search and filter cache as follows:
    $postCache = @new Search_Filter_Post_Cache();
    $postCache->setup_cached_search_forms();
    $postCache->update_post_cache($postid);
    ISSUES:
    – After the Process completed performance on the blog was severely impacted. Each call to the blog page with search and filter widget was timing out with the following errors:
    PHP Fatal error: Maximum execution time of 120 seconds exceeded in …wp-content\\plugins\\search-filter-pro\\public\\includes\\class-search-filter-cache.php on line 1149
    PHP Fatal error: Maximum execution time of 120 seconds exceeded in …wp-content\\plugins\\search-filter-pro\\public\\includes\\class-search-filter-cache.php on line 1161
    – After opening search and filter setting admin page rebuild cache process got triggered on it’s own which deleted all the records in wp_search_filter_cache table and restarted caching process from scratch

    Please advice on how to efficiently cache imported posts and make sure all of them are searchable\filterable without performance impact.

    Thank you

    Trevor Moderator
    (Private)  September 15, 2016 at 7:18 pm #58259

    Please accept some caveats before I reply:

    It is late here (2 hours past closing) and one should never reply when tired 😉
    30,000 blog posts is enormous. I doubt that we have tested it under that sort of load.
    Any import process effectively runs a loop. If you have access to the code, adding a small block of code at the end of each loop might help. See here.

    The resources you have available will be key. A PHP timeout error such as you see means that PHP is set to a fairly low and safe limit. This is so with PHP out of the box, because most hosts like it that way to preserve balance on their servers. A blog with that many posts ought to be on some sort of dedicated resource system. I am unsure how big though.

    What hosting is it on right now? Shared, reseller, VPS, Cloud, Dedicated ….. (the last three will have contractually available resource lists, so if it is one of them, what do you get for your money?

    Grosvenor
    (Private)  September 15, 2016 at 8:04 pm #58276

    Hi Trevor,

    Thanks for responding after hours.

    Regarding your suggestion to add individual post to the cache after each iteration: that seems like exactly what we did in APPROACH#2. Please refer my original post to see issues that we ran into with that.

    Our blog is hosted in AWS cloud on a t2.small instance.

    If we potentially switch to more resourceful server, which PHP settings do you recommend to change?

    Trevor Moderator
    (Private)  September 15, 2016 at 8:43 pm #58287
    This reply has been marked as private.
    Grosvenor
    (Private)  September 15, 2016 at 10:48 pm #58337
    This reply has been marked as private.
    Trevor Moderator
    (Private)  September 16, 2016 at 9:45 am #58360

    I am in the UK, so unless you are too, telephone may not be cost effective! I am not quite sure what you meant about Skype? You can’t use it outside of the work network, or can’t use it on the corporate network? Why not from home (or on a tablet with mic/headset in a Starbucks)?

    Grosvenor
    (Private)  September 21, 2016 at 10:19 pm #59386

    Hello again,

    So we were able to significantly lower the page load time by disabling Auto Count setting in S&F.
    For 30 000 posts the load time of a page with S&F widget went down from 6 minutes (!!) to 7 seconds.

    As per our our conversation, we are considering chunking our posts in smaller groups. We ran some performance tests for 5 000 posts with Auto Count disabled. Here are the results:
    Load page with S&F widget ~3.6 sec
    Perform a search ~27 seconds

    While this timings are looking much better, 27 seconds is way too slow for searching 5 thousand posts.
    Is there anything else we are missing?

    Some info about our environment:

    Physical server: AWS t2.small; 2 GB memory; CPU 1 core up to 3.3 GHz
    OS: Windows Server 2012
    Webserver: IIS 8.5
    PHP 5.6.22
    Wordpress 4.6.1
    S&F Pro 4.6.1
    MySQL 5.6.27

    php max_memory_limit 256M;
    WP_MEMORY_LIMIT 256M;
    WP_MAX_MEMORY_LIMIT 256M;

    Snippet from S&F System Status:
    Database
    Table wp_search_filter_cache Exists: ✔ 204875 Rows Found
    Table wp_search_filter_term_results Exists: ✔ 9916 Rows Found
    Total Number of Fields Cached: 4

    Grosvenor
    (Private)  September 22, 2016 at 2:02 am #59404

    UPDATE:
    We use “Search Everything” plugin to control which content should be searchable. Looking closer at its configuration we disabled “Search every custom field” setting because our custom fields are already searchable (thanks to S&F?). That decreased the search time drastically.

    5000 posts
    Load page with S&F widget: ~4 sec
    Perform search: ~8 sec

    30000 posts
    Load page with S&F widget: ~7 sec
    Perform search: ~18 sec

    Our blog is usable again, although these timings are still far from a great user experience. Any ideas on how to further improve search performance are much appreciated.

    Ross Moderator
    (Private)  September 22, 2016 at 1:53 pm #59534

    Hey Rob

    I’m looking at performance at the moment. You’re definitely at the higher end of usage for the plugins capabilites – I’ve tested regularly with 4k posts & autocount enabled, but speed is often affected by how many fields & options you have.

    Do you have any fields with a considerable amount of options (+100)?

    Thanks

    Grosvenor
    (Private)  September 23, 2016 at 3:32 pm #59756

    Hey Ross,

    Yes, we have 4 custom fields with the options count varying from few hundred to few thousand.

    We do need all of them be available though. Is there a more efficient way to handle such fields?

  • Author
    Posts
Viewing 10 posts - 1 through 10 (of 27 total)
1 2 3

The forum ‘Search & Filter Pro’ is closed to new topics and replies.