Gracefully import 30000 posts

Ross Moderator

#60518

Hi Rob

Thanks for your extensive research in to this – you’re really getting into the nitty gritty of S&F code 🙂

I’ve noticed recently that fields with a high number of options really seem to slow things down, which of course means some bad optimizations – I think you’ve highlighted one above.

Most of what you mention makes sense, I’m working on these areas now and taking a look where things can be improved…

What software/plugin do you use for your performance testing? I’ve been using query monitor – https://en-gb.wordpress.org/plugins/query-monitor/ but generally testing with datasets much smaller than yours.

Best

Ross Moderator

#60522

BTW, as a better way to handle large fields, it might be best to have a autocomplete field, where a user starts typing and gets a list of suggestions? Do you think that would work in your scenario?

This is something I’m planning on adding in the very near future.

Best

Anonymous

#60645

Hi Ross,

Glad to hear my input was useful. I mainly just use Chrome DevTools and MySQL’s native Performance Schema via MySQL Workbench for my performance testing.

One other thing that helped to speed up the search query by about 15% is creating a FULLTEXT index on wp_postmeta(meta_value) and using it in a query. So if you could include this feature in your upcoming performance fix – that’d be great.

Also could you please kindly provide a timeframe for these improvements to go live? We’ve put some good effort to tune up our Search & Filter widget and would like to stick to it, at the same time we have our own deadlines to meet.

As far as autocomplete solution – we already partially employ it. The list of available options gets filtered as user types, although if nothing is typed in we prefer all the options to be visible for the user, so they know what they can pick from.

Thanks!

Ross Moderator

#61999

Hey Rob

Just to let you know I’ve not forgotten about this, been working on tweaking performance for a while now, testing in different environments etc etc.. still a fair bit to go…

RE the FULLTEXT, do you mean putting on a column in the S&F tables? Because this is where the query is mostly performed, not on the WP Posts table :/

Best

Anonymous

#65138

Hi Ross,

I meant introducing a FULLTEXT index for tables with extensive text values that being searched against. In our case it is wp_postmeta because that’s where we store the contents of our custom post type.
So we ended up building a FULLTEXT index on that table and intercepting all WordPress search queries to modify them to use the index – e. g. replace LIKE with MATCH AGAINST.
At the same time we are excluding unneeded meta fields from the query as mentioned above.

Also I realized that one of the biggest bottlenecks on a page load was generating html for the dropdowns with thousands of items. So I modified class-search-filter-generate-input.php to not do that for fields with certain flag and enqueueing javascripts for lazy load instead. It sounds like something like this will be supported in out of the box in the next release of SF – so will be excited to see that.

One other feature that would be really handy is so called archiving. Like being able to limit search and filter to only posts that satisfy certain condition. For example only posts that were created within last 5 years. And then search, filter and generate filter options accordingly.

Ross Moderator

#71095

Hi Rob

I’ve made quite a few changes & speed improvements… I’ve just sent you the beta in case you want to test, this will likely be the next release after a bug finding mission.

Thanks

Anonymous

#79045

Hey Ross, where should I look for beta?

Trevor

#79056

Gosh, that was 6 weeks ago. It would have been sent to the email address you registered with and, as it has a zip attachment, almost certainly into spam or junk, and probably long since deleted 🙁

Is that address OK to use again?

Anonymous

#79168

This reply has been marked as private.

Trevor

#79172

Sent. Check spam and junk folders.

[Author]

Posts