I figured out that the main problem is that WordPress doesn’t seem to update term-relationships for terms in the post type ‘attachment’. I think that’s the main problem and that’s not really related to Search and Filter.

I wrote an SQL statement to update the count, and added it to the hooks ‘add_attachment’ and ‘edit_attachment’:
“UPDATE wp_term_taxonomy SET count = (
SELECT COUNT(*) FROM wp_term_relationships rel
LEFT JOIN wp_posts po ON (po.ID = rel.object_id)
WHERE
rel.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id
)”

This works, the count on the taxonomy page seems correct now. However, I added some extra filters in the ‘Post Meta’ tab of Search & Filter. The filter counters in Search & Filter don’t seem to take those into account. Do you notice this as well?