To provide a ‘normal’ page, a well configured VPS does not touch the actual database (on the disc drive) very much, and so the response can easily be a few milliseconds. Search and Filter has to use the database, and so if a server has a bottleneck at that point, and the processing of that data, then you can see huge delays as the page has to wait for enough server resources to become available. 20-30 seconds is not an unusual wait under such circumstances.
A VPS can be slow. Having a VPS does not guarantee speed. Configuring these type of servers is a dark art. I have seen a site that suffered delays such as you see, switching to a different host to see the page load in 95-98% less time. It can vary that much.
.
