An invisible displacer beast that you can’t see should not be able to use its displacement abilities in combat with you. (If you have see invisible, it should still be able to use them.)