Ghosts lose their physical touch attack in favor of a reusable attack that scare-paralyzes you. Possibly, if this is unflavorful due to it just being the same ghost, change their AI to make them turn invisible after a certain amount of time being visible and not next to the player. Then they path towards you, and if you either stumble into them or they are next to you and get a move, they turn visible, paralyzing you. Since the actual action of them appearing out of nowhere would be what frightens you, see invisible would negate the effect.