Covetous warpers can only warp to the location of the item they covet. This means that if you are carrying it, they can warp to you all they want, but they can’t warp back and forth to the stairs.