The Amulet drains your power when you cast spells, even if you’re not carrying it, if it’s on the ground nearby. The amount of power drained is proportional to your closeness to it.