The demon lords take no interest in you until you are on the ascension run with the Amulet. At which point they attack.