Casting spells decreases your spell memory by a small amount but one that would be significant if you cast the spell heavily.