Replace the whole speed system with the one from Tales of Maj’Eyal, another roguelike. This has a strict concept of a “global speed”, i.e. a number of movement points you are given every global turn. Every possible action has an associated movement point cost, which allows for things like zapping a wand to take less time and be done more frequently than making a melee attack. This gets rid of the weird existing system where, in ToME terms, your most recent action defines how much global speed you get, particularly if riding is involved.

One way to start implementing this is to have all the action functions that currently return 0 for no time taken or 1 for time taken to instead return a bitmask of all types of actions taken: “moved”, “attacked”, “cast a spell”, etc. Then the main loop code that handles these return values deducts movement points corresponding to the slowest action in this bitmask.