A hard part about setting hard limits to magic in your narrative is trying not to rob it of its whimsy. You also don't want to have a magic system that breaks any tension in your world.
And then you have the option to write passive-aggressively and give yourself an out.