As the story goes on, Xerxes starts taking more responsibility. Right after the thing at Alanoc happened, he wasn't fully aware of the scale of such an attack, as he was still mentally on the 'honeymoon' of having achieved his dream of getting together an adventuring party. Basically, Volume 1 (Chapters 1 through 4) is Xerxes' character arc around the theme of maturing. Not that he's ever going to become super serious, just evolve from 'some guy' to 'leader'.