No complaints about difficulty drop, templars are one of the most formidable generic enemies one can fight and combined with a higher demon, fights like these are extremely tough, especially on first playthrough (when you don't know the resistances / immunities of enemies, which early talents are insanely good etc.).
I remember these tower encounters very vividly even from second playthrough. I bet you would eventually make it through even on hard with some advanced tactical tricks and babysitting everyone every second, but it is a healthier choice for both you and the viewers :D