The architecture for the Tower, and the City are exactly the same. If the Ayleids made the tower, then you'd at least half-expect the Elder Council chamber to look like a giant Ayleid tomb, complete with all those glowing crystals.
Are we supposed to believe that because there are a few dragons painted around that it's all Imperial?