With new lore info, could we conclude that the Totem was made so it could react to the Dragonborn? Maybe Talos thought he was the only Dragonborn and thus wrote that requirement thinking that only his children will be able to use it.
Which would also mean that: Gorthwog, Mannimarco, Zurin Arctus and the three royal families are also Dragonborn.