Exactly. If Bethesda wanted to realistically introduce guns, they'd have to nix armor, or use some stupid dues ex machina as to why armor still works. Becuase in the real world, early guns spelled the end of armor.
If you want new technology, have repeating crossbows and such. Not guns, becuase that will change combat. Besides, who's to say that there is even saltpeter in Tamriel?
I don't think it would be game breaking.
For one, the real world has no magical armor which can provide magical degrees of protection. Even as it stands right now in Elder Scrolls, the more mundane armors (Fur, Leather, Iron, Steel) which have real world equivalents are some of the lowest quality available and pretty much everything punches through them anyhow, so the less realistic (Elven, Orcish, Glass, Deadric) armors are already seen as far superior; so it's safe to assume that they could offer some protection against low-velocity fire arms.
Second, the accuracy or early fire-arms was atrocious. Simply equip them (in-game) with a wide targetting reticule, as opposed to the precision of the bow and they'd be nearly useless at long range.
Third, they take forever to reload. So what you'd expect is maybe one good shot at close range, and then switch to a melee weapon; otherwise you risk being pounded to death while reloading. Sure, higher levels in the firearm (or appropriate) skill set could reduce reload times somewhat, but still, I don't see them as a primary weapon.
I think they'd fit in fine.