I'm not "sneaking in" variables. Skill is the only real factor in PvP, as the devs have already stated that it's possible for a skilled low-level player to take out an unskilled high level player, even with the gear disparity. Min/maxing may work for some, but if you spend all your time at spreadsheets (which is going to be hard, given how many variables there already are within builds), you're still not going to be able to beat someone who's just better than you.
But. Let's pretend that there would ever be an encounter between a number of players, all of exactly equal skill. I still don't see racial passives making any difference when compared with active ability selection, and I still don't see an Imperial caster being somehow "gimped" compared to a Breton. They're just going to have different advantages/disadvantages.