I really think... if you are, infact talking about specializations (aka, Combat, Stealth, and Magic)... the only difference is the RATE at which those level. I think they level up faster or something.
Once again, not a big deal in vanilla (Not that I have experienced)
However, I see in other posts that you have specific mods. I cannot help you much in those cases and someone else more familiar with the mods may be able to help