Hopefully I'm worrying about nothing.
I'd say that this is correct. You're worrying about nothing. The only real flaw I see is your inability to deduct that Stealth does indeed have OFFENSIVE OPTIONS. Granted, you might slowly raise a weapon skill along with those offensive options, but is that really an issue? To me, that's akin to non-combat specialized characters running or swimming in the previous games. Running or swimming in the previous games increased Athletics. Did your stealth characters not run in the previous games?
As for Guardian Stones, only three have been revealed. For all we know, there could be a Guardian Stone that increases the skill up rate for your selection of stealth skills and weapon skills.
At this rate, I wonder how long is it before "Warriors" start complaining that combat specialized skills don't have a way to open locks for better loot.
PS.
+30% means absolutely nothing when we don't what the base number is. I.e. 1.0 -> 1.3 or 10 -> 13. The only time any of those percentages meant anything to me was when it was concerning Athletics. Why? Because that skill is a pain in the ass to raise manually, taking 70+ hours of swimming without any specialization or major skill increase.