For me, it is any armor skill. They always seem to lag behind the rest of my skills. I work at NOT getting hit. I also like to take down opponents as quickly as possible. If I fall too far behind, I find myself some mudcrabs and just heal as they hit me for awhile, but that is so unnatural. I don't think I have "legendary'd out" any armor skill yet.
What are the skill(s) you find difficult to level up?