WHAT IS THE BUILD?
An unarmed Khajiit build would go as follows: obviously, you'd be a khajiit. You would max out heavy armor, including getting the important Fists of Steel perk.
With this perk, your armor rating for your gauntlets is added to your damage. I am not sure whether the addition is 1 to 1 or whether it is a fraction thereof, but it does increase the damage when you have better gauntlets. As such, with this build, you obviously want to max out smithing early and get some Legendary quality Dragonplate or Daedric gauntlets in order to maximize your damage.
I think the big concern with unarmed specialization is whether it can compete in damage output at later parts in the game. The concern is that there are no enchantments on your hands and that there is no unarmed skill, so you don't have lots of ways to improve your damage. However, I believe it can compete in damage with other weapon types. Here's how:
WHY THE DAMAGE IS SIMILAR
Damage for normal weapons goes like this. The weapon you have has a base damage amount. This base damage is increased based on how many points you have in your relevant skill. It is also increased based on improving the weapon (ie. making it fine, exquisite, etc). You can also put damage enchantments on the weapon. Perks can increase your damage, and lastly, you can enchant other items to increase your damage (ie. fortify one handed damage enchantment).
The question, then, is whether an unarmed khajiit can compete with this damage. It can. When you have Fists of Steel, the equivalent to the base damage amount of a weapon is the base armor rating, and thus base damage increase, of your gauntlets. This base damage from your gauntlets is increased based on how many points you have in heavy armor. It is also increased based on improving the gauntlets. You cannot put damage enchantments on your fists. However, the khajiit gets 15 damage in unarmed attacks. This is, for all intents and purposes, a permanent enchantment bonus. It is unclear to me whether the khajiit bonus is +15 damage above and beyond the base damage (which may be like 8 or something) or whether it is just 15 instead of like 8. If it is above and beyond, then the total damage here can certainly compete with virtually any enchantment. You also have perks. The Juggernaut perk increases your damage just like the first perk of the weapon skills do. Furthermore, the Matching Set and Well Fitted perks increase your armor ratings by 25% each (and i believe this increases unarmed damage if you have Fists of Steel), which is surely fairly equivalent to the increases in damage from the various possible perks you can get with one-handed and two-handed weapons (ie. stuff like extra bleeding damage and better power attacks). Lastly, you can also enchant items to increase damage in unarmed attacks.
So it seems to me that for every method a one-handed or two-handed warrior has to increase his or her damage, an unarmed khajiit has a similar and seemingly equivalent method to increase his damage too. Thus, the damage output is likely fairly equivalent even at higher levels. The main difference is likely to be that unarmed attacks have a shorter range of attack, but a faster rate of attack. That is just a trade-off though.
ADVANTAGES TO UNARMED
With that said, I think going unarmed has a lot of interesting advantages:
1. EFFICIENT USE OF TRAINERS
When you create a warrior type character, you inevitably are going to want to have a high skill level in whatever type of armor type you are using AS WELL AS whatever type of weapon you are using. You want to deal damage AND not take much. This means that your optimal character will eventually require a high rating in two skills. An unarmed character merges his weapon and armor skill together. This means he must only get a high rating in one skill. Now, I do not believe this will result in higher skill level up from battles themselves (unarmed attacks do not increase heavy armor I don't think, so you're not actually increasing heavy armor faster than another warrior would). However, this does allow for one efficient tactic: that is, using a heavy armor trainer increases your damage and defense. That means you get more bang for your buck with trainers than another warrior possibly could.
2. EFFICIENT LEVELING UP
Along the same lines, you get more efficient leveling by only having one skill. You increase in level based on increases in skills. While the game is not quite like Oblivion in this regard, enemies do level up with you. An unarmed character can be just as powerful as a typical warrior while being a much lower level (and thus, facing weaker foes). This is easy to illustrate. Lets say an unarmed character starts with a 20 in heavy armor, while another character starts with 20 in one handed and 20 in heavy armor. They are equal in power now. In order to get up to 100 in both their defense and attack skills, the unarmed character will only level up 8 times, compared to 16 for the one handed character. This translates into being more powerful relative to your foes.
3. REQUIRES LESS USAGE OF PERKS
One last bonus related to combining attack and defense into one skill is related to perks. Most of the Heavy Armor perks will increase both armor rating AND damage for an unarmed character. This means that this character can have his optimal damage and armor while only getting one set of perks. A similar warrior using a weapon would require two sets of perks to get the same effect. This means the unarmed character can afford to invest in a whole bunch of other perks elsewhere that the weapon user would not be able to get. This is a big deal.
4. LESS WEIGHT USAGE
One of the most important things, to me, about going unarmed is the fact that you do not have to carry around a ton of weapons. Weapons weigh a lot. I know you can store stuff in a house or something and not have to worry about weight much, but the reality is that a large portion of players will struggle with the weight limit for much of the game. Using weapons, especially two handed ones, will squeeze you in this regard, especially if you carry around more than one weapon. Unarmed characters do not have to worry about this. They do not have any need for weapons, so they can carry more of other stuff. This is very convenient.
5. CHEAPER
Another consequence of not having to have weapons is that you inevitably end up with more gold due to this. Weapon users must either buy or find their weapons. Unarmed characters never use gold buying weapons and they can always sell the valuable weapons they find, instead of keeping them around in case they want to use them. The result is more gold, which cane be used on other stuff.
6. LESS USE OF VALUABLE MATERIALS
This is a relatively small issue, but you also end up being able to use various important objects on other things. For instance, let's take soul gems. A player might have one filled soul gem. If they have a weapon, they must choose whether to use it on a weapon enchantment, to refill a weapon enchant, or to enchant armor. The unarmed player can do similar damage as the player with an enchanted weapon but use the soul gem on the armor. This isn't a huge deal eventually because you will likely get a soul trap spell and Azura's Star, making soul gems relatively worthless, but it does matter early on. Similarly, you might have a limited amount of a crafting item. The unarmed player never has to choose whether to use an ingot to create/improve his armor or his damage; they are one in the same. A weapon user simply requires more crafting items to be optimally outfitted. Again, this won't matter too much when you're rich, but it will matter in the early game when your finances aren't quite as stable. Lastly, a weapon user will likely need to disenchant items with damage enchantments early on to get those enchantments. An unarmed player can arguably do without those enchantments and can simply sell the relatively valuable items. Again, this is a boon to your early game economic situation.
CONCLUSION
Thus, in my mind the khajiit unarmed is a great option. You will do similar damage as a weapon user (in fact, you'll do more early on when the +15 bonus is particularly massive). At the same time, you will level up more efficiently (due to efficient use of trainers and leveling up less while being as powerful), have to use fewer perks, have less issues with weight, and have more gold.