went to armorer to barter. in combat gear (cha 7), i could buy a piece of armor for 63. put on cha outfit (cha now 12) same piece of armor now 50. drop some graqe mentats (cha 17) 10% buy/sell bonus. same piece of armor still 50. is there a cap on the max discount i can get?

From my own little testing, anything beyond 12 is "wasted". I did my testing in Diamond City with Arturo. My Charisma is 9 and I wear clothing that boost it to 12. I have Cap Collector level 2 which I think also has an effect on prices as well. If I were to buy the power armor frame it would cost me 2,400 caps. I boosted my Chr using graqe mentats and the price was still 2,400 caps.

From everything I've heard yes this is a cap for buying and selling. I'm not sure what each cap is for buying or selling, but I heard that for selling it's 80% of value is the top.

12 cha with all the junktowns and all money perks; fusion cores sell for 160, buy for 240.

If we assume Fusion Cores are 200 Value, then it looks like 80% for selling and 120% for buying

Well you can't honestly expect the merchants to just give you their wares because you wear a nice hat and use drugs. They have to eat too you know.