Sometime, i saw in shop, item on counter... like upgrade for rostetome hall (sorry for the house name, i am just lazy to go check in my oblivion game or go on uesp), and even armor...

How to do that, since normally, NPC stock is in container? How to make a NPC sell an item not in container?

I am pretty sure they can sell anything in the room, the chests are basically just a place for them to store stuff they buy from you. I wont swear by it, but try just placing an item in the shop and see if he sells it.
Stuff not in a respawning chest will of course not respawn, so it will only be a one time sell.

Correct, but make sure he owns the items, either directly, or because they inherit ownership from the cell they're in. Almost all vendors will have a shop that they own, but there are exceptions. Obviously Shady Sam is one, as he isn't in an interior cell at all. He can still sell thing near him if he owns them, so they have explicit ownership set in the CS.

An FWIW, the items they buy go directly into their own inventory. I seem to remember issues where some would equip items, and you couldn't buy them back, but I don't recall the circumstances. The chests are, as you suggest, to support respawning, as well as to allow different chests to be enabled/disabled to change the vendor's range of goods.

Ah thanks for clearing it up didn't realize it went to there inventory, but yeah makes sense, i remember seeing some npcs equip items i sold em.