Like I did the Battlehorn Castle stuff and bought everything, is every container in the castle safe to use for long term storage or?

I don't think there's really a universal way to tell if a container is safe or not. The majority of containers in a player-owned house will be safe.

For Battlehorn Castle, you can find a list of safe containers http://www.uesp.net/wiki/Oblivion:Battlehorn_Castle_(place)#Safe_Containers.

Cool, thanks for the list. Can finally ditch a ton of these notes and stuff I've been carrying around for eons.

Any torn bags you can across will also store your stuff indefinetly.