A Silver weapon will hurt a ghost, but yes... you do need to be skilled in the weapon type to wield it efficiently.
Keep in mind there is a difference between Short and Long Swords. (as well as all the other type of weapon Skills that Oblivion didn't have, like Axe and Blunt being separate)
You need
at least a Skill level of 35-40 for a weapon to be used with any real efficiency. You also need to make sure your Fatigue is not empty... or it can be very difficult to hit anything (especially at low levels when your Agility is likely low.)
To clarify, Daedra and most Undead are immune to Normal weapons (Wood, Iron, Steel, etc). They can only be hurt with Enchanted weapons, or Silver, Glass, Daedric, Ebony.
- Make sure the weapon you're using is at a Skill of around 40, or you'll miss a lot.
- keep your Fatigue full