I know the bitter coast region is supposed to be good but I find the caves/ ruins are just a bit too hard. Is this where most of the easy caves are? At the moment I find myself saving before I go into a cave, going in, getting killed, then loading the saved game. I have to try out about 10 caves before I find anything do-able but even then I'm not able to clear out the whole cave. And I like to find my armour, not buy it...

I've just started the game, but the cave near Seyda Neen is easy. You won't find much for armor there though, I don't think. There's also an easy tomb near Seyda Neen that I haven't fully explored. In short I would look near Seyda Neen, but I know what you mean. I just walked into a cave with some Nord on the way to Balmora and I must have takes 300 of that guys fatigue away and hit him 20 times, but he was barely affected and killed me in 3 hits =x

If you're struggling at the start of the game, you may want to avoid most caves since they usually have NPCs in them. NPCs are not leveled to you in Morrowind, so you can find some tough people at level 1.
Instead, search http://www.uesp.net/wiki/Morrowind:Ancestral_Tombs. They usually just have creatures in them (which are leveled to you in most cases,) so you have better chances of fighting enemies that are leveled to your low level (ghosts and basic skeletons.)

Forget about poking into random caves before level 10...just do guild quests and level up. You'll get money and gear more easily this way and stick to places your character can handle, since the guild system is built to ensure a weak character can't do a quest above his skill level.

Have two "easy" caves sort of, on the way from balmora to pelagiad you have two caves to the right side of the path/way,if I remeber correctly the first one you encounter is slightly easier,the other cave later on the way to pelagiad has a nord with a two handed weapon,not good Do a save first just in case my memory fails.

Gnisis is easier,from the fort and the "normal" way,not the backpath,stay away from the first cave until much later,the next one further on the same way is kind of easy in comparision.

In both of the easy caves there is loot to match= not that good but some of it is it.Still recommend picking up some potions from the guilds though.They aren′t that easy for level 4