Just wondering because when I come across them they get mad if I get too close but I usually have a follower with me. I noticed that my husband doesn't ever get Giants getting mad when he gets close to them and he never has a follower.

Ask your companion to wait at a city and go meet the giants. I'm almost 30hrs in and I've never met one. Where do they roam at?

They have camps all over. Typically near Mammoths. Look for the mammoth icon on your map.

camp giants are agro. the roaming ones not

they are more territorial unless you can take on giants stay away in general