After a few years of saying the name it recently occurred to me that I might actually be saying the name wrong, what is the proper way to say this, for some odd reason to me, elusive word?

Far as i know it's the former. Ehl-No-Fey.

There is this one quest in ESO which features them or at least some aspects of them. Perhaps the dialogue will help you. Find it https://youtu.be/AAt9WKu1K34?t=22m8s

Thank you for this. I feel much better in knowing that Ive been saying right