Any hostile NPCs/animals can still know where I am and start attacking me whenever I walk near them.
I even have an enchanted shoe with muffle effect.

I thought with Invisibility + Muffle, nobody should be able to detect me at all. Otherwise, what is the point of having this spell?

What that puzzles me the most is high level Sneak allows you to sit in front of people and they won't even know you are there.
But people can see you even though, with Invisibility Spell, you are supposed to be transparent to their eyes.

Yes. Since I reached level XY the game gives me invisibility potions. I used one when I was bored and tried to murder the Jarl of Whiterun but the guards always knew where I was and even talked to me.

I doubt it's supposed to be that way. It either gets fixed by Beth or in January by the modders.

this spell does nothing but trouble.
whenever i cast invisibility i am detected right away, and NPCs start approaching me.
i think casting itself makes me detected, no matter that spell is invisibility or any other.
but if i use invisibility potion things get better - they don't detect me so easily.
still moving too close is like i'm visible entirely.

Casting a spell or shooting a bow makes noise, and enemies will move directly toward the source of that noise if they can't see the player (although if an arrow strikes a wall or floor, they'll move toward that instead). Without the Quiet Casting perk, enemies will hear you cast if you aren't far enough away. Invisibility isn't perfect — it makes you much more difficult to see, but not impossible, especially if enemies get close to you. Likewise, Muffle reduces armor and footstep noise, but doesn't make you totally silent. Only the Silence perk in Sneak does that.

Also, even with 100 Sneak + bonuses, in daylight outdoors most enemies can see you unless you use an Invisibility spell or potion. Very perceptive enemies (e.g. vampires, sabre cats) likewise can still see you fairly easily, though they can't hear you. Thus, Invisibility is still quite a useful tool even for high-level Sneak users.

If Muffle made you impossible to hear and Invisibility made you impossible to see, Stealth would be a useless skill. Those of you hoping to have Muffle boots, cast Invisibility without the perk, and then sprint around doing whatever you like are going to be disappointed — and it's not a bug.

But I do have Silence perk. Sneak was the first skill I maxed including most of its perks (I don't have Backstab, Deadly Aim, and Assassin's Blade).

I play as a sneaky mage who specialize in illusion and conjuration. Thus I also have Quiet Casting but it's not important because I've always cast a spell from a safe distance.

I just feel that Invisibility should be make you a lot harder to detect than Sneak, not the other way around. The spell cost a lot of magicka and last only 30 seconds. If it actually worst than Sneak which cost nothing and last forever, than this spell is practically useless.

I have Silence perk, wear only clothes, have shoes with muffle, and use invisibility. It's ridiculous that a bear still be able to detect me and charges right into me when I walk along the road.

Yeah I noticed that invisibility doesn't seem to do anything. svcks

Seems to work for me with stealth, not really without.

Enter a lit room with invisibility in one hand, a sword in the other.
Stealth kill one, cast invis, stealth kill another, rinse and repeat.
Or just cast when detected, the "eye" still shows and they move to where they think you are but the can't track you if stealthed.

I do think it doesn't work right without stealth though, it seems mobs will still agro if you just walk around invisible, possibly due to noise.

It hasn't been very reliable for me regardless of light, sound etc. I suppose it has to do more with accumulated aggro and perceptive enemies.

make sure the buff is active, weather it's from spell, enchant, equipment. pot or whatever.

I've had the game glitch my items on me where i've had to un-equip them, then re-equip for the buffs to be proper.

/edit
It seems to me Animals will just about ALWAYS detect you, as they should, having better senses and all of that.

The Illusion Perk tree has a "Quiet Casting" option.
Also, make sure you pick one type of armor and stick with it. (i.e. don't mix light and heavy armor types). Your armor will level up faster if it's all one type, plus there are many bonuses if you're wearing all the same type of armor (head, body, hands and feet).