I want to prevent the player from using a certain spell when enemies are nearby. How can I tell if the Player is in a potentially hostile area? InCombat is pretty obvious, but doesn't apply if the player is sneaking about in, say, a bandit camp and hasnt been discovered yet.

You can try using the wait function. If there are hostiles in the area you can't use it. If you can, you're probably safe.