I suppose we each have a differing 'set point' regarding challenge that is the 'just right sweet spot'. My (one and only) character is a no melee mystic archer so, naturally, her ability to kill with her bow is paramount. Her archery skill and damage-related perks are all taken to the max. I don't use any apparel or potions that boost archery. Where I adjust her 'challenge' is with the actual bow itself. Although I tend to use the CK to adjust her bow's damage to taste, the same is easy enough to do by the pure vanilla techniques of simply choosing a bow based on damage and, similarly, choosing how/if to enchant/temper the weapon. I find it hard to believe that somewhere between an unenchanted/untempered long bow and a dual enchanted legendary tempered daedric bow, one cannot find a weapon to support whatever challenge is desired.
Another thing I do as more of a fine point relates to the bow's speed and provides additional challenge in all scenarios except long range sniping. Individual mileage may vary depending on archery-related ini tweaks, but all things being equal, arrows fired from a slower bow have a flatter trajectory and fly farther. At very long ranges the effect is quite noticeable. Since my character is much more of a sniper than a 'skirmish archer' she uses a very slow bow (0.5 speed just like a daedric bow). Not only can she make wonderfully impressive very long shots, but I love the extra challenge the bow's slowness demands in closer quarters when being attacked - her bow is intentionally a lousy 'skirmish up close' bow. The feel of the weapon is very much that of a sniper weapon - large, slow, cumbersome, accurate and powerful.
Finally, one additional 'challenge' my no-melee archer faces is that she insists on traveling by horse. Horse archery is quite an art form / challenge that, over thousands of hours of practice, she has become amazingly good at.