We have few invalid answers. Gabriel though you did more writing than anyone ( which is great ) you failed to answer the question. Over all with everything they had on the table (and when you are not making it harder for youself). Who was more likely to hurt you/kill you/kill a follower..... and why????
Overall? With a lvl 35/30 character with no consideration for making a challenge for myself? On Very Hard?
Then neither.
Balance is horrid in both games and both fail to provide a decent challenge.
Difference between them is that if I nerf myself in FO3 it's
barely a challenge while if I nerf myself in Vegas then the challenge is
greatly increased.
Both of them hand out too many skill points, too many perks, too unbalanced perks, too great of gear while delivering crap gear and stats to the enemies.
In this case, Enclave's damage is pathetic with their nerfed plasma and laser weapons with only Hellfire being a problem but even Hellfire are easily disposed off since they follow the DR design.
Legion on the other hand follow the DT design so while they can prove to be more of a challenge, especially veterans or centurions their static spawns are still filled with recruits and decanus which doesn't have the best of gear.
A way to fix both scenarios would be to give Enclave soldiers a hidden perk which increases EW damage by 10% for regular, 30% for Tesla and 50% for Hellfire then to give Legion's firearms users AP rounds and a hidden +15% damage perk and also to give every single Legion member a 20% increase in running speed (since this is what they are suppose to be best at, closing the range between themselves and their enemies.).
Unless that is done then neither presents a challenge at top level.
So at mid level?
Neither there either, Enclave are level scaled and since laser weapons svck in FO3 and their armor DR% is crap they hardly present much of a challenge unless one plays as an unarmed character and have not found a power fist or deathclaw gauntlet yet.
At low level?
Legion wins there, I hate facing them at lower levels as they usually kick my ass.
Enclave on the other hand I never face during the lower levels as when I do the Waters Of Life quest I'm already at lvl 12+ so I couldn't comment on them.
Overall, Legion "can" be dangerous but it's not often and Enclave's Hellfire soldiers can be a tough nut at lvl 22 if one hasn't gotten good gear yet but otherwise they're also very easily to dispose of.
So it's a tie for me.
What faction I do think is hard is Talon Company, they have been more challenging than both of these factions for me.