I've never used swift learner because there is plenty of XP anyway
The Fortune finder perk is not bad but I don't use it because the game will become too easy
The explorer perk takes the fun out of exploring
Friend of the night - this is the only perk which I removed with a console command and assigned another one to my character.
Apart from this one, I don't find any perks particularly useless because I use them according to my characters, their skills and personalities, so I don't always use the same perks.