I think a drop in numbers over time is only to be expected - after all, the game was released in 2011 - and there's not many games (at least without a multiplayer component) that are still popular after so long. Even with mods, there's only so much to experience in the game - and, at least for most of the general gameplaying public, overfamiliarity and playing the same game for extended periods can lead to boredom.
I've noticed a distinct rise in ex-console players visiting the mod forums - those that enjoyed the vanilla game but have bought for pc are now experimenting a modded game for the first time.
I also imagine that the retail release of the legendary edition of the game in June will result in a (short-term) rise in numbers.
As with Morrowind, Oblivion, and Fallout, we'll continue to see both new and old visitors come and go, and return after missing the game...
There will still be a die-hard core of mod authors and mod users, but the sheer number of users in the past will naturally continue to die down.