I am on the verge of giving in to buying it on Xb1 (I already have it on Xb360). Thinking how many hours I will play the game (again) is swaying me plus I am curious to try certain mods. But I now have one major issue holding me back. If I use mods, will it disable me from gaining achievements?

yes it will

Special edition will disable achievements if mods are enabled, yes. The original PC version does not and the PC version of SE can have this be bypassed.

None of the achievements are hard to get, some are just a bit time consuming (basically, doing all the guild quests, getting to lv80 and iirc million gold?, and technically building every property, but if you get to million gold then it's just a matter of spending that on all the materials you need for those properties). Just get them in one playthrough and then install mods on a new character.