There are arcane arts you can practice to defeat the size limit. A google search will take you to the nexus site page for it. It takes a bit of magic, but you can build as much as your system will handle... which is likely a fair bit more than the small number settled on for console support.
This! Population is capped at 10+charisma. Although apparently things that boost charisma work if you keep them up long enough. Also you could just cheat in the console to arbitrarily boost charisma for the sake of settlement building. I think 15-20 settlers is enough for me personally... enough to man all the shop types, farm food, and post a few guards.