Is the FPS cap supposed to be applied when the "Limit Video FPS" option is ticked?
It didn't have any effect on my FPS so do I have to change other settings too?
Yes. However, you might need to select TESV.exe in the profile drop-down menu at the top, then deactivate the "Use Default Setting" checkbox, then set the FPS limit.
Also, be sure not to run programs like ATI Tray Tools since it somehow interferes with each other.