It has been a while since I installed MSGO, but I recall several times during the installation where I had to wait quite a while while nothing much seemed to be happening, and I have a pretty fast machine. The entire install took well over an hour, maybe two. Cannot remember exactly, but I remember it taking a long time. It is not uncommon for mod installation and management programs to say "not responding" when they are actually operating just fine and just need more time to complete their tasks.
So, how long did you wait for the system to recover before force exiting out? If you waited less than an hour, then I might suggest trying again and just letting it sit a while.
EDIT: You also might want to check the http://www.gamesas.com/topic/1504990-relz-morrowind-overhaul-sounds-graphics-30-thread-5/, and once you get installed, see the OP in that thread for a bunch of unofficial fixed to bugs in MGSO.
EDIT2: I just saw this thread where you are describing what sounds like a different problem (re-running MGSO Options after successful installation), whereas in the OP in this thread it sounds like you are talking about initial installation. . http://www.gamesas.com/topic/1504969-mgso-options-crashes-when-i-hit-apply/#entry23716006 It is hard to understand exactly what you are saying Re-running the "Options" in MGSO after your original installation and after you have installed a bunch of other mods can be somewhat problematic. Personally, I never re-run the MGSO "options" after initial installation and do any subsequent changes through Wrye Mash/and or manual installation.