I don't know if "1.1" is a valid value for a variable of type "short", but then I've heard ObScript handles all numerical values as "float" anyways, so I'm not sure this is to blame. Nevertheless, short is "meant" to be an integer type, so I wouldn't assign a floating point value ever by definition, if only to play safe.
Using different stages, 1 each for displaying the messagebox, which is left immediately afterwards to go into the respective buttonpress collection stage, should be the right way to go.
If it helps any, here's how I did a similar stacking of hierarchical messageboxes:
if(doConfig == 1) MessageBoxEX "Dragon Race > Do You want to keep all default settings?|Yes|No" set doConfig to -2 returnelseif(doConfig == -2) set choice to getButtonPressed if(choice == 0) ;stick to defaults MessageBoxEX "Dragon Race > Forced unequipping:%rBy default this will also affect items%rcovering more than 1 slot at once,%r(e.g. The Arena Raiments or DB Armor)%rleaving actors running around naked,%rif they weren't adapted yet.%rDo You wish to make an exception%rto allow non-fitting combined-slot items?|Yes|No" set doConfig to 12 elseif(choice == 1) ;customize MessageBoxEX "Dragon Race > Select Your shape:%rDo You want to be Full-Dragon shape?|Yes|No" set doConfig to 2 endif returnelseif(doConfig == 2) ;full-dragon or custom? set choice to getButtonPressed if(choice == 0) ;stay full-dragon ; ... do settings ... MessageBoxEX "Dragon Race > Select Your shape:%rDo You want controlled helmet slot,%rnot actually force-equipping anything%rbut keeping non-fitting helmets unequipped?|Yes|No" set doConfig to 5 elseif(choice == 1) ;further customize MessageBoxEX "Dragon Race > Select Your shape:%rWhich wings & tail combination do You prefer?|Wings and tail|Only wings|Only tail|None at all" set doConfig to 3 endif returnelseif(doConfig == 3) ;tail & wings options
and it's going on and on and on...
Basically it's the same as you're doing, so I'd check this thing with assigning a float value into a short variable first.