Oh yeah, I did unfortunatley date a fellow who liked me because I was so small. The creep. I can definetly see why you would want to be cautious there.
Gotta admit, it bothered me to the point I began to wonder what was going through the minds of all the men I had been with.
Did they like me for me, or was it the potential closet pedo?
Ahh, to be a tall, willowy female ......
There's also the potential 'problem' of remembering to be respectful of the person as an advlt. But that's something which shouldn't last long, just a matter of keeping it in mind until one gets used to the size difference.
I suppose the same is true for people with noticeable disabilities.
Damn those preferences for physical attributes! How dare they! Men! Ugh!
I have to admit that's a rather silly and self-conflicting question. Your physical attributes are a part of what makes you "you", therefore if someone likes you because of your height or some other physical attribute of yours they don't like you any less for "you" than if they liked you because of some emotional/psychological attribute of yours.
Its a little different when a bloke has admitted that he's attracted to you
because your small size makes you more childlike 
.
EDIT: Changed some wording for clarity.