i collected a few unusual gems but i don't want to join thieves guild with my lawful character. so.. how can I get rid of them? or i have to keep them forever?

Have to keep them, although I think you may be able to remove them with the console commands.

i'm playing on 360, no console commands for me

Well if you don't get caught the rest of Skyrim will still believe you are a 'lawful' character

Laws are more like recommendations than requirements...

OP's concern is legit, no reason to force you to carry items especially when they have weight

It's not illegal until you get caught!

I agree. I wouldn't mind if this were an arguably Robin Hood-esque Thieves Guild like in Oblivion, but this one is more like a mob.

I agree. I am playing a lawful character as well, and one is taking up space in my inventory.

This more than enough reason to cry foul. I don't need my hand to be held in order to complete quests!!! If I miss something or misplace it, then it is my own fault for losing an objective. This is yet another reason why I still consider TES III the best of the latest titles. (I haven't played TES II Daggerfall, but from what I have seen on youtube and uesp, it looks even better than Morrowind... in terms of game design.)

I'm experiencing the same problem. Really the game needs to offer you more choice in such matters. If it was up to me I'd hand in my unusual gems to the proper authorities after finding out their purpose.