So I had played through the main quest and later discovered that the side quest "No Stone Unturned" was part of main quest "Diplomatic Immunity". Unfortunately I didn't pick up the unusual gem the 1st time through. Is it possible to go back and get it or is it lost to me forever?

If you didn't pick it up, it should still be there.

However, I don't know if getting into the embassy a second time would be a problem.

Yes, go into the cave that you exit the Embassy from and it will be in there.

I tried finding the cave but I'm having a hard time locating it, but I'll give it another go.

No, but you do not need to go through the embassy in order to get to the cave. The cave can be found http://img4.wikia.nocookie.net/__cb20121215004004/elderscrolls/images/3/3e/Reeking_Cave_Maplocation.png, it is unmarked so keep your eyes open.

http://elderscrolls.wikia.com/wiki/Reeking_Cave

You can get back into the cave. I know I did as I missed the gem the first time through. Prior to Patch 1.4 is stayed in the building, but after, they moved the thing to the cave upon completion of Diplomatic Immunity just so this quest could be completed if someone was working on the MQ.

Cool thanks for all the help

But you can't get back on to the ledge without using a console command, but you can certainly get back in the cave.

The Gem is under that ledge, near a corpse. No need to get back on the ledge as you don't have to go back into the embassy.

I said that in case OP was feeling a little curious, like I was when I backtracked there one day.