After becoming Thane of Falkreath and buying Lakeview Manor, I got Rayya as a housecarl or whatever. But when I go to make her my steward, Kharjo leaves my side as a follower. I ran after him to make him follow me once more, then return to my carpenters bench to work on the exterior and he vanishes. I search all around the house and all around the land but I can't find him. So I fast travel to Whiterun stables thinking that he'll come back as he normally does, but he's gone. I run around for a while, coming up to the Whiterun gates and he comes running up to me. Thinking everything is back to normal, I turn around to walk back to my horse and he's vanished again! So I repeated this until I saw him once more and I tell him to stop following me, I turn around, vanished again. (In hopes that I could make him stop following me then make him follow me to undo the bug)

I hunt for his caravan and I find their tents and camp fire but the khajiits are all gone!

I'm really confused.. Help please?

Note: I do have a backup of before I asked Rayya to be my steward and Kharjo doesn't disappear constantly, but I would really like the help of Rayya to finish constructing my house.