No, there is only one time Ulfric addresses the Dunmer at all(where he says that he currently have larger concerns and if his steward could relay this information to them). His future plans for them, if any, remain unknown.
Jorleif: "Sir, there continues to be unrest in the Gray Quarter."
Ulfric: "Blasted http://www.uesp.net/wiki/Skyrim:Dunmer. I don't suppose you could tell them that I presently have larger concerns? Such as all of Skyrim?"
Jorleif: "They don't seem to be very sympathetic to our cause, sir."
Ulfric: "Let me know if you hear anything more substantial?"
Jorleif: "Of course, my lord."
This is the full quote of the conversation. It is dismissive, but not quite as hostile in tone as your paraphrase implies.