Forgot who mentions it, but someone in Elisif's court has the Opposite view, that Torygg was a huge really devoted fan of Ulfric and that it is very likely had he gone to him after he was high king and talked to him, he would have declared independence, he was captured because he was doing it out in the open and basically flaunting it, that was a stupid mistake on his part, had he done it in secret he would not have been captured in the first place. Sybille also points out that the reason Torygg was loyal to the empire was because Talos was it's creator and that he was not going to allow some elven scum to try and break it. Also, Why could they not have worked together, that is what they do at the end of the imperial quest line anyway. Also, yeah he ignored it and never told anyone, It is obvious she worships Talos, and somewhere (forgot where though i admit) it mentions that many of the imperial troops worship Talos Secretly. Tullius knows it but ignores it because he disagrees with the Thalmor.
At this point, We should agree to disagree, we will never change the others mind at this point and that is what the game planned and succeeded in doing