As long as you don't wear it in public, then sure, why not, wear it all day long. But unless you enjoy having every single person on the road or on the sidewalk staring at you while they pass by, then you probably don't want to wear it outside. Oh sure you can say "who cares what others think" and blah blah blah but I think you'll find not wearing a skirt in public to be a good thing.
I have to heartily disagree with this sentiment.
As someone who has worn stranger things than skirts / kilts out in public, I can safely say that, yes, it can sometimes be a bit difficult, especially considering that many people will stare and maybe even gawk. But screw 'em. If you feel comfortable wearing a skirt / kilt out in public, then do it. Just remember to cross your legs. Or... don't.
Despite popular belief, conforming to the norms of society is often advantageous. For example, going to work dressed like Catwoman is not a good idea. Walking around with an assault rifle strapped to your back, in areas where that's legal, is not a good idea. And likewise, wearing a skirt in public while also being a man is not a good idea. Oh, sure, sure, none of those things hurt anyone, and they could all be instances of the person "expressing themselves" or being "non-conformist," or just doing what they're allowed to do, but that does not make them good ideas.
They also aren't bad ideas, either. If you're just walking around wearing a skirt / kilt and people make judgments based upon what you wear, then that is their problem, not yours. Express yourself the way you want, so long as you feel comfortable doing so. If you know you won't lose your job, wear the skirt / kilt there. It's all about how comfortable you feel doing an activity and the possible consequences of performing that activity in relation to what sort of outcome you want.