Self-esteem is kind of a weird term. I guess the feeling that one is searching for is, no matter how anyone might judge you, it doesn't matter to you.
Fortunately or unfortunately people are social animals, and it is not easy to shrug off judgment just like that. If you have managed to score high in the societal judgement categories (like figure-skating judges holding up a score card), of course it is not a problem. Maybe from when you were born you were handsome, rich, smart, calm, and the whole question will never be an issue for you. If you happen to be ugly, poor, dumb and anxious, you have a big effing hill to climb.
Lots of us go through serious depression, we know how important this is. It's not totally under your control. You don't have the console commands for your brain, often there is not that much you can do about what you are thinking. It is pretty easy to get into cycles of negative thoughts that keep themselves going.
If you have to go through say a year of feeling really bad every single day no matter what else may happen, it's like a trial of fire. This is where you can become like a hero just by keeping your life stable.
In the end, it's all just electrical impulses bouncing around inside your skull. As far as I can tell, the best thing is to train your brain through meditation.
What can you do about those judges holding up scorecards, putting you on some BS scale of social acceptability? I guess you have to find something really authentic to fight them with. But if you can't be true to your own goals, then it will be very hard to beat them. So you do have to try to keep making progress, even if that is "what they expect you to do". There can be a point where rebellion only hurts yourself.
Total renunciation is a very difficult path. I could not become like one of those naked Sadhus, who cares about nothing material. Almost no-one can go this route, so don't kid yourself. And if you are unhappy every day, do talk it through with someone. You can come back from anything, and get back to feeling how incredible it is just to be alive for this short time.