Nothing. You should not feel bad about being privileged. No one expects you to give it up. You should not read that into anything anybody says to you. You were born with it, it’s practically impossible to not take advantage of, you should make the most out of it because you’re lucky to have it.
What’s wrong is assuming that you are a better person because you have it. You’re not a better person than the less successful. You don’t try harder than they do. You’re not more worthy of your fortune than they are. In fact they probably work harder than you. Why they don’t have what you have mostly boils down to luck. It’s wrong to assume anything else.