I used to think that you had to work on your personality in order to fix this problem. I no longer believe this. People either are compatible conversation partners or they aren't.
You can work at being able to have conversations with people you wouldn't ordinarily, but it's not a good skill to rely on for romance because it causes people to hold on to people they shouldn't be dating unnecessarily.
If it's awkward, consider it a bad fit and move on. You're just not going to click with everyone or even most people.