I’d ask Mercedes R. Lackey for more detail but the reason is book economics. People who like books tend to read a lot of them, so they don’t have a lot to spend on each one. Nobody knows how well any particular book is going to sell. So it makes a lot of sense to wring as much waste out of the industry as possible so as to make as much content, and money, as possible.
This dynamic favors the integration of the two genres into one group in a way you don’t see with, say, romance novels. Essentially the same sorts of people who like science fiction also generally read fantasy, for the same general reasons.
For decades, if you were wandering through a bookstore looking to buy a book, if you were interested in science fiction or fantasy, generally you wouldn’t mind checking out the other genre while you were there. If you were looking for romance, well, a hard science fiction book would stick out like a sore thumb. The thematic similarities are strong enough to where fantasy and science fiction are pretty much always right next to each other at libraries, where the lack of a profit motive means they can afford to organize more topically rather than in a way that is geared towards selling as much as possible. At least they’re no longer mixed together like a bag of apples and pears. (sacrilege!),
It was only recently that people started thinking of science fiction and fantasy as more than just books, and so the less punishing economics of other industries could provide for more stylistic differentiation of the two modes of storytelling. This makes it more and more clear to fans that there are real differences between them.
But to non-fans who look down on middlebrow culture, lumping them together is a cheap way to make yourself look cultured.