Our understanding of the world is based our perceptions of it, based on the physical evidence perceived with our senses. Beyond that, we also rely a lot on hearsay about things that accepted science has ascertained, whatever the method to procure it may be.
When you’re reading a story, the conventional expectation is that the events that happen inside the story are going to follow the same rules that are found in the actual world. If they do not, if things are allowed to happen within the story that are not allowed in the actual world (or vice versa, though that is rare) then the story in question is deemed “speculative” for the purpose of communicating with the audience.
We think of speculative fiction as having two distinct traditions and, to a certain extent, we are correct in this assumption. On the one hand, we have “Science Fiction” and on the other we have “Fantasy”. Both present things that do not happen in the world as we know it. But only one of them sets out with the express intent to show things that cannot happen.
The idea behind Science Fiction is expressly to present things that, according to science as we understand it now, could happen (or, in the case of the subgenre of Alternative History, could have happened, had historical circumstances been different). In fact, there have been cases in which a technology originally introduced as science fictional has become a reality; the most famous example is probably Jules Verne, with his submarines and moon rockets.
When someone sets out to write a work of Fantasy, though, there is no expectation of this. While it is possible that Dragons, in some way or other, might have existed at some point in Earth’s past, it would be a stretch to assume any connection between the kind of sympathetic magic often depicted in works of Fantasy, and the actual world we inhabit of agreed-upon science.
I once brought a script I’d written to a conference and showed it to an agent. The script was set in a world in which dragons and robots coexisted, and even—in an extended scene I am especially proud of—fought against each other. I had had some interest from other people who begged off on the basis that the script would be too expensive, but this particular agent said that she didn’t get it. “You’ve got dragons, you’ve got robots,” she decreed. “You don’t mix those. Audiences get confused.”
The idea that Science Fiction and Fantasy can’t coexist in the same story, is based on the theory that the traditions are incompatible. In essence, she was saying that when an audience departs from reality, they can go into one of two worlds, one of which is Science Fictional, the other Fantastical.
But the reality is that both Fantasy and Science Fiction are both extensions of reality. You’re always going to start off from something resembling our world in some way, shape or form. But Science Fiction is expressly not meant to be a different reality with different rules—it is a reality that obeys laws that we don’t have yet. This makes it an extension of our actual world.
Fantasy, on the other hand, is a contradiction of our actual world, presenting as it does things that cannot happen—could not, even in a Science Fictional set-up. So if Fantasy is going to contradict our world anyway, why not have the world it contradicts be a world that features science that we haven’t found yet. Magic (being defined here as anything Fantastical that doesn’t exist in the actual world) is an addition to reality on top of whatever Fictional Science has been provided.
And then of course we oughtn’t forget the fact that Dragons vs. Robots is just objectively cool. What audience in their right mind would turn that down, no matter what the supposed “contradictions”?