No matter how far I fare in literature - ancient classics, works of abstruse philosophy, acclaimed poetry, fantasy, nonfiction, and mundane human stories of the here and now - I never find anything to compare with science fiction. In any decent examples of the genre, there are no clear lines between characters and environment, but a recognition that they feed back into each other, change continuously (and often unpredictably) and yet in ways that still ring true. The very term "science fiction" is just a clumsy label for something both more alive than scientific speculation, and more profound than mere storytelling. It is herald and conjurer of humanity's highest aspirations, gravest nightmares, and most bizarre avenues of possibility.
We fly through the heavens in giant mechanical birds, look down at the mountain seats of ancient gods, and talk to people standing on the other side of the planet - we, living in dreams of the past come true, having become something more (and perhaps also less) than our distant ancestors would have defined as human. And it all became possible because people started figuring out how reality works, seeing the connections between things, and using the scientific method like a self-unfurling road into the sky.
But the Enlightenment was more than a revolution in tactics - it was an entirely new and potent mutation of human consciousness. Before this point, the logical and the imaginative were irreconcilable: People not only knew nothing about the world they inhabited, but didn't even know how to begin discovering its properties. Having no reality compass beyond common sense and flawed philosophical logic, they consigned the unknown and uncreated to the domain of God, and concerned themselves only with maintaining what they already understood.
That began to change as "natural philosophers" increasingly divested themselves of the religious establishment, and started probing nature for its secrets rather than just musing on them in the context of tradition. To their surprise and delight, these early pioneers found that the deeper they looked, the more exotic the world became: There was no bottom to the strangeness. And yet every new oddity cast existing knowledge in a new light, revealing patterns that could be extrapolated to produce a testable theory. In that was the birth of the future, and subsequently the literary genre concerned with it, science fiction.
Until relatively recently, the future was just a background in the genre: A new and more powerful set of technologies applied to a more or less unchanged human condition. In classic works such as Isaac Asimov's Foundation trilogy, we find a civilization that is easily comprehensible despite being 20,000 years in the future and encompassing an entire galaxy.
The reason is that the Foundation novels are not intended as prediction, but as illustration of large-scale sociological principles - a breathtakingly insightful set of revelations about how civilizations rise, develop, decay, and fall. Asimov communicates the same basic story as Gibbon's Decline and Fall of the Roman Empire, but with a crucial twist: Science is there to rescue humanity from the impending Dark Age. He is thus revealing human history not as an endless repetition, but as endless variation on repeating themes - an outward spiral into uncharted waters.
History doesn't repeat itself - at best it sometimes rhymes.
--Mark Twain (Samuel Clemens)
But the people of Asimov's Galactic Empire are nothing like what they would actually be in such a civilization, given that technology - they are written so that we relate to the principles being illustrated, and that was as far as the genre had evolved at that point: Placing ordinary people, as defined by the vantage of the reader, in a future context. There was relatively minimal attempt to realize how such an environment - or the history required to create that environment - would have shaped the people that inhabit it. Science fiction - indeed, even science itself - had not yet progressed to the point where audiences would have been receptive to that level of strangeness.
It was in the 1960s that science fiction first began to take a more systemic look at what is possible, and at this point a bifurcation took place in the genre. Two strains emerged, both enormously potent and generative of multiple sub-genres: One, which we can call the "Gestalten," and another, the "Kaleidoscopic." These terms are arbitrarily chosen to reflect the character of the two main families of science fiction that emerged at the time, and both still flourish to this day in a multitude of forms and sub-genres.
Gestalten science fiction is written to illuminate long-term patterns and larger-than-life realities, and thus commonly takes the form of epics occurring over a large span of time and space, and involving a substantial number of characters and plot-influencing forces. As a result, it tends to be associated with the "Space Opera" and ecological SF sub-genres, but neither are synonymous with it - a work of space opera or eco-SF, even of high quality, doesn't necessarily have to follow the Gestalten strain.
Another characteristic of this type is that, due to its holistic approach, it tends toward naturalism and humanism, perceiving the universe and civilization in terms of organic systems rather than easily-reducible functions. It has an ability to understand and indirectly represent abstract spirituality without indulging in fantasy or sentimentality. We can also say it strives to communicate directly with human experience by showing the audience pure examples in motion rather than trying to be didactic or entertaining. The subject is always larger than the reader, and meant to convey a perspective above that of common existence.
The apotheosis of this strain - and one of its early successes - was Frank Herbert's Dune series, and in particular the core tetralogy (Dune, Dune Messiah, Children of Dune, and God Emperor of Dune). These novels wrap up in one huge package the ecology of alien worlds, their effect on human cultures, the evolution of sexual politics, the power and danger of prescience, the fundamental difference between human consciousness and machine intelligence, the economics of empires founded on control of a crucial resource, the nature of commerce, the nature and limitations of political power, on and on.
And while many works of SF have dealt with these themes, and some have dealt with many of them simultaneously, Herbert's work doesn't stop at dealing with them - it shows them in dynamic operation, and illustrates truths about humanity and reality that are almost impossible to directly express. He creates a gestalten awareness in the reader of what he is attempting to show, with all of the pieces fusing seamlessly together into a complete universe with no limits and an ever-expanding context. Other works in this strain come nowhere near to that level of achievement, but they are often cherished, award-winning novels. It's related sub-genres are space opera, ecological, and future histories.
The second strain, the Kaleidoscopic, is reductionist and consists of exploring a multitude of facets rather than wholes. It tends to be cynical, compulsively focused on technical details and social transgressions while neglecting their dynamic consequences, and in some cases descends into outright nihilism. If the Gestalten is about perceiving things as a larger whole, the Kaleidoscopic is about deconstructing things and trying out weird combinations of the pieces, most of which don't necessarily mesh into anything inspiring - the basis of its reputation for cynicism and flip attitude.
A world built on the Kaleidoscopic strain is not grown in a conceptually organic way, but more like Frankenstein's monster - hypothetical technologies or circumstances are thrown together without necessarily caring how or why they exist, and then the author fills in the blanks to most sensationally demonstrate them. Pretty much every sub-genre of science fiction with the suffix "-punk" is Kaleidoscopic, although the original occurrence of that suffix - cyberpunk - has had some crossover work (e.g., Accelerando, A Fire Upon The Deep) that seeks to bridge the divide, occasionally with superb results.
This strain doesn't seek to show you higher truths, but lower truths - to disassemble reality and parade its components before the reader. To put your face closer and closer to the screen until the picture dissolves into a chaos of pixels. Its more potent offerings thus often have a vaguely demonaic, psychotic flavor to them, but it can still offer enjoyable and enlightening explorations. Some of the facets of human psychology and technological progress explored in this strain are quite interesting, and the "-punk" sub-genres are sometimes successful at creating a sense of coolness or even sexiness about future technologies.
Still, the inspiration factor is pretty low for work in the Kaleidoscopic strain - it tends to present dystopic futures and grimly dysfunctional individuals as a matter of convention, and resolutions are more often than not merely temporary escapes from just one more catastrophe in an environment defined by violent chaos or oppression. There is usually no overall trajectory of progress except the purely technical - human understanding and aspiration does not advance, and historical patterns are unduly avoided (rather than reinterpreted) in order to exaggerate the superficially exotic.
The epitome of Kaleidoscopic science fiction is Philip K. Dick's A Scanner Darkly - a psychologically toxic work of evil genius that progressively dismantles human consciousness in a frenzied and yet empty social environment without any trace of deep human connection on any level. It is a literary vivisection of a human mind, occurring in a moral vacuum devoid of emotional support or any anchor to humanity, and the reader experiences it right along with what passes for a protagonist. It is the darkest of all works in this strain, and also its purest crystallization. Related sub-genres: Cyberpunk, steampunk, psy-fi, ecopunk (overlaps with general ecological SF), biopunk, etc.
Of course, not all Kaleidoscopic work is cynical - some of it is just overly concerned with technical detail at the expense of any over-arching revelation, and some is just deliberately intended to be a freak show (like Iain M. Banks' Culture novels). Some writers - even ones of quite high caliber - create their books as if they were writing a policy proposal rather than telling a story to inspire generations, and that tends to be an abortive exercise. Others write about concepts limited enough that the distinction doesn't really matter, and one can read their work more as a diversionary thought experiment rather than an all-out leap into strangeness.
Overlap work, such as noted above, tends toward Singularity ideology, that treats information technology with the same awe and reverence that space travel receives in space opera: It's still reductionist, but with a wild card (the Singularity) thrown in to make the technology that usually creates standard cyberpunk dystopias into something more palatable or even utopic. Singularity SF is a niche within a sub-genre, and I only mention it in passing - the vast majority of work falls reasonably neatly within one of the two strains.
Examples of the Gestalten strain:
Examples of the Kaleidoscopic strain: