Sometimes I get the impression that the reason SF&F and the literary worlds tend to be at such odds -- with exceptions, and I'll go into one such example separately -- is because most of the folks involved with the latter surround themselves with an environment that discourages real experimentation.
I had my own brief experiences with this in my teenage years, during a summer creative writing course I took at a local university of major renown. Everyone involved was tremendously enthusiastic and upbeat; it was a fun experience, and I learned quite a bit. But what struck me most about the whole affair -- and only in retrospect, years later -- was a dynamic I saw unfold between at least one of the teachers and the students that made me wonder if it was symptomatic of the kind of dynamics that typically unfolded between writing students and today's professors of creative writing.
The students, being young and inexperienced, would bring in stuff that was pretty wild and out of control, but at least had the spark of life to it. One teacher, though, would provide counter-examples in the form of stories that exhibited the sort of MFA-manufactured dullness that I had at that point already started to reject because, well, it bored me. The stories that we were being encouraged to produce by this teacher were the ones that had -- to use someone's review of the albums of Mark/Almond -- "neither peaks nor valleys, just the same dull threnody, a passionless evocation of life's most minute pleasures."
I don't think any teacher who does this is being deliberately malicious, but I doubt the intent matters. If a system is gearing itself to produce boring writers because that's the best way to keep such a system together (after all, no system selects for iconoclasts), the only way to get interesting writing is to go outside the system.
I'm going to wager that of all the writers produced in the past thirty years, not one of the really interesting ones -- the most stimulating, the most daring, the most far-seeing, the most compassionate -- will have been produced by the academic apparatus designed to churn out creative writers to spec. (Oliver! won the Oscar for Best Picture, beating out 2001: a space odyssey. Which one still matters?)
Every now and then you have a mainstream literary author who touches on SF&F in their work -- e.g., David Mitchell, now with a new work that has some of the same time-hopping flavor of his earlier Cloud Atlas. But with the majority of such work, I can't help but feel the SF part of it is mainly there to add spice, in much the same way a generic romance is given pizzazz by being set in another time and place and thus being given the label "historical".
People in the literary camp seem to be afraid to build a story directly on top of an SF premise, out of the fear that it will be seen as too hopelessly nerdy to be accessible. But how much of that fear is genuine, and how much of it is borne of the fact that it's entirely too easy to write SF badly?
(Yes, the title is a Sisters of Mercy reference. Rise and reverberate.)
Other Lives Of The Mind