Many of the folks who heavily shaped my view of what science fiction and fantasy could be had pretty storied lives. One was Theodore Sturgeon, who according to the 'Pedia worked in the merchant marine, sold refrigerators door-to-door, managed a hotel in Jamaica, worked in construction and other blue-collar jobs, and wrote ad copy. Another was Frank Herbert, who worked as a journalist, photographer, political speechwriter, Navy Seabee (also as a photographer), and a few other things I can't find right now. They got around.
Compared to Sturgeon, or Herbert, or whoever, I haven't done anything, and I know it. And I have never been in much of a position to run off to have Adventures™. Neither the cash nor the opportunity — or for that matter, the desperation. For those who have write what you know mantra'd into their ears their whole lives, it's painful to realize you don't know much because you haven't lived much either.
Everyone gets handed a different deck of cards to play from, and my deck didn't have the cards in it that would have allowed me to run off to far-flung parts of the world and do whatever. I had to make up for it, and one of the few ways I could do that was by reading outside my comfort zone. (Side note: If you want a good overview of a nonfiction subject, go read a college text. It'll be dry, but correct. Don't dig up the latest Johann Hari or Malcolm Gladwell or any of the other too-clever-by-three-fourths books marketed at executives waiting in airport lounges.)
The key thing I had to do was always enforce within myself a sense that I had to work that much harder to know the world. I had to do my homework, and be humble about it. I studied Japan for a decade and change before trying to write a book that used it as a setting, and even then I felt like I'd only been able to leverage so much of my research. I had to stay hungry.