Pity the poor SF author! "New York, 3111." Pleasant though it might be to stop there, it's really just the beginning. New York in 3111 might be any one of an infinite number of New Yorks, and it seems unlikely that the reader will summon up a picture of Manhattan heading off for the near stars inside a gravity-polarized spindizzy bubble without a little guidance from the author. (Cities in Flight, James Blish. Took me less than five minutes to find the year that NY NY goes "Okie", I'm vaguely proud of myself.)
And save some of your pity for fantasy authors - at least people know what New York is! After the expectant reader is informed that it is the 3rd of Grune in the Year of the Reversed Ptarmigan in the Century of the Anchovy in the city of Ankh-Morpork, said reader may well be even more expectant.
The fulfillment of that expectation is the great strength of fantasy or science fiction, but tends to be handled in different ways. Some authors, like William Gibson, blend the process of setting the stage into the narrative - or, like William Gibson again, just go for it, and it's the reader's job to keep up. (Compare Neuromancer to Count Zero - there's a bit of overt scene-setting in Neuromancer, but Gibson certainly doesn't waste anyone's time explaining things in Count Zero, sequel or not.) Others choose to do a deliberate introduction of some sort: The Stars My Destination by Alfred Bester is a good example of a brief prefactory statement before jumping in with both feet, and I'd be spoiled for choice if I started looking for books with a second chapter that draws a map of the world, as it were. I am in no way critical of this approach, those chapters are usually beautiful, wonderful and marvellous, the author standing centre stage and describing their vision directly to the audience.
But my great personal irritant is the cliché of the expository speech, wherein the kindly but agèd scientist/wizard/elder/brood master is continually talking about why it is that things are as they are. Now, I grant you, this can be done with grace and style, and I'm willing to admit that dialogue is a useful tool - after all, there are times when the reader needs to get some backstory or nothing will make any sense at all, and having an uninformed hobbit or two asking leading questions certainly makes life easier. However, there have been many times when I've thought to myself, "Yes yes, here we go, the Old One is going to give us a little talk...You know as well as I do, young one, that ever since the time of the Great Struggle with the Darkness..." There should be some sort of demerit system for any sentence in a science fiction or fantasy novel that starts, "You know as well as I do" or anything along those lines.