William Eggleston, the near-mythic southern gentleman and father of color photography, who is placed in the pantheon of the greats alongside Walker Evans and Robert Frank, greeted me with a courtly little bow at his favorite hangout in New York City, El Quijote restaurant, the joint adjoining the Chelsea Hotel.

It would be a liquid lunch for the 76-year-old Mr. Eggleston, who pointed out, in his wry, gracious way, that if he had felt like lunch he would have surely had one of El Quijote’s small, signature lobsters. He began with champagne mixed with vodka, in a tall glass.

“You look well,” I said.

“Thank you,” he said in his light southern accent. “A compliment is always nice.”

He still lives in Memphis, Tennessee, where he was raised a son of privilege on a 12,000-acre plantation. “Ever used a gun?” I asked. “Certainly,” he replied, “but not seriously.”

He was supposed to take over the plantation from his parents. “It didn’t interest me. I thought, There’s nothing to do but watch the cotton grow.” He studied the piano as a child and loved to play Bach. “My parents suggested a concert career. But I said, ‘No, I just like to play for myself.’ ” He still does, playing the piano in solitude for hours most days. Music is as important to him as his other solo preoccupation, photography.

Yet he’s always been a sociable man equally at home in the dive bars of Memphis and, in earlier hipster days, among the Sex Pistols milieu of the once notorious Chelsea Hotel. He lived there during the 1970s with Viva, one of Andy Warhol’s most liberated superstars. “We’re still very close,” he said. “She moved to the West Coast, and every time I’m out in L.A. visiting one of my sons, she’ll come right over, and it’s just like we never separated.”

“You like women,” I ventured.

“You’re damn right!”

And women like him. I’d heard about another of his girlfriends, by the name of Misty Lavender. “She worked in Memphis nightclubs and described herself as a ‘titty dancer,’ ” he recalled. He took Misty home to meet Mom, who mistook her for a ballerina.

He was happily married to the beautiful, complaisant Rosa Kate Dossett, the daughter of plantation owners, during his various open relationships with other women. He even set up house in Memphis with a local lady who also came from a prominent local family, and nobody seemed to mind that either. “You’re a lucky man,” I said of the phenomenally tolerant women in his life.

“You’re so kind and, at the same time, so right,” he replied. Rosa, his longtime wife, died earlier this year. “It was 50 years,” he said. “Actually, it was 60 years, because for almost 10 we were courting. I’d give anything in the world to see her walking through the door right now.” And, at that, desolation overcame him.

William Eggleston’s first photography exhibition at the Museum of Modern Art, in 1976, was a now infamous critical disaster. His enigmatic pictures, saturated in color—a blood-red ceiling, a child’s tricycle, gas stations, a green tiled shower—encouraged new ways of seeing everyday life. Color had been used until then mostly in fashion photography and ads, and the snobby critics, accustomed to black-and-white photography as “art,” were outraged.

“What do you think of photography critics?” he asked me, and called for a double vodka with ice. I replied that they can seem as woolly and pretentious as wine critics. “It’s a sorry thing,” he said matter-of-factly.

Even one of his early, enduring influences, Henri Cartier-Bresson, said to him at a dinner party in Paris, “William, color is bullshit.” What did he reply? “I said, ‘Excuse me,’ and left the table. I thought it was the most polite thing to do.”

In 2012, Christie’s auctioned 36 Eggleston prints for almost $6 million. A single print of his now iconic tricycle, which appears on Earth like a found object from outer space, sold for $578,000. For good measure, I was meeting the man—whose photography has influenced film directors such as David Lynch, Sofia Coppola, and the Coen brothers—to celebrate The Democratic Forest, a 10-volume set of his 1980s work (published this month by Steidl).

We leafed together through several of its haunting visions of the American landscape—many of them without people. The people are present through their absence, as his friend Eudora Welty pointed out. “They once were there,” Eggleston added intriguingly, “but not now.”

He still takes pictures most days, functioning intuitively with one of his many Leicas. Photographers tend to take scores of shots of a single subject, but he believes that once is just about enough. Besides, he says that he doesn’t always look directly through the lens. He is the eye of the camera.

“But we’re all photographers now,” I suggested. He ordered one more for the road. “May I take a cell-phone picture of you?” I asked.

“Go ahead,” he obliged. I took three of him. Then he sportingly offered to take a cell-phone shot of me, although he confessed he didn’t know how. After a little explanation, he figured out my cell phone’s push button and took one picture, scarcely glancing through the lens.

The difference was laughable. Mine were just the usual snapshots, while his was a single, masterly composition of someone seated amid the day-for-night kitsch of El Quijote. He’s a magic man, that’s for sure.