High above the Pacific, a hermitage offers a world without distraction, where silence, serenity and joy reign supreme. I get into my car outside my mother’s home and drive up into the mountains. Past the estate where Ronald Reagan kept his Western White House, past Michael Jackson’s Neverland and the mock-Danish town of Solvang. On to Highway 101, which stretches all the way from Mexico to almost Canada, and then, after an hour, on to Highway 1, the two-lane road that winds around the coast of central California. Very soon there’s nothing to my right but dry hills and a few cows grazing in golden meadows, nothing to my left but the great blue plate of the Pacific Ocean.

I pass the turn that leads up to gaudy Hearst Castle, at the top of one of the peaks, and then the beach below the road where dozens of elephant seals lie along the sand like fallen boulders. At last I come to a single-lane path that winds for two miles up to the top of a hill. Each curve discloses a fresh view of the blue-green waters stretching to the south. Not a soul to disrupt the stillness. The silence, when I arrive, seems almost to vibrate, a positive presence.

There’s nowhere else I’d ever want to be. Not just because a donation (of a little over £100 a night) provides me with a room with a long desk, a private walled garden overlooking the sea, and hot food and showers a few feet away. And not only because there’s no phone to answer, no internet to distract, no TV to turn me into a fretful fool. The place is heaven simply because all the deadlines I was worrying about on the road below, all the arguments I was conducting in my head, the worry about next month, that fear for my ageing mother, are gone, completely gone. I’m encircled by more stars than I could ever count, a rabbit standing on my splintered fence and the occasional sound of tolling bells behind me.

Over 50 years of constant travel I’ve been lucky enough to visit Antarctica and Tibet, to feel shaken by the pilgrims of Ethiopia and spooked by the stone faces of Easter Island. But nowhere has so transformed me as the Catholic hermitage, New Camaldoli, to which I keep returning, more than 100 times over the past 33 years. I’m not a Christian and I don’t attend any of the services on offer. But simply taking long walks, sitting in my little garden watching the sun scintillate on the water and chatting with other retreatants feels like the greatest adventure of all. A true trip is one that sends you home a different person from the one who left – more directed, more joyful, more calm – and nowhere has this effect on me as does this simple monastery.

The first time I visited – February 1991 – the world was much less noisy and distracted than today. There were no smartphones, no social media, no constant updates; email was in its infancy. Even so, a few days in silence felt like radiant liberation. All the chatter in my head dissolved and I could remember what I truly love. And all my little plans and hopes and frustrations disappeared, so I could remember what I ought to be. The monks call this “recollection”, the recovery of a truth, a self, that’s always at hand but too often forgotten.

Of course the 60-mile stretch of coastline known as Big Sur is already one of the most transporting places on Earth; the calendar falls away, as do thoughts of the frantic world, as you begin nosing around curves for hours on a tiny road above the sea. The tall redwoods, the high cliffs, the ocean stretching across 30% of the planet – no boats or oil rigs or anything here but migrating whales – makes you feel a tiny part of a very big picture.

“There being nothing to improve on in the surroundings, the tendency is to set about improving oneself,” as Henry Miller wrote, having settled in a convict’s cabin here after travelling around Greece. We can “make our own Bibles”, he went so far as to write; the good-natured bum famous for slouching around among the sex-workers and gutters of Paris even admitted that “it was here at Big Sur that I first learned to say ‘Amen’.” It’s no surprise that the Esalen Institute, widely regarded as the birthplace of the “human potential” movement, is 20 minutes up the coast from New Camaldoli; among the tall trees farther north you’ll find funky inns, campgrounds, tiny churches, even the hidden homes of billionaires.

Sign up to The Traveller. Get travel inspiration, featured trips and local tips for your next break, as well as the latest deals from Guardian Holidays. after newsletter promotion. Inevitably, there are times when the days and nights in the wilderness can be terrifying. Soon after I began staying regularly at the Hermitage, I checked into one of the retreatant’s trailers on the hillside as a winter storm began to shake its fragile foundations. All night the rain beat against the roof. The ancient heater spluttered and I could not see a light or any sign of human habitation. I’d heard of mountain lions prowling through the hills, and just to get some milk would require a walk through the torrential dark.