In 1845, John O'Sullivan, an American of Irish descent (who was probably a distant relative of my great great-grandfather), coined the phrase "manifest destiny" to define a vision of a US empire encompassing what were then parts of Mexico.

O'Sullivan subsequently lost credibility for supporting the Confederate slave states in the American civil war. But the idea he summarised in two words has been a factor in US foreign policy for more than a century. It suggests that America has been given a special role in world affairs that is beneficent and irresistible.

The belief that God is American was shaken by the Great Depression, which began with the 1929 Wall Street crash. But destiny knocked again when Japan attacked the US fleet in Hawaii in 1941. When war ended in 1945, America was a global power.

Unlikely contender

Since the collapse of the Soviet Union, it has been the sole superpower. For most Americans, it seemed entirely natural that this should happen.

Senator Barack Obama, the Democratic Party's presumptive presidential nominee, is seeking to inherit this legacy of expectation and illusion. There has never been a more unlikely presidential contender. Practically nothing about his background conforms to the conventional formula for making an American president.

This may explain why he is yet to make a statement that unambiguously echoes the manifest destiny message that presidential contenders must normally proclaim. He does not believe it and seems to have trouble even pretending he does.

Reaction to Iraq

Successful presidential candidates invariably strike a chord with the American majority. The fact that Obama has a lead of any kind over John McCain, a manifest destiny candidate who won't stop saying that the US is best, suggests something profound has happened in America's heartlands. A growing proportion no longer believes their country, despite its many merits, is that special after all.

This may reflect the decline in religious practice. Believing God favours the US over every other nation is ridiculous to the rational majority.

There has been a reaction to US actions in Iraq. The administration argues that ends are justifying the means, but many Americans can't forget the exaggerations before the war, the excesses at Abu Ghraib and Guantanamo Bay and the apparently limitless costs being incurred in Iraq in terms of both money and lives.

And now the US economy is teetering on the brink of a recession which could be deep and lasting.

Blueprint

Obama may yet revert to the presidential nominee blueprint in the weeks following America's September Labour Day holiday. But should he win in November, he will have to have policies to reflect the new American mood of uncertainty about itself and the world.

This is both a challenge and an opportunity in the Middle East, which will be where an Obama administration is bound to want to quickly demonstrate that it is different to that of Bush.

Events are working his favour. Before Obama's visit to Israel, Iraq and Jordan in July, one of the State Department's most senior officials met Iranian counterparts in multilateral talks about Iran's nuclear programme. Iraqi Prime Minister Nouri al-Maliki endorsed, with caveats, Obama's 16-month deadline for the withdrawal of US troops. It is flawed, but there is an Arab-Israel negotiating process.

Diplomatic programme

American Middle East policy is favouring diplomacy in all areas. This is partly because the approach makes sense. It also reflects the weakening of America's strategic position since Baghdad fell in April 2003.

The difficulty facing Obama, should he become president, is to demonstrate that it is better thinking, not weakness, which has led to the rejection of the politics of pre-emption and confrontation.

This will require working constructively with America's Middle East allies, particularly in the GCC. Obama will need a coherent Middle East diplomatic programme if the US is not to suffer further strategic reverses in the region.

A new page is being turned in American history this summer and autumn. The era of certainty is passing in US foreign policy. But nothing will be any easier in the Middle East for the White House's next incumbent.