• Julian Borger, Terry Macalister and Martin Chulov
• guardian.co.uk, Wednesday 15 April 2009 21.00 BST

When British soldiers first arrived in Basra in March 2003, they quickly set about ensuring that the local economy continued to function. That involved, crucially, protecting the tankers that lined up at the Basra oil terminal and standing by as the pipeline was connected to their holds. Months later, the British realised they had unwittingly been helping audacious smugglers who had been taking advantage of the post-invasion chaos.

Iraq's oil has long attracted would-be profiteers. The country's reserves are the third-largest in the world, and have been left relatively untapped by decades of embargoes and war. In the south, it literally seeps from the landscape – a muddy brown expanse, spread by the spring rains, and peppered in all directions by giant flames that erupt like roman candles from the outlet pipes.

The burn-off of impurities causes giant flames that leap as high as 30 metres from the oil fields, creating a tangerine glow that mixes with dust and sand then spans the horizon.

The Bush administration always denied that oil-lust was a factor behind the invasion. In purely economic terms, it pointed out, it would have been cheaper to have simply cut a deal with Saddam Hussein, as many oil companies did. However, the expectation of huge oil revenues certainly contributed to the administration's optimism that the war would be quick and cheap.

A week after the invasion began Paul Wolfowitz, then the deputy secretary of defence, told Congress that the US was "dealing with a country that can really finance its own reconstruction and relatively soon". He predicted that Iraq's oil revenues would bring $50-100bn in less than three years. On the day Baghdad fell, Dick Cheney predicted output of three million barrels a day by the end of 2003. Six years on, output is still only two million barrels.

"Before the war the expectation was almost unanimous that Iraq would not only resume its pre-1990 level production but it would increase significantly. People were even talking about seven million barrels a day," said Gal Luft, an oil expert and head of the Institute for the Analysis of Global Security in Washington. "There were very high expectations that never really materialised."

Andy Bearpark, a British diplomat and reconstruction expert, became the director of operations and infrastructure for the new Coalition Provisional Authority (CPA) in mid-June 2003.

"All the pressure was to get the oil pumping as soon as possible, but it was proving to be a mammoth task," he recalled. "There was this constant mantra: We have to get to 2m barrels a day. That's what it was when Saddam was in power. That was the key figure that was put out each day and it was hardly ever above 1m. It was no more nor less than anything else done by the CPA – a total disaster."

Bearpark was supposed to be in charge of rebuilding the country's infrastructure, but Paul Bremer, the CPA head, made clear from the outset that his remit would not include the sprawling oil industry. That would be run by Americans.

The exclusivity of American influence caused deep unease back in the boardrooms of British and European oil companies. Fearing they would lose out to their American competitors, British oil companies held talks with No 10 before the invasion about the post-war distribution of contracts, insisting on a level playing field. In the aftermath of the war, multinationals haunted the CPA offices, aiming to stake their claim.

But as insurgent attacks on the oil infrastructure gathered momentum in the long, unbearable summer of 2003, high hopes for the Iraqi oil bonanza faded.

"When the war really began, the Saudis did not protect their border and thousands of jihadis went across. The Saudis preferred to sit on their hands and allowed this influx into Iraq," Gal Luft said. "Both Iran and Saudi Arabia were concerned that Iraqi oil would eat into their [Opec production] quotas. They have made a fortune from the lack of Iraqi production."

Erinys, a British private security firm, was given a $40m contract to guard the pipelines, and that led to a temporary improvement, buoying spirits in the CPA, but Bearpark said insurgents soon skirted around the new defences.

"Those of us inside the system were in a pretty good state of denial throughout 2003 and into 2004," he said. "Things kept going wrong, but because there was tremendous pressure to churn out good news for the American public, we began to believe it ourselves."

The big oil companies could see that the investment required was enormous, the returns uncertain - and began packing their bags. Meanwhile, Libya had voluntarily given up its nuclear programme and was open for business, a much more attractive proposition.

In the past few years the industry has begun to recover. Safer roads have led to more tanker convoys and new wells are being planned throughout the Basra province, along with two crude oil processing plants. A total of 45 wells are planned for the huge South Rumaila oilfield south of Basra, while processing plants are scheduled to be installed west of the city at the west Qurna site.

Roads leading to oil fields on each side of the main highway to Baghdad are now heavily guarded by Iraqi troops and are largely secure. So too is the highway itself, a straight, flat road of European standard, that was a death run for oil convoys until mid-2008. Output has remained stubbornly at the two million barrel level, and the country still does not have enough refining capacity to be self-sufficent. The prime minister, Nouri al-Maliki, is under pressure to bring in capital and expertise while not being seen to "sell out" Iraq's only real wealth-making industry to companies from the outgoing occupying countries.

The government and the oil companies have been haggling over Iraq's new oil laws, as the multinationals have sought to squeeze out ever better concessions out of their potential hosts, not least by arguing that the area has become less attractive due to a slump in the price of crude – from $150 a barrel last summer to $50. As recently as the middle of February, 32 foreign oil companies – including Shell and BP – met the Iraqis in Istanbul and asked them to come up with more favourable terms for the southern fields, which hold as much as 40bn barrels of recoverable oil, around a quarter of the country's total. Predictions of a final deal in June now look optimistic.

The tragedy for Iraq was its inability to capitalise on last year's oil price spike. With the current price slump, it is much harder to get the investment needed to salvage the industry.

"Iraq had an opportunity to make a lot of money and it was lost," Luft said. "I believe there will be another spike as we move out of recession. The question now is whether Iraq will be in a position to benefit the next time round."