Every six months the National Archives release previously classified documents. In among the eating or driving arrangements of politicians, and gossip about diplomacy lies long forgotten information about the workings of the British state.
There is the odd insight into the chaos of actual government, such as whether to have a disgraced US president visit. In the case of former Tory prime minister Margaret Thatcher it was decided that Richard Nixon was a bit toxic.
Others are concerned with how to deal with the fallout from a nuclear accident. They show that after the 1986 Chernobyl disaster the British government didn’t have a clue. It gave out the phone number of the pool of government drivers as the helpline number for information about radiation poisoning.
But there is a serious, recurring problem. The rate of document releases from the archives has increased recently but so has the amount of files that are missing or officially withheld.
This year some 190 of the 490 files scheduled for release from the Prime Minister’s office have been retained by the government.
Withheld files include dossiers on the 1988 Lockerbie bombing, the Scott arms-to-Iraq Inquiry, and the basing of US cruise missiles in Britain.
Even dozens of government files covering Britain’s European policy in the early 1990s have been held back. Of the 45 European files due to be released, the Cabinet Office has retained 38.
Worse, about another 1,000 files have gone missing after being removed by civil servants, according to a Freedom of Information request.
Officially they are “misplaced while on loan to a government department”. Documents on Britain’s war in Northern Ireland, British colonial rule in Palestine, tests on polio vaccines and much more have supposedly vanished.
Documents can be hugely important in fighting miscarriages of justice.
In 2014 investigators uncovered a1977 letter from the then home secretary, Merlyn Rees, to the prime minister of the day, James Callaghan. In the letter, Rees claimed that ministers had given permission for torture to be used in Northern Ireland. The information had been withheld from the European Court of Human Rights.
The same year the government said it could not release information about the CIA’s “extraordinary rendition” programme because the files had suffered “water damage”. They were water boarded, so to speak.
In 2013 it emerged that more than one million documents that should have been declassified were being unlawfully kept at a high-security compound in Buckinghamshire.
Their existence came to light when Kenyans who had been tortured during the 1950s Mau Mau rebellion took the government to the high court.
In other instances, papers from within files have been carefully selected and taken away.
Documents on Britain’s war in Northern Ireland and British colonial rule in Palestine have supposedly vanished.
Foreign Office officials removed a small number of papers in 2015 from a file concerning the 1978 murder of a dissident Bulgarian journalist.
The Ministry of Defence refused to consider a number of files for release under the Freedom of Information Act on the grounds that they may have been exposed to asbestos.
The files concerned arms sales to Saudi Arabia, operations by British special forces in Indonesia and torture techniques.
Perhaps most symbolic is the fate of the file on the infamous Zinoviev letter – in which MI6 officers nearly 100 years ago plotted to bring about the downfall of the first Labour government. It has vanished after Home Office civil servants took it away.
The Home Office declined to say why it was taken or when or how it was lost. Nor would it say whether any copies had been made.
The letter was originally published as a slur in 1924 in the Daily Mail newspaper. It took the British state 75 years to admit it was a fake – but it is still covering it up.
Irish governments abandoned the Birmingham Six to rot in jail
The Birmingham Six’s Paddy Hill accused successive Irish governments of abandoning the Six in their fight for freedom, documents released in Ireland reveal.
In a handwritten letter from his cell in HMP Gartree on 10 September 1987, Paddy said they had been offered nothing but false hope and false promises.
He wrote, “The British system don’t know how to spell the word JUSTICE never mind dispensing it.”
Hill's damning assessment of those who jailed him for 16 years is almost word for word what he delivered when he grabbed a microphone on the street outside the Old Bailey in 1991.
The Birmingham Six - Paddy Hill, Hugh Callaghan, Gerard Hunter, Richard McIlkenny, William Power and John Walker - were wrongly jailed for life in 1975 in England for the IRA bombings which killed 21 people.
After long campaigns their convictions were eventually quashed in the appeal court.
At the time of the letter Paddy had been in jail while four different Irish prime ministers took office. He said governments had done little or nothing to support him and the five others.
“The only thing successive Irish governments have done is help to keep INNOCENT IRISHMEN in prison,” he wrote. At the time of the letter he had been in jail for 12 years.
“I won’t be seeing any representatives of any Irish party. As far as I’m concerned they are all a load of shit,” he wrote.
“I had enough of them when they visited us at Long Lartin and they still haven’t had the courage to publicly declare that we are innocent and that we were TORTURED + FRAMED for something we know nothing about.”
After long campaigns the convictions were eventually quashed
Hill added,“We learned a long time ago that we could expect little or no help from the Irish government and everything to date bears that out.
“The only thing they have done for us is to give us FALSE HOPE and FALSE PROMISES. We got this far without the help of the Irish government and we will prove our INNOCENCE without their help.”
In a separate open letter from Paddy earlier that year he said the Birmingham Six were willing to undergo hypnosis, the truth drug and lie detector tests to prove their innocence.
Another file revealed Northern Ireland's SDLP deputy leader Seamus Mallon told an Irish official in August 1987 that he was nervous of getting involved in campaigns for the Birmingham Six, Guildford Four and the Maguire Seven at Westminster.
He thought campaigners, including Chris Mullin who was later a Labour MP and Labour’s Clare Short, were using them for “political propaganda”.