Facing Data Deluge, Secret U.K. Spying Report Warned of Intelligence Failure
Ryan Gallagher
The Intercept, June 7, 2016
A secret report warned that British spies may have put lives at risk because their surveillance systems were sweeping up more data than could be analyzed, leading them to miss clues to possible security threats.
The concern was sent to top British government officials in an explosive classified document, which outlined methods being developed by the United Kingdom’s domestic intelligence agency to covertly monitor internet communications.
The Security Service, also known as MI5, had become the “principal collector and exploiter” of digital communications within the U.K., the eight-page report noted, but the agency’s surveillance capabilities had “grown significantly over the last few years.”
MI5 “can currently collect (whether itself or through partners …) significantly more than it is able to exploit fully,” the report warned. “This creates a real risk of ‘intelligence failure’ i.e. from the Service being unable to access potentially life-saving intelligence from data that it has already collected.”
A draft copy of the report, obtained by The Intercept from National Security Agency whistleblower Edward Snowden, is marked with the classification “U.K. Secret” and dated February 12, 2010. It was prepared by British spy agency officials to brief the government’s Cabinet Office and Treasury Department about the U.K.’s surveillance capabilities.
Notably, three years after the report was authored, two Islamic extremists killed and attempted to decapitate a British soldier, Lee Rigby, on a London street. An investigation into the incident found that the two perpetrators were well-known to MI5, but the agency had missed significant warning signs about the men, including records of phone calls one of them had made to an al Qaeda-affiliated radical in Yemen, and an online message in which the same individual had discussed in graphic detail his intention to murder a soldier.
The new revelations raise questions about whether problems sifting through the troves of data collected by British spies may have been a factor in the failure to prevent the Rigby killing. But they are also of broader relevance to an ongoing debate in the U.K. about surveillance. In recent months, the British government has been trying to pass a new law, the Investigatory Powers Bill, which would grant MI5 and other agencies access to more data.
Silkie Carlo, a policy officer at the London-based human rights group Liberty, told The Intercept that the details contained in the secret report highlighted the need for a comprehensive independent review of the proposed new surveillance powers.
“Intelligence whistleblowers have warned that the agencies are drowning in data — and now we have it confirmed from the heart of the U.K. government,” Carlo said. “If our agencies have risked missing ‘life-saving intelligence’ by collecting ‘significantly’ more data than they can analyze, how can they justify casting the net yet wider in the toxic Investigatory Powers Bill?”
The British government’s Home Office, which handles media requests related to MI5, declined to comment for this story.
“Lack of staff and tools”
The leaked report outlines efforts by British agencies to conduct both “large-scale” and “small-scale” eavesdropping of domestic communications within the U.K. It focuses primarily on an MI5 program called DIGINT, or digital intelligence, which was aimed at transforming the agency’s ability to covertly monitor internet communications.
DIGINT was established for counterterrorism purposes, and “more generally for wider national security purposes,” the report said. The program was described as being focused on “the activities of key investigative targets, and on those exploitation activities that will drive greatest investigative benefits with respect to U.K. domestic threats.”
The amount of data being collected, however, proved difficult for MI5 to handle. In March 2010, in another secret report, concerns were reiterated about the agency’s difficulties processing the material it was harvesting. “There is an imbalance between collection and exploitation capabilities, resulting in a failure to make effective use of some of the intelligence collected today,” the report noted. “With the exception of the highest priority investigations, a lack of staff and tools means that investigators are presented with raw and unfiltered DIGINT data. Frequently, this material is not fully assessed because of the significant time required to review it.”
97 percent of the calls, messages, and data the program had collected were found to have been “not viewed” by the authorities.
The problem was not unique to MI5.
Many of the agency’s larger-scale surveillance operations were being conducted in coordination with the National Technical Assistance Centre, a unit of the electronic eavesdropping agency Government Communications Headquarters, better known as GCHQ.
The Centre plays a vital but little-known role. One of its main functions is to act as a kind of intermediary, managing the highly sensitive data-sharing relationships that exist among British telecommunications companies and law enforcement and spy agencies.
Perhaps the most important program the Centre helps deliver is code-named PRESTON, which covertly intercepts phone calls, text messages, and internet data sent or received by people or organizations in the U.K. who have been named as surveillance targets on warrants signed off by a government minister.
A top-secret 2009 study found that, in one six-month period, the PRESTON program had intercepted more than 5 million communications. Remarkably, 97 percent of the calls, messages, and data it had collected were found to have been “not viewed” by the authorities.
The authors of the study were alarmed because PRESTON was supposedly focused on known suspects, and yet most of the communications it was monitoring appeared to be getting ignored — meaning crucial intelligence could have been missed.
“Only a small proportion of the Preston Traffic is viewed,” they noted. “This is of concern as the collection is all warranted.”
preston-study-5
Chart: A top-secret study outlines PRESTON data collection by month.
“Politically contentious”
For most of the last decade, successive British governments have attempted to obtain more surveillance powers, but their efforts have met with public opposition and ultimately failed. The present government’s effort to push through a sweeping surveillance law — the Investigatory Powers Bill — is currently being considered by the Parliament.
Documents provided by Snowden show that the U.K.’s intelligence and security agencies have wanted to obtain new powers to store domestic data about internet communications to address the “growing range of services available to internet users.” This reflects the position that has been adopted publicly in recent years by the government, which has argued that expanded internet surveillance is necessary to keep up with changes in technology.
However, the Snowden documents also reveal a more candid internal assessment of the need for bolstered spy laws and shine light on major aspects of the U.K.’s existing surveillance apparatus that government and security officials have not publicly acknowledged in their pursuit of the new powers.
In one document dated from 2012, GCHQ stated that it was “not dependent” on a new surveillance law coming into force, presumably due to the extensive capabilities already at its disposal. GCHQ added that new powers were of greater importance to the U.K.’s law enforcement agencies, which were facing “a significant decline” in ability to intercept communications due to people increasingly using internet services — as opposed to conventional landlines and cellphones — to talk or exchange messages.
But passing a new surveillance law would be a “politically contentious [and] technically complex” process, GCHQ said in the document. In the meantime, therefore, it devised something of a workaround by creating a secret stop-gap surveillance solution for law enforcement officials.
As part of a program named MILKWHITE, GCHQ made some of its huge troves of metadata about people’s online activities accessible to MI5, London’s Metropolitan Police, the tax agency Her Majesty’s Revenue and Customs, the Serious Organized Crime Agency (now merged into the National Crime Agency), the Police Service of Northern Ireland, and an obscure Scotland-based surveillance unit called the Scottish Recording Centre.
Metadata reveals information about communications — such as the sender and recipient of an email, or the phone numbers someone called and at what time — but not the written content of the message or the audio of the call. GCHQ’s definition of metadata is broad and also encompasses location data that can be used to track people’s movements, login passwords, and website browsing histories, as The Intercept has previously revealed.
The MILKWHITE program was developed as early as September 2009, and it seems to have been operational under both the Labour and the Conservative-Liberal Democrat governments of that period. One of its purposes was to allow law enforcement agencies and MI5 to sift through the troves of metadata to discover internet “selectors” for their surveillance targets — meaning unique identifiers, such as a username or IP address, that can be used to home in on and monitor a person’s online activities.
“It now appears it has been ‘business as usual’ for the tax man to access mass internet data for years.”
GCHQ focuses primarily on intercepting foreign communications that are “external” to the U.K. But in the process of doing so — by tapping into international cables that carry phone calls and internet traffic between countries — the agency vacuums up large quantities of data on British calls, emails, and web browsing habits, too. It is this British data — some of which appears to have been made accessible through MILKWHITE — that would be of most interest to MI5, police, and tax officers, as it is their role to conduct “internal” investigations within the U.K.
A GCHQ document dated from late 2010 indicated that MILKWHITE was storing data about people’s usage of smartphone chat apps like WhatsApp and Viber, instant messenger services such as Jabber, and social networking websites, including Facebook, MySpace, and LinkedIn. Access to the data was provided to law enforcement through an “internet data unit” hosted by the Serious Organized Crime Agency and it was accessible to tax investigators through what one GCHQ document described as established “business as usual” channels.
By March 2011, GCHQ noted that there was “increasing customer demand” for the service offered by MILKWHITE and the agency planned to grow its capacity, seeking £20.8 million ($30.6 million) to update the program’s “advanced analytics” capabilities and to maintain its “bulk” storage of metadata records. “Bulk” is a term GCHQ uses to refer to large troves of data that are not focused on individual targets; rather, they include millions and in some cases billions of records about ordinary people’s communications and internet activity.
Carlo, the policy analyst with Liberty, said the revelations about MILKWHITE suggested members of Parliament had been misled about how so-called bulk data is handled. “While MPs have been told that bulk powers have been used only by the intelligence community, it now appears it has been ‘business as usual’ for the tax man to access mass internet data for years,” she said. “This vindicates the warnings of security experts and the call by opposition parties for an urgent, independent review of bulk powers. The compromise review recently announced is a poor substitute and without the time and technical expertise, will struggle to address this issue of national importance.”
GCHQ declined to answer questions for this story. A spokesperson for the agency said in a statement: “It is long-standing policy that we do not comment on intelligence matters. Furthermore, all of GCHQ’s work is carried out in accordance with a strict legal and policy framework, which ensures that our activities are authorized, necessary and proportionate, and that there is rigorous oversight, including from the Secretary of State, the Interception and Intelligence Services Commissioners and the Parliamentary Intelligence and Security Committee. All our operational processes rigorously support this position. In addition, the U.K.’s interception regime is entirely compatible with the European Convention on Human Rights.”
Documents published with this article:
Digint Narrative
Milkwhite
Preston Business Processes
Communications Capabilities Development Programme
NTAC Overview
Preston Architecture
Digint Imbalance
Mobile Apps Checkpoint Meeting Archives
Preston Study
Ryan Gallagher
The Intercept, June 7, 2016
A secret report warned that British spies may have put lives at risk because their surveillance systems were sweeping up more data than could be analyzed, leading them to miss clues to possible security threats.
The concern was sent to top British government officials in an explosive classified document, which outlined methods being developed by the United Kingdom’s domestic intelligence agency to covertly monitor internet communications.
The Security Service, also known as MI5, had become the “principal collector and exploiter” of digital communications within the U.K., the eight-page report noted, but the agency’s surveillance capabilities had “grown significantly over the last few years.”
MI5 “can currently collect (whether itself or through partners …) significantly more than it is able to exploit fully,” the report warned. “This creates a real risk of ‘intelligence failure’ i.e. from the Service being unable to access potentially life-saving intelligence from data that it has already collected.”
A draft copy of the report, obtained by The Intercept from National Security Agency whistleblower Edward Snowden, is marked with the classification “U.K. Secret” and dated February 12, 2010. It was prepared by British spy agency officials to brief the government’s Cabinet Office and Treasury Department about the U.K.’s surveillance capabilities.
Notably, three years after the report was authored, two Islamic extremists killed and attempted to decapitate a British soldier, Lee Rigby, on a London street. An investigation into the incident found that the two perpetrators were well-known to MI5, but the agency had missed significant warning signs about the men, including records of phone calls one of them had made to an al Qaeda-affiliated radical in Yemen, and an online message in which the same individual had discussed in graphic detail his intention to murder a soldier.
The new revelations raise questions about whether problems sifting through the troves of data collected by British spies may have been a factor in the failure to prevent the Rigby killing. But they are also of broader relevance to an ongoing debate in the U.K. about surveillance. In recent months, the British government has been trying to pass a new law, the Investigatory Powers Bill, which would grant MI5 and other agencies access to more data.
Silkie Carlo, a policy officer at the London-based human rights group Liberty, told The Intercept that the details contained in the secret report highlighted the need for a comprehensive independent review of the proposed new surveillance powers.
“Intelligence whistleblowers have warned that the agencies are drowning in data — and now we have it confirmed from the heart of the U.K. government,” Carlo said. “If our agencies have risked missing ‘life-saving intelligence’ by collecting ‘significantly’ more data than they can analyze, how can they justify casting the net yet wider in the toxic Investigatory Powers Bill?”
The British government’s Home Office, which handles media requests related to MI5, declined to comment for this story.
“Lack of staff and tools”
The leaked report outlines efforts by British agencies to conduct both “large-scale” and “small-scale” eavesdropping of domestic communications within the U.K. It focuses primarily on an MI5 program called DIGINT, or digital intelligence, which was aimed at transforming the agency’s ability to covertly monitor internet communications.
DIGINT was established for counterterrorism purposes, and “more generally for wider national security purposes,” the report said. The program was described as being focused on “the activities of key investigative targets, and on those exploitation activities that will drive greatest investigative benefits with respect to U.K. domestic threats.”
The amount of data being collected, however, proved difficult for MI5 to handle. In March 2010, in another secret report, concerns were reiterated about the agency’s difficulties processing the material it was harvesting. “There is an imbalance between collection and exploitation capabilities, resulting in a failure to make effective use of some of the intelligence collected today,” the report noted. “With the exception of the highest priority investigations, a lack of staff and tools means that investigators are presented with raw and unfiltered DIGINT data. Frequently, this material is not fully assessed because of the significant time required to review it.”
97 percent of the calls, messages, and data the program had collected were found to have been “not viewed” by the authorities.
The problem was not unique to MI5.
Many of the agency’s larger-scale surveillance operations were being conducted in coordination with the National Technical Assistance Centre, a unit of the electronic eavesdropping agency Government Communications Headquarters, better known as GCHQ.
The Centre plays a vital but little-known role. One of its main functions is to act as a kind of intermediary, managing the highly sensitive data-sharing relationships that exist among British telecommunications companies and law enforcement and spy agencies.
Perhaps the most important program the Centre helps deliver is code-named PRESTON, which covertly intercepts phone calls, text messages, and internet data sent or received by people or organizations in the U.K. who have been named as surveillance targets on warrants signed off by a government minister.
A top-secret 2009 study found that, in one six-month period, the PRESTON program had intercepted more than 5 million communications. Remarkably, 97 percent of the calls, messages, and data it had collected were found to have been “not viewed” by the authorities.
The authors of the study were alarmed because PRESTON was supposedly focused on known suspects, and yet most of the communications it was monitoring appeared to be getting ignored — meaning crucial intelligence could have been missed.
“Only a small proportion of the Preston Traffic is viewed,” they noted. “This is of concern as the collection is all warranted.”
preston-study-5
Chart: A top-secret study outlines PRESTON data collection by month.
“Politically contentious”
For most of the last decade, successive British governments have attempted to obtain more surveillance powers, but their efforts have met with public opposition and ultimately failed. The present government’s effort to push through a sweeping surveillance law — the Investigatory Powers Bill — is currently being considered by the Parliament.
Documents provided by Snowden show that the U.K.’s intelligence and security agencies have wanted to obtain new powers to store domestic data about internet communications to address the “growing range of services available to internet users.” This reflects the position that has been adopted publicly in recent years by the government, which has argued that expanded internet surveillance is necessary to keep up with changes in technology.
However, the Snowden documents also reveal a more candid internal assessment of the need for bolstered spy laws and shine light on major aspects of the U.K.’s existing surveillance apparatus that government and security officials have not publicly acknowledged in their pursuit of the new powers.
In one document dated from 2012, GCHQ stated that it was “not dependent” on a new surveillance law coming into force, presumably due to the extensive capabilities already at its disposal. GCHQ added that new powers were of greater importance to the U.K.’s law enforcement agencies, which were facing “a significant decline” in ability to intercept communications due to people increasingly using internet services — as opposed to conventional landlines and cellphones — to talk or exchange messages.
But passing a new surveillance law would be a “politically contentious [and] technically complex” process, GCHQ said in the document. In the meantime, therefore, it devised something of a workaround by creating a secret stop-gap surveillance solution for law enforcement officials.
As part of a program named MILKWHITE, GCHQ made some of its huge troves of metadata about people’s online activities accessible to MI5, London’s Metropolitan Police, the tax agency Her Majesty’s Revenue and Customs, the Serious Organized Crime Agency (now merged into the National Crime Agency), the Police Service of Northern Ireland, and an obscure Scotland-based surveillance unit called the Scottish Recording Centre.
Metadata reveals information about communications — such as the sender and recipient of an email, or the phone numbers someone called and at what time — but not the written content of the message or the audio of the call. GCHQ’s definition of metadata is broad and also encompasses location data that can be used to track people’s movements, login passwords, and website browsing histories, as The Intercept has previously revealed.
The MILKWHITE program was developed as early as September 2009, and it seems to have been operational under both the Labour and the Conservative-Liberal Democrat governments of that period. One of its purposes was to allow law enforcement agencies and MI5 to sift through the troves of metadata to discover internet “selectors” for their surveillance targets — meaning unique identifiers, such as a username or IP address, that can be used to home in on and monitor a person’s online activities.
“It now appears it has been ‘business as usual’ for the tax man to access mass internet data for years.”
GCHQ focuses primarily on intercepting foreign communications that are “external” to the U.K. But in the process of doing so — by tapping into international cables that carry phone calls and internet traffic between countries — the agency vacuums up large quantities of data on British calls, emails, and web browsing habits, too. It is this British data — some of which appears to have been made accessible through MILKWHITE — that would be of most interest to MI5, police, and tax officers, as it is their role to conduct “internal” investigations within the U.K.
A GCHQ document dated from late 2010 indicated that MILKWHITE was storing data about people’s usage of smartphone chat apps like WhatsApp and Viber, instant messenger services such as Jabber, and social networking websites, including Facebook, MySpace, and LinkedIn. Access to the data was provided to law enforcement through an “internet data unit” hosted by the Serious Organized Crime Agency and it was accessible to tax investigators through what one GCHQ document described as established “business as usual” channels.
By March 2011, GCHQ noted that there was “increasing customer demand” for the service offered by MILKWHITE and the agency planned to grow its capacity, seeking £20.8 million ($30.6 million) to update the program’s “advanced analytics” capabilities and to maintain its “bulk” storage of metadata records. “Bulk” is a term GCHQ uses to refer to large troves of data that are not focused on individual targets; rather, they include millions and in some cases billions of records about ordinary people’s communications and internet activity.
Carlo, the policy analyst with Liberty, said the revelations about MILKWHITE suggested members of Parliament had been misled about how so-called bulk data is handled. “While MPs have been told that bulk powers have been used only by the intelligence community, it now appears it has been ‘business as usual’ for the tax man to access mass internet data for years,” she said. “This vindicates the warnings of security experts and the call by opposition parties for an urgent, independent review of bulk powers. The compromise review recently announced is a poor substitute and without the time and technical expertise, will struggle to address this issue of national importance.”
GCHQ declined to answer questions for this story. A spokesperson for the agency said in a statement: “It is long-standing policy that we do not comment on intelligence matters. Furthermore, all of GCHQ’s work is carried out in accordance with a strict legal and policy framework, which ensures that our activities are authorized, necessary and proportionate, and that there is rigorous oversight, including from the Secretary of State, the Interception and Intelligence Services Commissioners and the Parliamentary Intelligence and Security Committee. All our operational processes rigorously support this position. In addition, the U.K.’s interception regime is entirely compatible with the European Convention on Human Rights.”
Documents published with this article:
Digint Narrative
Milkwhite
Preston Business Processes
Communications Capabilities Development Programme
NTAC Overview
Preston Architecture
Digint Imbalance
Mobile Apps Checkpoint Meeting Archives
Preston Study
No comments:
Post a Comment