The internet metadata collection program was halted in 2011 for 'operational and resource reasons'. Photograph: Pablo Martinez Monsivais/AP
The Obama administration for more than two years permitted the National Security Agency to continue collecting vast amounts of records detailing the email and internet usage of Americans, according to secret documents obtained by the Guardian.
The documents indicate that under the program, launched in 2001, a federal judge sitting on the secret surveillance panel called the Fisa court would approve a bulk collection order for internet metadata "every 90 days". A senior administration official confirmed the program, stating that it ended in 2011.
The collection of these records began under the Bush administration's wide-ranging warrantless surveillance program, collectively known by the NSA codename Stellar Wind.
According to a top-secret draft report by the NSA's inspector general – published for the first time today by the Guardian – the agency began "collection of bulk internet metadata" involving "communications with at least one communicant outside the United States or for which no communicant was known to be a citizen of the United States".
Eventually, the NSA gained authority to "analyze communications metadata associated with United States persons and persons believed to be in the United States", according to a 2007 Justice Department memo, which is marked secret.
The Guardian revealed earlier this month that the NSA was collecting the call records of millions of US Verizon customers under a Fisa court order that, it later emerged, is renewed every 90 days. Similar orders are in place for other phone carriers.
The internet metadata of the sort NSA collected for at least a decade details the accounts to which Americans sent emails and from which they received emails. It also details the internet protocol addresses (IP) used by people inside the United States when sending emails – information which can reflect their physical location. It did not include the content of emails.
"The internet metadata collection program authorized by the Fisa court was discontinued in 2011 for operational and resource reasons and has not been restarted," Shawn Turner, the Obama administration's director of communications for National Intelligence, said in a statement to the Guardian.
"The program was discontinued by the executive branch as the result of an interagency review," Turner continued. He would not elaborate further.
But while that specific program has ended, additional secret NSA documents seen by the Guardian show that some collection of Americans' online records continues today. In December 2012, for example, the NSA launched one new program allowing it to analyze communications with one end inside the US, leading to a doubling of the amount of data passing through its filters.
"Toll records, phone records like this, that don't include any content, are not covered by the fourth amendment because people don't have a reasonable expectation of privacy in who they called and when they called," Cole testified to the House intelligence committee on June 18. "That's something you show to the phone company. That's something you show to many, many people within the phone company on a regular basis."
But email metadata is different. Customers' data bills do not itemize online activity by detailing the addresses a customer emailed or the IP addresses from which customer devices accessed the internet.
Internal government documents describe how revealing these email records are. One 2008 document, signed by the US defense secretary and attorney general, states that the collection and subsequent analysis included "the information appearing on the 'to,' 'from' or 'bcc' lines of a standard email or other electronic communication" from Americans.
In reality, it is hard to distinguish email metadata from email content. Distinctions that might make sense for telephone conversations and data about those conversations do not always hold for online communications.
"The calls you make can reveal a lot, but now that so much of our lives are mediated by the internet, your IP [internet protocol] logs are really a real-time map of your brain: what are you reading about, what are you curious about, what personal ad are you responding to (with a dedicated email linked to that specific ad), what online discussions are you participating in, and how often?" said Julian Sanchez of the Cato Institute.
"Seeing your IP logs – and especially feeding them through sophisticated analytic tools – is a way of getting inside your head that's in many ways on par with reading your diary," Sanchez added.
The purpose of this internet metadata collection program is detailed in the full classified March 2009 draft report prepared by the NSA's inspector general (IG).
One function of this internet record collection is what is commonly referred to as "data mining", and which the NSA calls "contact chaining". The agency "analyzed networks with two degrees of separation (two hops) from the target", the report says. In other words, the NSA studied the online records of people who communicated with people who communicated with targeted individuals.
Contact chaining was considered off-limits inside the NSA before 9/11. In the 1990s, according to the draft IG report, the idea was nixed when the Justice Department "told NSA that the proposal fell within one of the Fisa definitions of electronic surveillance and, therefore, was not permissible when applied to metadata associated with presumed US persons".
"NSA was authorized to acquire telephony and internet metadata for communications with at least one communicant outside the United States or for which no communicant was known to be a citizen of the United States," the draft report states.
George W Bush briefly "discontinued" that bulk internet metadata collection, involving Americans, after a dramatic rebellion in March 2004 by senior figures at the Justice Department and FBI, as the Washington Post first reported. One of the leaders of that rebellion was deputy attorney general James Comey, whom Barack Obama nominated last week to run the FBI.
But Comey's act of defiance did not end the IP metadata collection, the documents reveal. It simply brought it under a newly created legal framework.
As soon as the NSA lost the blessing under the president's directive for collecting bulk internet metadata, the NSA IG report reads, "DoJ [the Department of Justice] and NSA immediately began efforts to recreate this authority."
The DoJ quickly convinced the Fisa court to authorize ongoing bulk collection of email metadata records. On 14 July 2004, barely two months after Bush stopped the collection, Fisa court chief judge Collen Kollar-Kotelly legally blessed it under a new order – the first time the surveillance court exercised its authority over a two-and-a-half-year-old surveillance program.
Kollar-Kotelly's order "essentially gave NSA the same authority to collect bulk internet metadata that it had under the PSP [Bush's program], except that it specified the datalinks from which NSA could collect, and it limited the number of people that could access the data".
The purpose of that memorandum was to advise Mukasey of the Pentagon's view that these restrictions were excessive, and to obtain permission for the NSA to expand its "contact chains" deeper into Americans' email records. The agency, the memo noted, already had "in its databases a large amount of communications metadata associated with persons in the United States".
But, Wainstein continued, "NSA's present practice is to 'stop' when a chain hits a telephone number or [internet] address believed to be used by a United States person."
Wainstein told Mukasey that giving NSA broader leeway to study Americans' online habits would give the surveillance agency, ironically, greater visibility into the online habits of foreigners – NSA's original mandate.
"NSA believes that it is over-identifying numbers and addresses that belong to United States persons and that modifying its practice to chain through all telephone numbers and addresses, including those reasonably believed to be used by a United States person," Wainstein wrote, "will yield valuable foreign intelligence information primarily concerning non-United States persons outside the United States."
The procedures "would clarify that the National Security Agency (NSA) may analyze communications metadata associated with United States persons and persons believed to be in the United States", Wainstein wrote.
In October 2007, Robert Gates, the secretary of defense, signed a set of "Supplemental Procedures" on internet metadata, including what it could do with Americans' data linked in its contact chains. Mukasey affixed his signature to the document in January 2008.
"NSA will continue to disseminate the results of its contact chaining and other analysis of communications metadata in accordance with current procedures governing the dissemination of information concerning US persons," the document states, without detailing the "current procedures".
It was this program that continued for more than two years into the Obama administration.
Turner, the director of national intelligence spokesman, did not respond to the Guardian's request for additional details of the metadata program or the reasons why it was stopped.
A senior administration official queried by the Washington Post denied that the Obama administration was "using this program" to "collect internet metadata in bulk", but added: "I'm not going to say we're not collecting any internet metadata."
The documents indicate that under the program, launched in 2001, a federal judge sitting on the secret surveillance panel called the Fisa court would approve a bulk collection order for internet metadata "every 90 days". A senior administration official confirmed the program, stating that it ended in 2011.
The collection of these records began under the Bush administration's wide-ranging warrantless surveillance program, collectively known by the NSA codename Stellar Wind.
According to a top-secret draft report by the NSA's inspector general – published for the first time today by the Guardian – the agency began "collection of bulk internet metadata" involving "communications with at least one communicant outside the United States or for which no communicant was known to be a citizen of the United States".
Eventually, the NSA gained authority to "analyze communications metadata associated with United States persons and persons believed to be in the United States", according to a 2007 Justice Department memo, which is marked secret.
The Guardian revealed earlier this month that the NSA was collecting the call records of millions of US Verizon customers under a Fisa court order that, it later emerged, is renewed every 90 days. Similar orders are in place for other phone carriers.
The internet metadata of the sort NSA collected for at least a decade details the accounts to which Americans sent emails and from which they received emails. It also details the internet protocol addresses (IP) used by people inside the United States when sending emails – information which can reflect their physical location. It did not include the content of emails.
"The internet metadata collection program authorized by the Fisa court was discontinued in 2011 for operational and resource reasons and has not been restarted," Shawn Turner, the Obama administration's director of communications for National Intelligence, said in a statement to the Guardian.
"The program was discontinued by the executive branch as the result of an interagency review," Turner continued. He would not elaborate further.
But while that specific program has ended, additional secret NSA documents seen by the Guardian show that some collection of Americans' online records continues today. In December 2012, for example, the NSA launched one new program allowing it to analyze communications with one end inside the US, leading to a doubling of the amount of data passing through its filters.
What your email metadata reveals
The Obama administration argues that its internal checks on NSA surveillance programs, as well as review by the Fisa court, protect Americans' privacy. Deputy attorney general James Cole defended the bulk collection of Americans' phone records as outside the scope of the fourth amendment's protections against unreasonable searches and seizures."Toll records, phone records like this, that don't include any content, are not covered by the fourth amendment because people don't have a reasonable expectation of privacy in who they called and when they called," Cole testified to the House intelligence committee on June 18. "That's something you show to the phone company. That's something you show to many, many people within the phone company on a regular basis."
But email metadata is different. Customers' data bills do not itemize online activity by detailing the addresses a customer emailed or the IP addresses from which customer devices accessed the internet.
Internal government documents describe how revealing these email records are. One 2008 document, signed by the US defense secretary and attorney general, states that the collection and subsequent analysis included "the information appearing on the 'to,' 'from' or 'bcc' lines of a standard email or other electronic communication" from Americans.
In reality, it is hard to distinguish email metadata from email content. Distinctions that might make sense for telephone conversations and data about those conversations do not always hold for online communications.
"The calls you make can reveal a lot, but now that so much of our lives are mediated by the internet, your IP [internet protocol] logs are really a real-time map of your brain: what are you reading about, what are you curious about, what personal ad are you responding to (with a dedicated email linked to that specific ad), what online discussions are you participating in, and how often?" said Julian Sanchez of the Cato Institute.
"Seeing your IP logs – and especially feeding them through sophisticated analytic tools – is a way of getting inside your head that's in many ways on par with reading your diary," Sanchez added.
The purpose of this internet metadata collection program is detailed in the full classified March 2009 draft report prepared by the NSA's inspector general (IG).
One function of this internet record collection is what is commonly referred to as "data mining", and which the NSA calls "contact chaining". The agency "analyzed networks with two degrees of separation (two hops) from the target", the report says. In other words, the NSA studied the online records of people who communicated with people who communicated with targeted individuals.
Contact chaining was considered off-limits inside the NSA before 9/11. In the 1990s, according to the draft IG report, the idea was nixed when the Justice Department "told NSA that the proposal fell within one of the Fisa definitions of electronic surveillance and, therefore, was not permissible when applied to metadata associated with presumed US persons".
How the US government came to collect Americans' email records
The collection of email metadata on Americans began in late 2001, under a top-secret NSA program started shortly after 9/11, according to the documents. Known as Stellar Wind, the program initially did not rely on the authority of any court – and initially restricted the NSA from analyzing records of emails between communicants wholly inside the US."NSA was authorized to acquire telephony and internet metadata for communications with at least one communicant outside the United States or for which no communicant was known to be a citizen of the United States," the draft report states.
George W Bush briefly "discontinued" that bulk internet metadata collection, involving Americans, after a dramatic rebellion in March 2004 by senior figures at the Justice Department and FBI, as the Washington Post first reported. One of the leaders of that rebellion was deputy attorney general James Comey, whom Barack Obama nominated last week to run the FBI.
But Comey's act of defiance did not end the IP metadata collection, the documents reveal. It simply brought it under a newly created legal framework.
As soon as the NSA lost the blessing under the president's directive for collecting bulk internet metadata, the NSA IG report reads, "DoJ [the Department of Justice] and NSA immediately began efforts to recreate this authority."
The DoJ quickly convinced the Fisa court to authorize ongoing bulk collection of email metadata records. On 14 July 2004, barely two months after Bush stopped the collection, Fisa court chief judge Collen Kollar-Kotelly legally blessed it under a new order – the first time the surveillance court exercised its authority over a two-and-a-half-year-old surveillance program.
Kollar-Kotelly's order "essentially gave NSA the same authority to collect bulk internet metadata that it had under the PSP [Bush's program], except that it specified the datalinks from which NSA could collect, and it limited the number of people that could access the data".
How NSA gained more power to study Americans' online habits
The Bush email metadata program had restrictions on the scope of the bulk email records the NSA could analyze. Those restrictions are detailed in a legal memorandum written in a 27 November 2007, by assistant attorney general Kenneth Wainstein to his new boss, attorney general Michael Mukasey, who had taken office just a few weeks earlier.The purpose of that memorandum was to advise Mukasey of the Pentagon's view that these restrictions were excessive, and to obtain permission for the NSA to expand its "contact chains" deeper into Americans' email records. The agency, the memo noted, already had "in its databases a large amount of communications metadata associated with persons in the United States".
But, Wainstein continued, "NSA's present practice is to 'stop' when a chain hits a telephone number or [internet] address believed to be used by a United States person."
Wainstein told Mukasey that giving NSA broader leeway to study Americans' online habits would give the surveillance agency, ironically, greater visibility into the online habits of foreigners – NSA's original mandate.
"NSA believes that it is over-identifying numbers and addresses that belong to United States persons and that modifying its practice to chain through all telephone numbers and addresses, including those reasonably believed to be used by a United States person," Wainstein wrote, "will yield valuable foreign intelligence information primarily concerning non-United States persons outside the United States."
The procedures "would clarify that the National Security Agency (NSA) may analyze communications metadata associated with United States persons and persons believed to be in the United States", Wainstein wrote.
In October 2007, Robert Gates, the secretary of defense, signed a set of "Supplemental Procedures" on internet metadata, including what it could do with Americans' data linked in its contact chains. Mukasey affixed his signature to the document in January 2008.
"NSA will continue to disseminate the results of its contact chaining and other analysis of communications metadata in accordance with current procedures governing the dissemination of information concerning US persons," the document states, without detailing the "current procedures".
It was this program that continued for more than two years into the Obama administration.
Turner, the director of national intelligence spokesman, did not respond to the Guardian's request for additional details of the metadata program or the reasons why it was stopped.
A senior administration official queried by the Washington Post denied that the Obama administration was "using this program" to "collect internet metadata in bulk", but added: "I'm not going to say we're not collecting any internet metadata."
