private server configuration

The FBI was unable to confirm hackers broke into Clinton’s system, but it cites an inability to gather enough evidence to do so.

The FBI Clinton email investigation’s final report, released on this day, states, “FBI investigation and forensic analysis did not find evidence confirming that Clinton’s email server systems were compromised by cyber means.” (Elsewhere in the report, it is mentioned that one email account on the server appears to have been broken into by hackers.)

A generic sample of what an attempted hack would look like in the log data. (Credit: public domain)

But the report goes on to state, “The FBI’s inability to recover all server equipment and the lack of complete server log data for the relevant time period limited the FBI’s forensic analysis of the server systems. As a result, FBI cyber analysis relied, in large part, on witness statements, email correspondence, and related forensic content found on other devices to understand the setup, maintenance, administration, and security of the server systems.”

Elsewhere in the report, it is noted that the FBI was unable to recover any of 13 the BlackBerry mobile devices Clinton used while or shortly after her tenure as secretary of state, a laptop containing a back-up of her emails was lost, the server most recently containing her emails was wiped with BleachBit software, the server used for her first two months in office was also lost, hard drive back-ups made were also lost, and so on.  (Federal Bureau of Investigation, 9/2/2016)

At the conclusion of the FBI’s investigation on July 5, 2016, FBI Director James Comey said there was no “direct evidence” Clinton’s email account had been successfully hacked. But the next day, the New York Times reported, “both private experts and federal investigators immediately understood his meaning: It very likely had been breached, but the intruders were far too skilled to leave evidence of their work.”

FBI Director James Comey says Clinton’s private server was less secure than the State Department’s computer network or a commercial email provider.

160707JamesComeyJackGruberUSAToday

Comey testifies to the House Benghazi Committee on July 7, 2016. (Credit: Jack Gruber / USA Today)

In a Congressional hearing, Comey says, “The challenge of security is not binary, it’s just degrees of security. [Clinton’s private server] was less secure than one at the State Department, or as I said, even one at a private commercial provider like a Gmail.” (CNN, 7/7/2016)

Representative Rod Blum (R) asks, “Director Comey, are you implying in [your comments] that the private email servers of Secretary Clinton’s were perhaps less secure than a Gmail account that is used for free by a billion people around this planet?”

Comey replies, “Yes. And I’m not looking to pick on Gmail. Their security is actually pretty good; the weakness is individual users. But, yes, Gmail has full-time security staff and thinks about patching, and logging, and protecting their systems in a way that was not the case here.”

Blum also comments, “I know some security experts in the industry. I check with them. The going rate to hack into somebody’s Gmail account, $129. For corporate emails, they can be hacked for $500 or less. If you want to hack into an IP address, it’s around $100. I’m sure the FBI can probably do it cheaper. This is the going rate.” (CNN, 7/7/2016)

Comey’s comments indicate it is “very likely” Clinton’s emails were hacked, but solid proof may never be found.

In a July 5, 2016 public speech, FBI Director James Comey addresses the possibility that Clinton’s emails were accessed by outsiders. He says, “We did not find direct evidence that Secretary Clinton’s personal email domain, in its various configurations since 2009, was successfully hacked. But, given the nature of the system and of the actors potentially involved, we assess that we would be unlikely to see such direct evidence. We do assess that hostile actors gained access to the private commercial email accounts of people with whom Secretary Clinton was in regular contact from her personal account. We also assess that Secretary Clinton’s use of a personal email domain was both known by a large number of people and readily apparent. She also used her personal email extensively while outside the United States, including sending and receiving work-related emails in the territory of sophisticated adversaries. Given that combination of factors, we assess it is possible that hostile actors gained access to Secretary Clinton’s personal email account.” (Federal Bureau of Investigation, 7/5/2016)

The next day, the New York Times reports that although Comey said there was no “direct evidence” Clinton’s email account had been successfully hacked, “both private experts and federal investigators immediately understood his meaning: It very likely had been breached, but the intruders were far too skilled to leave evidence of their work.”

The Times says that Comey’s comments were a “blistering” critique of Clinton’s “email practices that left Mrs. Clinton’s systems wide open to Russian and Chinese hackers, and an array of others.” However, “the central mystery — who got into the system, if anyone — may never be resolved.”

Adam Segal (Credit: public domain)

Adam Segal (Credit: public domain)

Adam Segal, a cybersecurity expert at the Council on Foreign Relations (CFR), says, “Reading between the lines and following Comey’s logic, it does sound as if the FBI believes a compromise of Clinton’s email is more likely than not. Sophisticated attackers would have known of the existence of the account, would have targeted it, and would not have been seen.”

Before Comey’s comments, Clinton and her spokespeople had said on numerous occasions that her server had never been hacked. In an October 2015 interview, President Obama came to a similar conclusion about her server: “I don’t think it posed a national security problem.”

The Times also comments that Comey’s “most surprising suggestion” may have been his comment that Clinton used her private email while in the territory of “sophisticated adversaries.” This is understood to mean China and Russia and possibly a few more countries.

Former government cybersecurity expert James Lewis says, “If she used it in Russia or China, they almost certainly picked it up.” (The New York Times, 7/6/2016)

Cybersecurity consultant Morgan Wright says the most likely suspects are Russia, China and Israel, “in that order.”

Ben Johnson, a former National Security Agency official and security strategist, says “Certainly foreign military and intelligence services” would have targeted Clinton’s emails. “They’re going to have a lot of means and motives to do this.” He also says it wasn’t just likely countries such as China and Russia, but “any country that’s looking to potentially have adversarial relations with us or just [desires] more relations with us.” He specifically cites Middle East countries specifically as having a likely motive. (Politico, 7/5/2016)

FBI Director Comey announces he will not recommend Clinton’s indictment on any charge, but he calls her “extremely careless” in handling highly classified information.

FBI Director James Comey announces his recommendation for Clinton and her aides on July 5, 2016. (Credit: Cliff Owen / The Associated Press)

FBI Director James Comey announces his recommendation in a press conference on July 5, 2016. (Credit: Cliff Owen / The Associated Press)

FBI Director James Comey gives a public speech in front of a group of reporters. The timing is surprising, since this brings an end to the FBI’s investigation of Clinton’s email practices, and just a Sunday and the Fourth of July holiday separate this from the FBI’s interview of Clinton on July 2, 2016. Comey spends most of his speech criticizing Clinton, but ends it by saying he will not recommend that the Justice Department pursue any indictment of Clinton or her aides.

Comey’s fifteen-minute speech includes the following information, in order, with key phrases bolded to assist in understanding.

Comey begins by describing the FBI investigation:

  • The investigation started with a referral from Intelligence Community Inspector General Charles McCullough, and “focused on whether classified information was transmitted” on Clinton’s personal email server during her time as secretary of state. It specifically “looked at whether there is evidence classified information was improperly stored or transmitted on that personal system, in violation of a federal statute making it a felony to mishandle classified information either intentionally or in a grossly negligent way, or a second statute making it a misdemeanor to knowingly remove classified information from appropriate systems or storage facilities.” The FBI “also investigated to determine whether there is evidence of computer intrusion in connection with the personal email server by any foreign power, or other hostile actors.”
  • The FBI found that Clinton “used several different servers and administrators of those servers during her four years at the State Department, and used numerous mobile devices to view and send email on that personal domain. As new servers and equipment were employed, older servers were taken out of service, stored, and decommissioned in various ways…”
  • The FBI analyzed the over 30,000 work emails that Clinton did turn over to the State Department in December 2014, working with other US government departments to determine which emails contained truly classified information at the time they were sent, and which ones were justifiably classified later.
  • James Comey (Credit: Fox News)

    James Comey (Credit: Fox News)

    From the group of 30,068 emails Clinton returned to the State Department, “110 emails in 52 email chains have been determined by the owning agency to contain classified information at the time they were sent or received. Eight of those chains contained information that was ‘top secret’ at the time they were sent; 36 chains contained ‘secret’ information at the time; and eight contained ‘confidential’ information, which is the lowest level of classification. Separate from those, about 2,000 additional emails were ‘up-classified’ to make them ‘confidential’; the information in those had not been classified at the time the emails were sent.”

  • It had previously been reported that the FBI had recovered most or all of the 31,830 emails that Clinton had deleted, allegedly because they contained personal information only. However, Comey reveals that was not the case, and thousands of emails were not recovered. He gives an example of how when one of Clinton’s servers was decommissioned in 2013, the email was removed and broken up into millions of fragments.
  • The FBI “discovered several thousand work-related emails” that were not included in the 30,068 emails Clinton returned to the State Department, even though Clinton claimed under oath that she had returned all her work-related emails. The FBI found these after they “had been deleted over the years and we found traces of them on devices that supported or were connected to the private email domain.” Others were found in the archived government email accounts of other government employees whom Clinton frequently communicated with. Still others were found “from the laborious review of the millions of email fragments” of the server decommissioned in 2013.
  • Out of these additional work emails, three were classified at the time they were sent or received – none at the ‘top secret’ level, one at the ‘secret’ level, and two at the ‘confidential’ level. None were found to have been deemed classified later.
  • Furthermore, Comey claims “we found no evidence that any of the additional work-related emails were intentionally deleted in an effort to conceal them. Our assessment is that, like many email users, Secretary Clinton periodically deleted emails or emails were purged from the system when devices were changed. Because she was not using a government account—or even a commercial account like Gmail—there was no archiving at all of her emails, so it is not surprising that we discovered emails that were not on Secretary Clinton’s system in 2014, when she produced the 30,000 emails to the State Department.”
  • 160705DeletingAttorneys

    The three Clinton attorneys who deleted emails are David Kendall (left), Cheryl Mills (center), and Heather Samuelson (right). (Credit: public domain)

    However, he also admits that “It could also be that some of the additional work-related emails we recovered were among those deleted as ‘personal’ by Secretary Clinton’s lawyers when they reviewed and sorted her emails for production in 2014.” He claims that the three lawyers who sorted the emails for Clinton in late 2014 (David Kendall, Cheryl Mills, and Heather Samuelson) “did not individually read the content of all of her emails…” Instead, they used keyword searches to determine which emails were work related, and it is “highly likely their search terms missed some work-related emails” that were later found by the FBI elsewhere.

  • Comey states it is “likely” that some emails may have disappeared forever. because Clinton’s three lawyers “deleted all emails they did not return to State, and the lawyers cleaned their devices in such a way as to preclude complete forensic recovery.” But he says that after interviews and technical examination, “we believe our investigation has been sufficient to give us reasonable confidence there was no intentional misconduct in connection with that sorting effort.”

Comey then begins stating his findings:

  • “Although we did not find clear evidence that Secretary Clinton or her colleagues intended to violate laws governing the handling of classified information, there is evidence that they were extremely careless in their handling of very sensitive, highly classified information.”
  • As an example, he points out that “seven email chains concern matters that were classified at the ‘Top Secret/Special Access Program’ [TP/SAP] level when they were sent and received. These chains involved Secretary Clinton both sending emails about those matters and receiving emails from others about the same matters. There is evidence to support a conclusion that any reasonable person in Secretary Clinton’s position, or in the position of those government employees with whom she was corresponding about these matters, should have known that an unclassified system was no place for that conversation.”
  • He adds that it was a similar situation with emails classified at the “secret” level when they were sent, although he doesn’t specify how many.
  • He comments, “None of these emails should have been on any kind of unclassified system, but their presence is especially concerning because all of these emails were housed on unclassified personal servers not even supported by full-time security staff, like those found at departments and agencies of the US government—or even with a commercial service like Gmail.”
  • He notes that “only a very small number of the emails containing classified information bore markings indicating the presence of classified information. But even if information is not marked ‘classified’ in an email, participants who know or should know that the subject matter is classified are still obligated to protect it.”
  • He then criticizes the State Department as a whole. The FBI found evidence that “the security culture” of the State Department “was generally lacking in the kind of care for classified information found elsewhere in the government.” This was especially true regarding the use of unclassified email systems.
  • Then he addresses whether “hostile actors” were able to gain access to Clinton’s emails. Although no direct evidence of any successful hacking was found, he points out that “given the nature of the system and of the actors potentially involved, we assess that we would be unlikely to see such direct evidence. We do assess that hostile actors gained access to the private commercial email accounts of people with whom Secretary Clinton was in regular contact from her personal account. We also assess that Secretary Clinton’s use of a personal email domain was both known by a large number of people and readily apparent. She also used her personal email extensively while outside the United States, including sending and receiving work-related emails in the territory of sophisticated adversaries. Given that combination of factors, we assess it is possible that hostile actors gained access to Secretary Clinton’s personal email account.”

After laying out the evidence of what the FBI found, Comey moves to the FBI’s recommendation to the Justice Department. He admits that it is highly unusual to publicly reveal the FBI’s recommendation, but “in this case, given the importance of the matter, I think unusual transparency is in order.”

James Comey (Credit: NPR)

James Comey (Credit: NPR)

Then he comes to these conclusions:

  • “Although there is evidence of potential violations of the statutes regarding the handling of classified information, our judgment is that no reasonable prosecutor would bring such a case. Prosecutors necessarily weigh a number of factors before bringing charges. There are obvious considerations, like the strength of the evidence, especially regarding intent. Responsible decisions also consider the context of a person’s actions, and how similar situations have been handled in the past.”
  • To justify this decision, he claims he examined other cases involving the mishandling or removal of classified information, and “we cannot find a case that would support bringing criminal charges on these facts. All the cases prosecuted involved some combination of clearly intentional and willful mishandling of classified information; or vast quantities of materials exposed in such a way as to support an inference of intentional misconduct; or indications of disloyalty to the United States; or efforts to obstruct justice. We do not see those things here.”
  • He then says, “To be clear, this is not to suggest that in similar circumstances, a person who engaged in this activity would face no consequences. To the contrary, those individuals are often subject to security or administrative sanctions. But that is not what we are deciding now. As a result, although the Department of Justice makes final decisions on matters like this, we are expressing to Justice our view that no charges are appropriate in this case.”
  • He concludes by saying the FBI’s investigation was done competently, honestly, and independently, and without any kind of outside influence.

He doesn’t address the possibility of recommending the indictment of any of Clinton’s aides or other figures like Sid Blumenthal or Justin Cooper. He also doesn’t make any mention of the Clinton Foundation, though there have been media reports the FBI has been investigating it as well. After finishing his speech, he leaves without taking any questions from the media. (Federal Bureau of Investigation, 7/5/2016)

The company managing Clinton’s private server is worried they will be blamed for a change of policy that results in the deletion of Clinton’s emails.

Platte River Networks (PRN) has been managing Clinton’s private email server. According to a New York Post article in September 2016, around August 2015, PRN wants to double check their behavior after media reports that the FBI is investigating Clinton’s server. “Company execs scrambled to find proof that Clinton’s reps had months earlier asked to cut the retention of emails from 60 days to 30 days.”

Paul Combetta (left) Bill Thornton (right) (Credit: AP)

Paul Combetta (left) Bill Thornton (right) (Credit: AP)

On August 12, 2015, PRN employee Bill Thornton writes, “OK, we may want to work with our attorneys to draft up something that absolves us of that question. I can only assume that will be the first and last question for us, ‘Why did we have backups of the system since the time of inception, then decide to cut them back to just 60 or 30 days?’ If we can get that from them in writing, I would feel a whole lot better about this.”

The other PRN employee who has been actively managing the Clinton account with Thornton, Bill Combetta, responds that he believes the request was made to PRN by phone.

An email exchange between the two on the same topic several days later will make clear that the Clinton representatives are employees of Clinton Executive Services Corp. (CESC) the Clinton family company that has been paying PRN. (The New York Post, 9/18/2016)

A company recommends improving security for Clinton’s server, which is still in use, but the FBI wants no changes.

At some point in August 2015, employees at Datto, Inc., a company that specializes in backing up computer data, realize that a private server they have been backing up belongs to Clinton. The server is being managed by Platte River Networks (PRN), and Datto made the connection after media reports revealed PRN’s role.

According to an unnamed Datto official, due to worries about the “sensitive high profile nature of the data,” Datto then recommends that PRN should upgrade security by adding sophisticated encryption technology to its backup systems.

150801AndyBoianFoxNews

Andy Boian (Credit: Fox News)

PRN spokesperson Andy Boian later acknowledges receiving upgrade requests from Datto, but he says, “It’s not that we ignored them, but the FBI had told us not to change or adjust anything.”

Boian adds, however, the company did not take Datto’s concerns to the FBI.

The newest version of the server is still in use by the Clintons’ personal office at the time, despite being in news headlines since March 2015. (The Washington Post, 10/7/2015)

On August 12, 2015, the FBI takes an older version of the server from PRN’s control. The FBI doesn’t realize Clinton’s emails were moved from the old server to the new one. They eventually will figure this out and take the new server away as well, on October 3, 2015.

Changes are made to the security settings of Clinton’s private server after its existence was revealed in the media.

In the days following a New York Times article revealing Clinton’s use of her private server, Cheryl Mills, who is one of Clinton’s lawyers as well as her former chief of staff, requests that Platte River Networks (PRN), the computer company managing Clinton’s server, conduct a complete inventory of all equipment related to the server. Two unnamed PRN employees do so.

This results in some changes to the server’s security settings around March 7, 2015. According to a September 2016 FBI report, these changes “include disabling the server’s public-facing VPN page and switching from SSL protocol to TLS to increase security.”

The FBI will explain: “TLS is a protocol that ensures privacy between communicating applications, such as web browsing, email, and instant-messaging, with their users on the Internet. TLS ensures that no third-party eavesdrops on the two-way conummication. TLS is the successor to SSL and is considered more secure.” (Federal Bureau of Investigation, 9/2/2016)

Cheryl Mills has a computer company check on the condition of Clinton’s private server after the media makes Clinton’s use of the server front-page news.

On March 2, 2015, the New York Times publishes a front-page story about Clinton’s emails practices and her use of a private email server.

The Equinix data center in Secaucus, NY. (Credit: public domain)

In the days following the publication of the article, Cheryl Mills, who is one of Clinton’s lawyers as well as her former chief of staff, requests that Platte River Networks (PRN), the computer company managing Clinton’s server, conduct a complete inventory of all equipment related to the server.

In response to this request, an unnamed PRN employee travels to the Equinix data center in Secaucus, New Jersey, where the server is located, to conduct an onsite review of the equipment. At the same time, another unnamed PRN employee logs in to the server remotely to check on it.

This will result in some changes to the security settings of the server  around March 7, 2015. Additionally, many emails (other than Clinton’s) are deleted from the server on March 8, 2015. (Federal Bureau of Investigation, 9/2/2016)

The company managing Clinton’s private server fails to fully test its security vulnerabilities.

Johannes Ullrich (Credit: LinkedIn)

Johannes Ullrich (Credit: LinkedIn)

Platte River Networks (PRN) is the company managing Clinton’s private server. Due to a wave of hacking attacks on the server following the public revelation of the server on March 2, 2015, PRN considers doing penetration testing. That  means hiring someone to try to hack the server in order to expose its vulnerabilities so they can be fixed.

Cybersecurity expert Johannes Ullrich will later comment, “It’s a good idea, and it’s also commonly done.”

However, the penetration testing never happens. It isn’t clear why. (The New York Post, 9/18/2016) (Federal Bureau of Investigation, 9/2/2016)

A surge of hacking attempts follows the revelation of Clinton’s use of a private email server in the media.

On March 2, 2015, a New York Times article publicly reveals Clinton’s use of a personal email account and private server to conduct government business. The FBI’s Clinton email investigation will later identify an increased number of login attempts to her server and its associated domain controller just after this article comes out.

According to the FBI in September 2016, “Forensic analysis revealed none of the login attempts were successful. [The] FBI investigation also identified an increase in unauthorized login attempts into the Apple iCloud account likely associated with Clinton’s email address during this time period.” (Clinton’s email address, which had been publicly revealed in March 2013, was still used as the user name for the account.) “Investigation determined all potentially suspicious Apple iCloud login attempts were unsuccessful.”

Despite all this, Clinton does not simply turn the server off. Instead, Platte River Networks (PRN) employees, who are managing the server, make some security improvements around March 7, 2015.

PRN staff also discuss the possibility of conducting penetration testing against the server to highlight vulnerabilities, so they can be fixed. However, the penetration testing ultimately doesn’t happen. (Federal Bureau of Investigation, 9/2/2016)