Datto cloud service
One company that possessed Clinton’s emails is accused of having shockingly poor security.
From around June 2013 until August 2015, Clinton’s private server containing her emails from her time as secretary of state was managed by Platte River Networks. But another company, Datto Inc., was making monthly back-up copies of all the server’s data in the Internet cloud. Datto has 600 employees and is valued at $1 billion, but two people tell the Daily Mail that the company is extremely incompetent.
Marc Tamarin, president of Virtual IT Consulting, was a Datto business partner from 2009 until early 2016. He says he frequently worked with Datto’s technical support, but “Those guys were really morons. They weren’t qualified to handle our back-up and that was the biggest concern for us. … If they’re inept at the basic principles of technology, how are they going to handle something advanced like security? Most companies like mine trust their vendor that they are doing due diligence. I’ve never heard anything this bad before in my life, the dataincompetence was shocking.”
An unnamed former employee, who spent three years at the company, has even more complaints. “If you’re talking about high-level data security, at the political, presidential level, the security level of data [at Datto] … was nowhere near something that could have been protected from a good hacker that knows how to spread out their points at which to infiltrate. It’s not something that Datto was focused on. It was more about getting the data off-site quickly and cost-effectively than securing the data and keeping it from being hacked. There’s no doubt in my mind that someone could easily hack them – even today.”
He calls Datto’s security “a joke.” He claims a potential hacker could walk in off the street and sit down at an unused computer and access all the company’s data. There were no security guards, the receptionists didn’t ask questions of strangers, there was no key card access or other security features, passwords were not regularly changed, and so on. People who said they had lost their security pass would be let in without questions. Unused computers were frequently left on and logged in to the network.
He says, “For years, any Datto employee, even low-level ones, could go in any customer’s device, see their backups, restore files, and delete files.” Oftentimes, Datto customers would find themselves logged into the data of another customer without even wanting to. Datto’s internal servers were hacked in 2010. However, complaints were swept under the rug and security was not improved. (The Daily Mail, 6/30/2016)
The company managing Clinton’s private server learns that another company has been making back-up copies of all the server data in the Internet “cloud” since 2013.
Clinton’s server has been managed by Platte River Networks (PRN) since June 2013. And since that time, PRN has subcontracted Datto, Inc. to make periodic back-ups of all the data on the server. PRN has thought that the back-ups have been only made through a device attached to the server called the Datto SIRIS S2000.
However, on August 1, 2015, an unnamed PRN employee notices that data from the server was possibly being sent to an off-site Datto location. On August 6, 2015, Sam Hickler, PRN’s vice president of operations, contacts Datto employee Leif McKinley about this, CCing PRN employees Paul Combetta and Treve Suazo.
McKinley confirms that, due to a misunderstanding, Datto has been making periodic back-ups of the server data through the Internet “cloud” as well as locally through the device. Furthermore, periodic back-ups have been made this way since June 2013.
Suazo, the CEO of PRN, tells Datto on August 6, 2015, that “This is a problem.” This is because the Clinton Executive Services Corp. (CESC), the Clinton family company that hired PRN to manage the server, explicitly stated from the beginning that they didn’t want any remote back-ups to be made. Thus, PRN employees tell Datto not to delete whatever data was stored in the cloud, and instead work to get it back to the control of PRN.
On August 7, 2015, Datto and PRN employees discuss saving the data on a thumb drive and sending it to PRN. Then, according to an email from one unnamed PRN employee to another, they would have Datto “wipe [the data] from the cloud.”
This is according to a letter that will be sent in October 5, 2015 to Datto CEO Austin McChord by Senator Ron Johnson (R). Johnson is chair of the Senate Homeland Security and Government Affairs Committee, and is conducting oversight of the FBI’s Clinton email investigation. However, Johnson will be unable to determine what happened next, such as if the thumb drive was sent and the data was wiped. Furthermore, McChord will not be able to reveal that information to Johnson because Datto needs PRN’s permission to share that information and PRN won’t give it. (US Congress, 9/12/2016) (US Congress, 9/12/2016)
An Internet cloud back-up of Clinton’s server is deleted at this time, despite the company managing the server seemingly not knowing the cloud copy exists.
On November 19, 2015, an unnamed Datto executive will be interviewed by the FBI. Datto had provided back-up service and equipment to Platte Rivers Networks (PRN) when PRN was managing Clinton’s private server from June 2013 onwards. It will later be reported that in early August 2015, PRN employees discovered that in addition to a Datto back-up device attached to Clinton’s server, Datto had been also backing up Clinton’s server to the Internet “cloud.” Some internal PRN emails from early August 2015 show some employees acting surprised after being told about this.
However, according to a later FBI summary of the Datto executive’s interview, he said that PRN must have known about the cloud back-up all along. “As evidence, [he] stated the partner portal, that PRN had log-in credentials to, had a feature displaying backed-up data an options to ‘delete cloud’ or ‘delete local.’ [He] stated PN would have seen their back-ups under ‘delete cloud.'”
More crucially, during the interview, the FBI will show him a Datto document “indicating email records were manually deleted from the Datto secure cloud back-ups of the [Clinton] server in March 2015.” He then will tell the FBI that it couldn’t have been a Datto employee who made the deletions, because there would have been a work ticket created showing that. Furthermore, IP addresses associated with the deletions indicate that someone from PRN must have done it, although PRN had a shared account so it can’t be proven who exactly made the deletions. (Federal Bureau of Investigation, 10/17/2016)
A Datto letter sent to the FBI in October 2015 will indicate that Datto technical experts reviewed administrative files and discovered through the device’s Internet interface that a series of deletions took place on the device on March 31, 2015, between 11:27 a.m. and 12:41 a.m. Furthermore, a much greater amount of data had been “deleted automatically based on the local device’s then-configured pruning parameters.” (US Congress, 9/12/2016) It is unclear if this refers to data deleted from the local Datto device or the Internet cloud back-up.
Although it is unknown who made these deletions, in a May 2016 FBI interview, PRN employee Paul Combetta will confess to deleting all of Clinton’s emails on her server as well as the Datto back-up device in precisely this time period, between March 25, 2015 and March 31, 2015.
Some of Clinton’s emails are later recovered due to a back-up of computer files made on this date.
In June 2013, Platte River Networks (PRN) takes over management of Clinton’s server. Late in the month, they replace the server with a new one and then transfer the data to it. They subcontract with the company Datto, Inc. and purchase a device called the Datto SIRIS S2000 to make periodic back-ups of all the data on the new server. The first such back-up takes place on June 24, 2013.
But data is still being transferred from the old server to the new one. The June 29, 2013 back-up will later prove to be the most important one for FBI investigators, as it apparently is the first one after the data transfer is completed. From that point onwards, emails from Clinton’s four years as secretary of state are likely to only get lost from the server, not added.
The FBI will later report that all of Clinton’s emails at the start of Clinton’s tenure as secretary of state, from January 23, 2009 to March 17, 2009 were missing from the over 30,000 emails Clinton handed over. But the FBI’s Clinton investigation recovered some these emails because they were “captured through a Datto backup on June 29, 2013. However, the emails obtained are likely only a subset of the emails sent or received by Clinton during this time period.”
Clinton’s first server was replaced around March 18, 2009 by the same server that PRN then decided to replace in June 2013. But presumably some of the emails on the first server were transferred to the second server, from instance by being in email inboxes, and then were transferred again by PRN to the newest (and third) server.
One thing that isn’t clear is how many of the emails from after March 18, 2009 were recovered by the FBI. It also isn’t clear if the FBI recovered emails from a Datto device attached to the new server, or if it was from a copy of the data that Datto kept in the “cloud,” over the Internet. (Federal Bureau of Investigation, 9/2/2016)
Clinton’s server is relocated and then replaced by a new server, but the old server keeps running.
After Platte River Networks (PRN) is selected to manage Clinton’s private email server on May 31, 2013, the company decides to immediately relocate the server and then also replace it with a better one.
PRN assigns two employees to manage the new server (which will be the third server used by Clinton). The FBI will later redact the names of these two employees, but it is known that one of them works remotely from his home in some unnamed town and will handle the day-to-day administration of the server, and the other one works at PRN’s headquarters in Denver, Colorado, and handles all hardware installation and any required physical maintenance of the server. Media reports will later name the two employees as Paul Combetta, who works from Rhode Island, and Bill Thornton.
The employee at PRN’s headquarters (who logically would be Thorton) works with Clinton’s computer technician Bryan Pagliano to help with the transition. Around June 4, 2013, this person is granted administrator access to the server, as well as any accompanying services.
On June 23, 2013, this person travels to Clinton’s house in Chappaqua, New York, shuts down the server, and transports it to a data center in Secaucus, New Jersey, run by Equinix, Inc. This older server will stay at the Equinix facility until it is given to the FBI on October 3, 2015.
The PRN headquarters employee (still likely to be Thornton) turns the old server back on in the Equinix data center so users can continue to access their email accounts. Then he spends a few days there setting up a new server. When he leaves, all the physical equipment for the new server is successfully installed except for an intrusion detection device, which Equinix installs later, once it gets shipped.
Meanwhile, the PRN employee who works remotely (Combetta) does his remote work to get the new server online. Around June 30, 2013, this employee begins to transfer all the email accounts from the old server to the new one. After several days, all email accounts hosted on the presidentclinton.com, wjcoffice.com, and clintonemail.com domains are transferred. However, PRN keeps the old server online at the Equinix data center along with the new server to ensure email continues to be delivered. But the old server no longer hosts email services for the Clintons.
According to an FBI report made public in September 2016, “The new Clinton email server hosted email for [Hillary] Clinton, President Clinton, [redacted], and their respective staffs.”
This same FBI report will explain that the new server consists of the following equipment: “a Dell PowerEdge R620 server hosting four virtual machines, including four separate virtual machines for Microsoft Exchange email hosting, a BES for the management of BlackBerry devices, a domain controller to authenticate password requests, and an administrative server to manage the other three virtual machines, a Datto SfRlS 2000 to store onsite and remote backups of the server system, a CloudJacket device for intrusion prevention, two Dell switches, and two Fortinet Fortigate 80C firewalls.” (Federal Bureau of Investigation, 9/2/2016)
The FBI report will not make entirely clear what happens to the data on the old server. But a September 2015 Washington Post article will assert that after PRN moved all the data onto a new server, everything on the original server was deleted until it is “blank.” However, it was not wiped, which means having the old files overwritten several times with new data until they can never be recovered. (The Washington Post, 9/12/2015)
- Bill Thornton
- Bryan Pagliano
- Chappaqua (New York)
- Datto cloud service
- Datto SIRIS S2000
- Dell PowerEdge R620
- Equinix data center
- Equinix Inc.
- FBI's Clinton email investigaton final report
- Federal Bureau of Investigations (FBI)
- New Jersey
- New York (New York)
- Paul Combetta
- Platte River Networks (PRN)
- private server
- private server configuration
- Secaucus (New Jersey)
A device is bought to make back-ups of Clinton’s private server, but a Clinton company makes clear it doesn’t want any back-up data stored remotely.
On May 31, 2013, Platte River Networks (PRN) takes over management of Clinton’s private server. On the same day, PRN buys a Datto SIRIS S2000 data storage device, which is made by Datto, Inc. Over the next month, this is attached to Clinton’s server to provide periodic back-up copies of the data on the server. PRN sends a bill for the device to Clinton Executive Service Corp. (CESC), which is a Clinton family company.
CESC employees work with PRN employees on how the Datto device is configured. Datto offers a local back-up and a remote back-up using the Internet “cloud.” CESC asks for a local back-up and specifically requests that no data be stored in the Internet cloud at any time.
However, due to an apparent misunderstanding, back-up copies of the server will be periodically made both locally and in the cloud. This will only be discovered by PRN as a whole in August 2015. (US Congress, 9/12/2016)
However, despite internal PRN emails from August 2015 indicating many PRN employees didn’t know about the Datto cloud back-up until that time, the FBI will later find evidence that an unknown PRN employee deleted data from the cloud back-up in March 2015, meaning that at least one PRN employee had to have known about the cloud back-up by that time.