The server was running LAMP and Drupal, but that's not how it got hacked, so I don't think that mattered.
I gave my server a DNS name of drupal.samsclass.info. I Tweeted a link to it and added it to my homepage, to attract attacks.
Dec 23 11:14:45 drupal sshd[29125]: Accepted password for root from 59.45.79.50 port 48173 ssh2
Dec 23 19:59:24 drupal sshd[24570]: Accepted password for root from 59.45.79.50 port 24904 ssh2
Dec 24 07:11:34 drupal sshd[21526]: Accepted password for root from 175.126.82.235 port 55355 ssh2
However, the 7 AM Dec 24 login added a rootkit to the server, as shown below.
A couple more ssh logins, but still nothing in bash_history Add the bash_history HTML stuff to startup Make another snapshot: 6-bash
Thu Dec 24 07:05:01 2015 Total violations found: 3
Thu Dec 24 07:10:01 2015 Total violations found: 3
Thu Dec 24 07:15:01 2015 Total violations found: 25
Thu Dec 24 07:20:01 2015 Total violations found: 25
md5sum shows that there are only two different icald files:
root@drupal:~# md5sum /usr/bin/icaldzcoww
b877518b1dedd733be145bd3f2d6f6f3 /usr/bin/icaldzcoww
root@drupal:~# md5sum /etc/init.d/icaldzcoww
a7ea5e651fb3bc3e8993a69c1331a9df /etc/init.d/icaldzcoww
You have new mail in /var/mail/root
root@drupal:~# md5sum /etc/rc1.d/S90icaldzcoww
a7ea5e651fb3bc3e8993a69c1331a9df /etc/rc1.d/S90icaldzcoww
root@drupal:~# md5sum /etc/rc2.d/S90icaldzcoww
a7ea5e651fb3bc3e8993a69c1331a9df /etc/rc2.d/S90icaldzcoww
root@drupal:~# md5sum /etc/rc3.d/S90icaldzcoww
a7ea5e651fb3bc3e8993a69c1331a9df /etc/rc3.d/S90icaldzcoww
root@drupal:~# md5sum /etc/rc4.d/S90icaldzcoww
a7ea5e651fb3bc3e8993a69c1331a9df /etc/rc4.d/S90icaldzcoww
root@drupal:~# md5sum /etc/rc5.d/S90icaldzcoww
a7ea5e651fb3bc3e8993a69c1331a9df /etc/rc5.d/S90icaldzcoww
root@drupal:~# cp /etc/rc5.d/S90icaldzcoww /root/Dropbox/
root@drupal:~# netstat -pant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 159.203.215.48:44073 175.126.82.12:6002 ESTABLISHED 1124/route -n
Wireshark shows connections once per second or so:
Most are just handshakes, but one shows a repetitive ASCII string. I don't know what it is, perhaps just an identifier of my bot.
root@drupal:~# ls -al
total 72448
drwx------ 10 root root 4096 Dec 24 07:11 .
drwxr-xr-x 23 root root 4096 Dec 24 11:22 ..
drwxr-xr-x 2 root root 4096 Dec 24 07:11 i
#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
for i in `cat /proc/net/dev|grep :|awk -F: {'print $1'}`; do ifconfig $i up& done
cp /lib/libudev.so /lib/libudev.so.6
/lib/libudev.so.6
S90icaldzcoww contains:
#!/bin/sh
# chkconfig: 12345 90 90
# description: icaldzcoww
### BEGIN INIT INFO
# Provides: icaldzcoww
# Required-Start:
# Required-Stop:
# Default-Start: 1 2 3 4 5
# Default-Stop:
# Short-Description: icaldzcoww
### END INIT INFO
case $1 in
start)
/usr/bin/icaldzcoww
;;
stop)
;;
*)
/usr/bin/icaldzcoww
;;
esac
Searching VirusTotal with the MD5 hash finds it:
ESET-NOD32 a variant of Linux/Xorddos.C
This is a known rootkit:
http://www.scmagazine.com/malware-targets-linux-and-arm-architecture/article/391497/
https://www.fireeye.com/blog/threat-research/2015/02/anatomy_of_a_brutef.html
https://isc.sans.edu/forums/diary/XOR+DDOS+Mitigation+and+Analysis/19827/
http://bartblaze.blogspot.com/2015/09/notes-on-linuxxorddos.html
I checked for more modified files; none found
cd /
root@drupal:/# ls -ltR . > allfiles
cat allfiles2.txt | grep 07:1
drwx------ 10 root root 4096 Dec 24 07:11 root
drwxr-xr-x 2 root root 4096 Dec 24 07:11 cron.hourly
drwxr-xr-x 2 root root 4096 Dec 24 07:11 init.d
drwxr-xr-x 2 root root 4096 Dec 24 07:11 i
-rw-r--r-- 1 root root 3078 Dec 24 07:15 drupal-20151224-071501.twr
-rw-r--r-- 1 root root 2190 Dec 24 07:10 drupal-20151224-071001.twr
-rw-r--r-- 1 root root 3078 Dec 23 07:15 drupal-20151223-071501.twr
-rw-r--r-- 1 root root 3078 Dec 23 07:10 drupal-20151223-071001.twr
-rwxr-xr-x 1 root root 625718 Dec 24 07:11 icaldzcoww
-rw-r--r-- 1 root root 895 Oct 14 07:18 envvars-std
Reverse DNS (PTR record) mail.hyaudi.co.kr, in South Korea.
The reverse DNS record is from hanyoung motors, but it doesn't seem to be used currently by that company. Perhaps the reverse DNS record is leftover from an earlier owner of that IP address.
Nmap shows a Windows box at that address:
Sams-MacBook-Pro:~ sambowne$ sudo nmap 175.126.82.12 -p0-65535 -Pn -A
Starting Nmap 7.01 ( https://nmap.org ) at 2015-12-24 13:04 PST
Nmap scan report for mail.hyaudi.co.kr (175.126.82.12)
Host is up (0.17s latency).
Not shown: 65531 filtered ports
PORT STATE SERVICE VERSION
3389/tcp open tcpwrapped
6001/tcp open tcpwrapped
|_x11-access: ERROR: Script execution failed (use -d to debug)
6003/tcp open tcpwrapped
|_x11-access: ERROR: Script execution failed (use -d to debug)
6004/tcp open tcpwrapped
|_x11-access: ERROR: Script execution failed (use -d to debug)
49154/tcp open msrpc Microsoft Windows RPC
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose|specialized|phone
Running: Microsoft Windows 2008|7|Phone|Vista
OS CPE: cpe:/o:microsoft:windows_server_2008::beta3 cpe:/o:microsoft:windows_server_2008 cpe:/o:microsoft:windows_7::-:professional cpe:/o:microsoft:windows_8 cpe:/o:microsoft:windows_7 cpe:/o:microsoft:windows cpe:/o:microsoft:windows_vista::- cpe:/o:microsoft:windows_vista::sp1
OS details: Microsoft Windows Server 2008 or 2008 Beta 3, Windows Server 2008 R2, Microsoft Windows 7 Professional or Windows 8, Microsoft Windows Embedded Standard 7, Microsoft Windows Phone 7.5 or 8.0, Microsoft Windows Vista SP0 or SP1, Windows Server 2008 SP1, or Windows 7, Microsoft Windows Vista SP2, Windows 7 SP1, or Windows Server 2008
Network Distance: 24 hops
Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows
TRACEROUTE (using port 3389/tcp)
HOP RTT ADDRESS
1 8.19 ms 10.0.0.1
2 16.39 ms 50.156.104.1
3 16.27 ms te-0-7-0-2-sur04.sfmission.ca.sfba.comcast.net (68.87.195.17)
4 17.53 ms be-310-ar01.hayward.ca.sfba.comcast.net (162.151.79.81)
5 13.26 ms hu-0-19-0-0-ar01.santaclara.ca.sfba.comcast.net (68.85.154.253)
6 ...
7 14.65 ms be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158)
8 16.66 ms he-0-13-0-1-pe03.11greatoaks.ca.ibone.comcast.net (68.86.83.138)
9 16.44 ms 144.232.6.77
10 14.59 ms sl-st31-sj-.sprintlink.net (144.232.0.181)
11 12.88 ms 144.232.12.213
12 25.01 ms 144.232.25.78
13 23.02 ms 144.232.24.42
14 25.26 ms sl-st55-la-.sprintlink.net (144.232.25.168)
15 27.54 ms sl-hanar2-864138-0.sprintlink.net (144.232.143.38)
16 167.48 ms 39.115.132.9
17 171.12 ms 39.115.132.198
18 170.13 ms 39.115.132.42
19 174.95 ms 218.50.5.254
20 178.49 ms 175.126.75.38
21 176.60 ms 218.38.138.22
22 ... 23
24 166.48 ms mail.hyaudi.co.kr (175.126.82.12)
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 1620.96 seconds
Tcpdump captured a lot of traffic!
In the future, I'll slow down my networking to avoid this, as explained here.
I got some pcaps from my server via Dropbox :). It's sending a SYN flood, with each packet containing the same long string of gibberish. Perhaps that gibberish is intended to make filtering it more difficult, but I think just blocking abnormally large SYN packets would work.
The target seems to be a Chinese web server at 183.60.110.239.
It seems to have been used as the attacker in a DDoS in March, 2015. So perhaps my server was being used in an attack from one criminal gang on another.