Got blasted by the ssh bot

[DM]

RH7.3 on an intel box.
Looks like one of my neglected machines got nailed by that ssh bot. I'm
using it as a squid server for our remote offices, and I also have some
users who ssh into it for various reasons. I usually block all ssh
except for specific IP addresses and ranges, with iptables. I had an ssh
user at home w/a dynamic ip and, in a moment of laziness, I opened all
ssh "temporarily" and apparently left it that way( I have a script on my
mailserver to prevent my absent-minded professor syndrome by restoring
the default rules hourly).
I'm running tripwire, so I was able to tell what was added/replaced.
I copied known good files from the mailserver of the same version, and
copied most of them back to the compromised machine in single user mode.
However - I rec'd errors while copying some of them, and decided to boot
from floppy and try the same. I rec'd the same error even when booting
from floppy. Here is a list of the files in /bin that were
modified/replaced this is NOT the replaced files, but a list of the
replacement files :
-rwxr-xr-x 1 root root 541096 Oct 24 10:55 bash*
-rwxr-xr-x 1 root root 16424 Oct 24 10:43 chgrp*
-rwxr-xr-x 1 root root 16680 Oct 24 10:44 chmod*
-rwxr-xr-x 1 root root 18280 Oct 24 10:44 chown*
-rwxr-xr-x 1 root root 36360 Oct 24 10:44 cp*
-rwxr-xr-x 1 root root 64705 Oct 24 10:45 cpio*
-rwxr-xr-x 1 root root 28616 Oct 24 10:45 dd*
-rwxr-xr-x 1 root root 26376 Oct 24 10:45 df*
-rwxr-xr-x 1 root root 83064 Oct 24 10:45 ed*
-rwxr-xr-x 1 root root 248748 Oct 24 10:45 gawk*
-rwxr-xr-x 1 root root 248748 Oct 24 10:45 gawk-3.1.0*
-rwxr-xr-x 1 root root 12426 Oct 24 10:46 hostname*
-rwxr-xr-x 1 root root 20104 Oct 24 10:55 ln*
-rwxr-xr-x 1 root root 46888 Oct 24 10:55 ls*
-rwxr-xr-x 1 root root 66492 Oct 24 10:55 mail*
-rwxr-xr-x 1 root root 17992 Oct 24 10:45 mkdir*
-rwxr-xr-x 1 root root 12952 Oct 24 10:55 mt*
-rwxr-xr-x 1 root root 100173 Oct 24 10:55 netstat*
-rwsr-xr-x 1 root root 35192 Oct 24 10:45 ping*
-r-xr-xr-x 1 root root 63304 Oct 24 10:55 ps*
-rwxr-xr-x 1 root root 16700 Oct 24 10:45 setserial*

Also syslogd was modified/replaced.

The files that would not copy were ls, ps, setserial, and /sbin/syslogd
I rec'd Operation not permitted, Permission denied. I had them all (
except syslogd) in a tarball and just tarred them into that directory -
tar -xzvf tarball.tar.gz.

What do I need to do here? I'm clearly missing something critical...


-D

[Wim Godden]

DM wrote:

> The files that would not copy were ls, ps, setserial, and /sbin/syslogd
> I rec'd Operation not permitted, Permission denied. I had them all (
> except syslogd) in a tarball and just tarred them into that directory -
> tar -xzvf tarball.tar.gz.
>
> What do I need to do here? I'm clearly missing something critical...

Best way to go is :
- use the interactive boot method in RedHat and answer NO to everything
- do a 'chattr -i' on all the files
- replace 'kill' just to make sure
- kill ALL processes except the ones required to keep Linux running
- replace the other files
- Check using rootkit hunter

If that doesn't fix it, it's time to backup all the data and reinstall from
scratch.

Wim

[Tauno Voipio]

Wim Godden wrote:
> DM wrote:
>
>
>>The files that would not copy were ls, ps, setserial, and /sbin/syslogd
>>I rec'd Operation not permitted, Permission denied. I had them all (
>>except syslogd) in a tarball and just tarred them into that directory -
>>tar -xzvf tarball.tar.gz.
>>
>>What do I need to do here? I'm clearly missing something critical...
>
>
> Best way to go is :
> - use the interactive boot method in RedHat and answer NO to everything
> - do a 'chattr -i' on all the files
> - replace 'kill' just to make sure
> - kill ALL processes except the ones required to keep Linux running
> - replace the other files
> - Check using rootkit hunter
>
> If that doesn't fix it, it's time to backup all the data and reinstall from
> scratch.

You forgot the first rule:

Pull the network cable - NOW!

--

Tauno Voipio
tauno voipio (at) iki fi

[Roger Parks]

>
> What do I need to do here? I'm clearly missing something critical...
>

sshd.conf:

#PermitRootLogin no ?

[DM]

Roger Parks wrote:
>>What do I need to do here? I'm clearly missing something critical...
>>
>
>
> sshd.conf:
>
> #PermitRootLogin no ?
>
Yep - Thats set okay.

[Scott R. Haven]

DM wrote:
> RH7.3 on an intel box.
> Looks like one of my neglected machines got nailed by that ssh bot. I'm
> using it as a squid server for our remote offices, and I also have some
> users who ssh into it for various reasons. I usually block all ssh
> except for specific IP addresses and ranges, with iptables. I had an ssh
> user at home w/a dynamic ip and, in a moment of laziness, I opened all
> ssh "temporarily" and apparently left it that way( I have a script on my
> mailserver to prevent my absent-minded professor syndrome by restoring
> the default rules hourly).
> I'm running tripwire, so I was able to tell what was added/replaced. I
> copied known good files from the mailserver of the same version, and
> copied most of them back to the compromised machine in single user mode.
> However - I rec'd errors while copying some of them, and decided to boot
> from floppy and try the same. I rec'd the same error even when booting
> from floppy. Here is a list of the files in /bin that were
> modified/replaced this is NOT the replaced files, but a list of the
> replacement files :
> -rwxr-xr-x 1 root root 541096 Oct 24 10:55 bash*
> -rwxr-xr-x 1 root root 16424 Oct 24 10:43 chgrp*
> -rwxr-xr-x 1 root root 16680 Oct 24 10:44 chmod*
> -rwxr-xr-x 1 root root 18280 Oct 24 10:44 chown*
> -rwxr-xr-x 1 root root 36360 Oct 24 10:44 cp*
> -rwxr-xr-x 1 root root 64705 Oct 24 10:45 cpio*
> -rwxr-xr-x 1 root root 28616 Oct 24 10:45 dd*
> -rwxr-xr-x 1 root root 26376 Oct 24 10:45 df*
> -rwxr-xr-x 1 root root 83064 Oct 24 10:45 ed*
> -rwxr-xr-x 1 root root 248748 Oct 24 10:45 gawk*
> -rwxr-xr-x 1 root root 248748 Oct 24 10:45 gawk-3.1.0*
> -rwxr-xr-x 1 root root 12426 Oct 24 10:46 hostname*
> -rwxr-xr-x 1 root root 20104 Oct 24 10:55 ln*
> -rwxr-xr-x 1 root root 46888 Oct 24 10:55 ls*
> -rwxr-xr-x 1 root root 66492 Oct 24 10:55 mail*
> -rwxr-xr-x 1 root root 17992 Oct 24 10:45 mkdir*
> -rwxr-xr-x 1 root root 12952 Oct 24 10:55 mt*
> -rwxr-xr-x 1 root root 100173 Oct 24 10:55 netstat*
> -rwsr-xr-x 1 root root 35192 Oct 24 10:45 ping*
> -r-xr-xr-x 1 root root 63304 Oct 24 10:55 ps*
> -rwxr-xr-x 1 root root 16700 Oct 24 10:45 setserial*
>
> Also syslogd was modified/replaced.
>
> The files that would not copy were ls, ps, setserial, and /sbin/syslogd
> I rec'd Operation not permitted, Permission denied. I had them all (
> except syslogd) in a tarball and just tarred them into that directory -
> tar -xzvf tarball.tar.gz.
>
> What do I need to do here? I'm clearly missing something critical...
>
>
> -D

D,

At this point I'd backup the data and start from scratch.

It'd probably be faster and you'd sleep better at night too.

Scott R. Haven
Sr. Systems Engineer
Paisley Systems Inc.
managed services, consulting, and support
www.paisleysystems.com

[Unruh]

lsattr
chattr
Look at the i attribute.

"Scott R. Haven" <not.real@paisleysystems.com> writes:

>DM wrote:
>> RH7.3 on an intel box.
>> Looks like one of my neglected machines got nailed by that ssh bot. I'm
>> using it as a squid server for our remote offices, and I also have some
>> users who ssh into it for various reasons. I usually block all ssh
>> except for specific IP addresses and ranges, with iptables. I had an ssh
>> user at home w/a dynamic ip and, in a moment of laziness, I opened all
>> ssh "temporarily" and apparently left it that way( I have a script on my
>> mailserver to prevent my absent-minded professor syndrome by restoring
>> the default rules hourly).
>> I'm running tripwire, so I was able to tell what was added/replaced. I
>> copied known good files from the mailserver of the same version, and
>> copied most of them back to the compromised machine in single user mode.
>> However - I rec'd errors while copying some of them, and decided to boot
>> from floppy and try the same. I rec'd the same error even when booting
>> from floppy. Here is a list of the files in /bin that were
>> modified/replaced this is NOT the replaced files, but a list of the
>> replacement files :
>> -rwxr-xr-x 1 root root 541096 Oct 24 10:55 bash*
>> -rwxr-xr-x 1 root root 16424 Oct 24 10:43 chgrp*
>> -rwxr-xr-x 1 root root 16680 Oct 24 10:44 chmod*
>> -rwxr-xr-x 1 root root 18280 Oct 24 10:44 chown*
>> -rwxr-xr-x 1 root root 36360 Oct 24 10:44 cp*
>> -rwxr-xr-x 1 root root 64705 Oct 24 10:45 cpio*
>> -rwxr-xr-x 1 root root 28616 Oct 24 10:45 dd*
>> -rwxr-xr-x 1 root root 26376 Oct 24 10:45 df*
>> -rwxr-xr-x 1 root root 83064 Oct 24 10:45 ed*
>> -rwxr-xr-x 1 root root 248748 Oct 24 10:45 gawk*
>> -rwxr-xr-x 1 root root 248748 Oct 24 10:45 gawk-3.1.0*
>> -rwxr-xr-x 1 root root 12426 Oct 24 10:46 hostname*
>> -rwxr-xr-x 1 root root 20104 Oct 24 10:55 ln*
>> -rwxr-xr-x 1 root root 46888 Oct 24 10:55 ls*
>> -rwxr-xr-x 1 root root 66492 Oct 24 10:55 mail*
>> -rwxr-xr-x 1 root root 17992 Oct 24 10:45 mkdir*
>> -rwxr-xr-x 1 root root 12952 Oct 24 10:55 mt*
>> -rwxr-xr-x 1 root root 100173 Oct 24 10:55 netstat*
>> -rwsr-xr-x 1 root root 35192 Oct 24 10:45 ping*
>> -r-xr-xr-x 1 root root 63304 Oct 24 10:55 ps*
>> -rwxr-xr-x 1 root root 16700 Oct 24 10:45 setserial*
>>
>> Also syslogd was modified/replaced.
>>
>> The files that would not copy were ls, ps, setserial, and /sbin/syslogd
>> I rec'd Operation not permitted, Permission denied. I had them all (
>> except syslogd) in a tarball and just tarred them into that directory -
>> tar -xzvf tarball.tar.gz.
>>
>> What do I need to do here? I'm clearly missing something critical...
>>
>>
>> -D

>D,

>At this point I'd backup the data and start from scratch.

>It'd probably be faster and you'd sleep better at night too.

>Scott R. Haven
>Sr. Systems Engineer
>Paisley Systems Inc.
>managed services, consulting, and support
>www.paisleysystems.com