small brain dump and more

Sometimes it happen that on file system I have file or directory which I cannot delete, even if Im root. Cannot do chown or chmod , ls –al showing all OK, but still cannot delete it. Whatever I try , simple “permission denied” and similar message. Im the root !!! so, what to do ?

drwxrwxrwx 2 apache root 4096 Jan 28 10:08 blog
 -rwxrwxrwx 1 root   root    0 Feb  9  2009 click.jpg
 -rwxrwxrwx 1 root   root    0 Feb  9  2009 first.jpg
 -rwxrwxrwx 1 root   root   11 Feb 10  2009 index.html
 drwxrwxrwx 2 root   root 4096 Feb 10  2009 monitor
 drwxrwxrwx 2 root   root 4096 Feb 10  2009 pl
 -rwxrwxrwx 1 root   root   10 Feb 10  2009 Red.html
 -rwxrwxrwx 1 root   root   17 Feb  8  2009 robots.txt
 -rwxrwxrwx 1 root   root    0 Feb  9  2009 second.jpg
 [root@test]# chown root.root * -R
 [root@test]# unlink robots.txt
 unlink: cannot unlink `robots.txt': Permission denied
 [root@test]# rm -v index.html
 rm: remove regular file `index.html'? y
 rm: cannot remove `index.html': Permission denied

As I do some testing , searching, acl checking, and lot of trying , finally I endup with debugfs command. Before start, check first in which partition is your directory or file. Use simple df –h to see all partitions. And then magic begin 😉

Please, remember, steps are quite easy but damage can be big. Via debugfs , load your desired partition in write mode. Then use standart ls command to navigate. Also be aware that when you mount partition, it become root directory , so in case of /boot – it become new root for debugfs shell.

[root@localhost /]# df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
 120G   13G  100G  12% /
 /dev/hda1              99M   12M   82M  13% /boot
 tmpfs                1014M     0 1014M   0% /dev/shm
 [root@localhost /]#
 [root@localhost /]#
 [root@localhost /]# debugfs
 debugfs 1.39 (29-May-2006)
 debugfs:  open -w /dev/hda1
 debugfs:  ls
 2  (12) .    2  (12) ..    11  (20) lost+found    18073  (12) grub
 17  (36) initrd-2.6.18-164.el5.img    16065  (12) as
 12  (36) .vmlinuz-2.6.18-164.el5.hmac    13  (36) System.map-2.6.18-164.el5
 14  (32) config-2.6.18-164.el5    15  (36) symvers-2.6.18-164.el5.gz
 16  (780) vmlinuz-2.6.18-164.el5
 debugfs:  rmdir as
debugfs:  close
 debugfs:  q

At the end, remeber run fsck and/or reboot server to be sure , everything goes fine.