I have a Solaris 10 server which is reporting that a file system is filling up:
# df -h /opt/CSCObac/var
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL
# df -F vxfs /opt/CSCObac/var/
/opt/CSCObac/var (/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL): 4469796 blocks 558723 files
# df -h | fgrep '/opt/'
/dev/vx/dsk/RDU-APP-DG/RDU-APP-VOL 68G 547M 63G 1% /opt/CSCObac
/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL 134G 132G 2.0G 99% /opt/CSCObac/var
/dev/vx/dsk/RDU-LOG-DG/RDU-LOG-VOL 134G 5.2G 121G 5% /opt/CSCObac/var/rdu
Except there are no files on the filesystem which occupy 100G+, df reports it's not out of free blocks, and there's no lack of free inodes either:
# du -sh /opt/CSCObac/var
5.1G /opt/CSCObac/var
134G 132G 2.0G 99% /opt/CSCObac/var
# df -g /opt/CSCObac/var/
/opt/CSCObac/var (/dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL): 8192 block size 1024 frag size
280938496 total blocks 4469536 free blocks 4190192 available 558720 total files
558691 free files 74450896 filesys id
vxfs fstype 0x00000004 flag 255 filename length
Also, no open file descriptors:
# lsof /opt/CSCObac/var/
#
This is a vxfs filesystem, see below for volume and size configurations:
# fstyp -v /dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL
vxfs
magic a501fcf5 version 7 ctime November 15, 2013 2:15:55 PM CET
logstart 0 logend 0
bsize 1024 size 140469248 dsize 0 ninode 140469248 nau 0
defiextsize 0 ilbsize 0 immedlen 96 ndaddr 10
aufirst 0 emap 0 imap 0 iextop 0 istart 0
bstart 0 femap 0 fimap 0 fiextop 0 fistart 0 fbstart 0
nindir 2048 aulen 32768 auimlen 0 auemlen 8
auilen 0 aupad 0 aublocks 32768 maxtier 15
inopb 4 inopau 0 ndiripau 0 iaddrlen 8 bshift 10
inoshift 2 bmask fffffc00 boffmask 3ff checksum f790321b
oltext1 32 oltext2 1282 oltsize 1 checksum2 0
free 2234899 ifree 0
efree 5 9 7 2 3 5 4 4 2 0 3 1 2 3 2 0 1 2 1 1 1 0 0 0 0 0 0 0 0 0 0 0
# vxlist disk | grep 'VAR'
disk disk_4 RDU-VAR-DG01 RDU-VAR-DG 133.96g 608.00k imported
disk disk_8 RDU-VAR-DG02 RDU-VAR-DG 133.96g 608.00k imported
# vxlist vol RDU-VAR-VOL
Volume: RDU-VAR-VOL
GUID: {0de33acc-1dd2-11b2-a8f9-00212800c1e0}
Disk Group: name=RDU-VAR-DG id=1384520670.51.rdu2a
Host: rdu2a
Size: 133.9g
Type: fsgen
Status: healthy
Layout: concat-mirror
NMirrors: 2
Read Policy: SELECT
Flags: open writeback
Device Path: /dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL
Permissions: user=root group=root mode=0600
filesystem
TY FSTYPE %USED VSET MOUNT-POINT
fs vxfs 98% - /opt/CSCObac/var
stats
TY OPERATION COUNT BLOCKS TIME(MSEC)
stat read 1160490 1731963 17538.5
stat write 3928639 294438233 7579.4
plexes
TY NAME TYPE STATUS
plex RDU-VAR-VOL-03 simple attached
# vxlist fs | fgrep 'VAR'
fs var vxfs 133.96g 2046.10m 99% /dev/vx/dsk/RDU-VAR-DG/RDU-VAR-VOL /opt/CSCObac/var
# vxassist -g RDU-VAR-DG -p maxgrow RDU-VAR-VOL
VxVM vxassist ERROR V-5-1-1178 Volume RDU-VAR-VOL cannot be extend within the given constraints
# fsadm -D /opt/CSCObac/var/
Directory Fragmentation Report
Dirs Total Immed Immeds Dirs to Blocks to
Searched Blocks Dirs to Add Reduce Reduce
total 11 3 8 0 0 0
What could be filling up my filesystem beyond inodes, blocks and file descriptors?
After some lenghty troubleshooting with our support vendor, we found the solution. There was a process running that was started before the mounting of the vxfs filesystems. This process was writing data to /opt/CSCObac/var/rdu which is a nested directory of the /opt/CSCObac/var/ mountpoint. After killing this process, the disk usage was back to the expected 1% The OS could not report back on contents of this directory as the rdu mountpoint obscures it. The long running process started writing to a file in this directory mountpoint before the filesystem was mounted, which explains why it was increasing in size without seeing an increase in the visible files within the directory.
User contributions licensed under CC BY-SA 3.0