Verbose Data

Data is reported in this form when either --verbose is used OR if there is at least one type of data requested that doesn't have a brief form such as any detail data or ionodes, processes or slabs. Specifying some of the lustre output options with --lustopts such as B, D and M will also force verbose format.

CPU, collectl -sc

# CPU SUMMARY (INTR, CTXSW & PROC /sec)
# USER  NICE   SYS  WAIT   IRQ  SOFT STEAL  IDLE  INTR  CTXSW  PROC  RUNQ   RUN   AVG1  AVG5 AVG15
These are the percentage of time the system in running is one of the modes, noting that these are averaged across all CPUs. While User and Sys modes are self-eplanitory, the others may not be:

User Time spent in User mode, not including time spend in "nice" mode.
Nice Time spent in Nice mode, that is lower priority as adjusted by the nice command and have the "N" status flag set when examined with "ps".
Sys This is time spent in "pure" system time.
Wait Also known as "iowait", this is the time the CPU was idle during an outstanding disk I/O request. This is not considered to be part of the total or system times reported in brief mode.
Irq Time spent processing interrupts and also considered to be part of the summary system time reported in "brief" mode.
Soft Time spent processing soft interrupts and also considered to be part of the summary system time reported in "brief" mode.
Steal Time spend in involuntary wait state while the hypervisor was servicing another virtual processor.

This next set of fields apply to processes

ProcProcess creations/sec.
RunqNumber of processes in the run queue.
RunNumber of processes in the run state.
Avg1, Avg5, Avg15Load average over the last 1,5 and 15 minutes.

Disks, collectl -sd

# DISK SUMMARY (/sec)
#KBRead RMerged  Reads SizeKB   KBWrit WMerged Writes SizeKB
KBReadKB read/sec
RMerged Read requests merged per second when being dequeued. These statistics are not available in older kernels which only record disk statistics in /proc/stat.
ReadsNumber of reads/sec
SizeKBAverage read size in KB
KBWriteKB written/sec
WMerged Write requests merged per second when being dequeued.
WritesNumber of writes/sec
SizeKBAverage write size in KB

Inodes/Filesystem, collectl -si

# INODE SUMMARY
#    Dentries      File Handles    Inodes
# Number  Unused   Alloc   % Max   Number
   40585   39442     576    0.17    38348
DCache
NumberNumber of entries in directory cache
UnusedNumber of unused entries in directory cache
HandlesNumber of allocated file handles
% MaxPercentage of maximum available file handles
InodeNumber of used inode handles
NOTE - as of this writing I'm baffled by the dentry unused field. No matter how many files and/or directories I create, this number goes up! Sholdn't it go down?

Infiniband, collectl -sx

# INFINIBAND SUMMARY (/sec)
#  KBIn   PktIn  SizeIn   KBOut  PktOut SizeOut  Errors
KBInKB received/sec.
PktInPackets received/sec.
SizeInAverage incoming packet size in KB
KBOutKB transmitted/sec.
PktOutPackets transmitted/sec.
SizeOutAverage outgoing packet size in KB
ErrsCount of current errors. Since these are typically infrequent, it is felt that reporting them as a rate would result in either not seeing them OR round-off hiding their values.

Lustre

Lustre Client, collectl -sl

There are several formats here controlled by the --lustopts switch. There is also detail data for these available as well. Specifying -sL results in data broken out by the file system and --lustopts O further breaks it out by OST. Also note the average read/write sizes are only reported when --lustopts is not specified.

# LUSTRE CLIENT SUMMARY
# KBRead  Reads SizeKB  KBWrite Writes SizeKB
KBReadKB/sec delivered to the client.
ReadsReads/sec delivered to the client, not necessarily from the lustre storage servers.
SizeKBAverage read size in KB
KBWriteKB Writes/sec delievered to the storage servers.
WritesWrites/sec delievered to the storage servers.
SizeKBAverage write size in KB
# LUSTRE CLIENT SUMMARY: METADATA
# KBRead  Reads KBWrite Writes  Open Close GAttr SAttr  Seek Fsynk DrtHit DrtMis
KBReadKB/sec delivered to the client.
ReadsReads/sec delivered to the client, not necessarily from the lustre storage servers.
KBWriteKB Writes/sec delievered to the storage servers.
WritesWrites/sec delievered to the storage servers.
OpenFile opens/sec
CloseFile closes/sec
GAttrgetattrs/sec
Seekseeks/sec
Fsyncfsyncs/sec
DrtHitdirty hits/sec
DrtMisdirty misses/sec
# LUSTRE CLIENT SUMMARY: READAHEAD
# KBRead  Reads KBWrite Writes  Pend  Hits Misses NotCon MisWin FalGrb LckFal  Discrd ZFile ZerWin RA2Eof HitMax  Wrong
KBReadKB/sec delivered to the client.
ReadsReads/sec delivered to the client, not necessarily from the lustre storage servers.
KBWriteKB Writes/sec delievered to the storage servers.
WritesWrites/sec delievered to the storage servers.
PendPending issued pages
Hitsprefetch cache hits
Missesprefetch cache misses
NotConThe current pages read that were not consecutive with the previous ones./td>
MisWinMiss inside window. The pages that were expected to be in the prefetch cache but weren't. They were probably reclaimed due to memory pressure
LckFalFailed grab_cache_pages. Tried to prefetch page but it was locked.
DiscrdRead but discarded. Prefetched pages (but not read by applicatin) have been discarded either becuase of memory pressure or lock revocation.
ZFileZero length file.
ZerWinZero size window.
RA2EofRead ahead to end of file
HitMaxHit maximum readahead issue. The read-ahead window has grown to the maximum specified by max_read_ahead_mb
# LUSTRE CLIENT SUMMARY: RPC-BUFFERS (pages)
#RdK  Rds   1K   2K   ...  WrtK Wrts   1K   2K   ...
This display shows the size of rpc buffer distribution buckets in K-pages. You can find the page size for you system in the header (collectl --showheader).

RdKKBs read/sec
RdsReads/sec
nKNumber of pages of of this size read
WrtKKBs written/sec
WrtsWrites/sec
nKNumber of pages of of this size written

Lustre Meta-Data Server, collectl -sl

As of Lustre 1.6.5, the data reported for the MDS had changed, breaking out the Reint data into 5 individual buckets which are the last 5 fields described below. For earlier versions those 5 fields will be replaced by a single one named Reint.

# LUSTRE MDS SUMMARY
#Getattr GttrLck  StatFS    Sync  Gxattr  Sxattr Connect Disconn Create   Link Setattr Rename Unlink
GetattrNumber of getattr calls, for example lfs osts. Note that this counter is not incremented as the result of ls - see Gxattr
GttrLckThese are getattrs that also return a lock on the file
StatFSNumber of stat calls, for example df or lfs df. Note that lustre caches data for up to a second so many calls within a second may only show up as a single statfs
SyncNumber of sync calls
GxattrExtended attribute get operations, for example getfattr, getfacl or even ls. Note that the MDS must have been mounted with -o acl for this counter to be enabled.
SxattrExtended attribute set operations, for example setfattr or setfacl
ConnectClient mount operations
DisconnClient umount operations
CreateCount of mknod and mkdir operations, also used by NFS servers internally when creating files
LinkHard and symbolic links, for example ln
SetattrAll operations that modify inode attributes including chmod, chown, touch, etc
RenameFile and directory renames, for example mv
UnlinkFile/directory removals, for example rm or rmdir

The following display is very similar the the RPC buffers in that the sizes of different size I/O requests are reported. In this case there are requests sent to the disk driver. Note that this report is only available for HP's SFS.

# LUSTRE DISK BLOCK LEVEL SUMMARY
#Rds  RdK 0.5K   1K   ...  Wrts WrtK 0.5K   1K   ...
RdsReads/sec
RdKKBs read/sec
nKNumber of blocks of of this size read
WrtsWrites/sec
WrtKKBs written/sec
nKNumber of blocks of of this size written

Lustre Object Storage Server, collectl -sl

# LUSTRE OST SUMMARY
# KBRead   Reads  SizeKB KBWrite  Writes  SizeKB
KBReadKB/sec read
ReadsReads/sec
SizeKBAverage read size in KB
KBWriteKB/sec written
WritesWrites/sec
SizeKBAverage write size in KB

Lustre Object Storage Server, collectl -sl --lustopts B

As with client data, when you only get read/write average sizes when --lustopt is not specified.

# LUSTRE OST SUMMARY
#<--------reads-----------|----writes-----------------
#RdK  Rds   1K   2K   ...  WrtK Wrts   1K   2K   ....
RdKKBs read/sec
RdsReads/sec
nKNumber of pages of of this size read
WrtKKBs written/sec
WrtsWrites/sec
nKNumber of pages of of this size written

Lustre Object Storage Server, collectl -sl --lustopts D

# LUSTRE DISK BLOCK LEVEL SUMMARY
#RdK  Rds 0.5K   1K   ...   WrtK Wrts 0.5K   1K   ...
RdKKBs read/sec
RdsReads/sec
nKNumber of blocks of of this size read
WrtKKBs written/sec
WrtsWrites/sec
nKNumber of blocks of of this size written

Memory, collectl -sm

# MEMORY STATISTICS
#<------------------------Physical Memory-----------------------><-----------Swap----------><-Inactive->
#   TOTAL    USED    FREE    BUFF  CACHED    SLAB  MAPPED  COMMIT     TOTAL    USED    FREE     TOTAL     IN    OUT
Total Total physical memory
Used Used physical memory. This does not include memory used by the kernel itself.
Commit Accorting to RedHat: "An estimate of how much RAM you would need to make a 99.99% guarantee that there never is OOM (out of memory) for this workload."
Swap Total Total Swap
Swap Used Used Swap
Swap Free Free Swap
Inactive Inactive pages. On ealier kernels this number is the sum of the clean, dirty and laundry pages.
Pages/Sec In Total number of pages read by block devices
Pages/Sec Out Total number of pages written by block devices

Network, collectl -sn

The entries for error counts are actually the total of several types of errors. To get individual error counts, you must report details on individual interfaces in plot format by specifying -P. Transmission errors are categorized by errors, dropped, fifo, collisions and carrier. Receive errors are broken out for errors, dropped, fifo and fragments.

# NETWORK SUMMARY (/sec)
# KBIn  PktIn SizeIn  MultI   CmpI  ErrIn  KBOut PktOut  SizeO   CmpO ErrOut
KBIn Incoming KB/sec
PktIn Incoming packets/sec
SizeI Average incoming packet size in bytes
MultI Incoming multicast packets/sec
CmpI Incoming compressed packets/sec
ErrIn Incoming errors/sec
KBOut Outgoing KB/sec
PktOut Outgoing packets/sec
SizeO Average outgoing packet size in bytes
CmpO Outgoing compressed packets/sec
ErrOut Outgoing errors/sec

NFS, collectl -sf

These statistics will be reported for V3 servers by default but you can choose a different version and/or client data via --nfsopts. They correspond to the net, rpc and protocol specific sections of the nfsstat utility.

# NFS SERVER (/sec)
#<----------Network-------><----------RPC---------><---NFS V3--->
#PKTS   UDP   TCP  TCPCONN  CALLS  BADAUTH  BADCLNT   READ  WRITE
PktsTotal network packets, which is the sum of UDP and TCP
UDPNumber of UDP packets/sec
TCPNumber of TCP packets/sec
TCPConnNumber of TCP connections/sec
CallsNumber of RPC calls/sec
BadAuthNumber of authentication failures/sec
BadClntNumber of unknown clients/sec
ReadNumber of reads/sec
WriteNumber of writes/sec

NFS, collectl -sf -nfsopts C

The data reported for clients is slightly different, specifically the retrans and authref fields.

# NFS CLIENT (/sec)
#<----------RPC---------><---NFS V3--->
#CALLS  RETRANS  AUTHREF    READ  WRITE
CallsNumber of RPC calls/sec
RetransRetransmitted calls
AuthrefAuthentication failed
ReadNumber of reads/sec
WriteNumber of writes/sec

Slabs, collectl -sy

As of the 2.6.22 kernel, there is a new slab allocator, called SLUB, and since there is not a 1:1 mapping between what it reports and the older slab allocator, the format of this listing will depend on which allocator is being used. The following format is for the older allocator.

# SLAB SUMMARY
#<------------Objects------------><--------Slab Allocation-------><--Caches--->
#  InUse   Bytes    Alloc   Bytes   InUse   Bytes   Total   Bytes  InUse  Total
Objects
InUse Total number of objects that are currently in use.
Bytes Total size of all the objects in use.
Alloc Total number of objects that have been allocated but not necessarily in use.
Bytes Total size of all the allocated objects whether in use or not.
Slab Allocation
InUse Number of slabs that have at least one active object in them.
Bytes Total size of all the slabs.
Total Total number of slabs that have been allocated whether in use or not.
Bytes Total size of all the slabs that have been allocted whether in use or not.
Caches
InUse Not all caches are actully in use. This included only those with non-zero counts.
Total This is the count of all caches, whether currently in use or not.

This is format for the new slub allocator

# SLAB SUMMARY
#<---Objects---><-Slabs-><-----memory----->
# In Use   Avail  Number      Used    Total
One should note that this report summarizes those slabs being monitored. In general this represents all slabs, but if filering is being used these numbers will only apply to those slabs that have matched the filter.

Objects
InUse The total number of objects that have been allocated to processes.
Avail The total number of objects that are available in the currently allocated slabs. This includes those that have already been allocated toprocesses.
Slabs
Number This is the number of individual slabs that have been allocated and taking physical memory.
Memory
Used Used memory corresponds to those objects that have been allocated to processes.
Total Total physical memory allocated to processes. When there is no filtering in effect, this number will be equal to the Slabs field reported by -sm.

Sockets, collectl -ss

# SOCKET STATISTICS
#      <-------------Tcp------------->   Udp   Raw   <---Frag-->
#Used  Inuse Orphan    Tw  Alloc   Mem  Inuse Inuse  Inuse   Mem
UsedTotal number if socket allocated which can include additional types such as domain.
Tcp
InuseNumber of TCP connections in use
OrphanNumber of TCP orphaned connections
TwNumber of connections in TIME_WAIT
AllocTCP sockets allocated
Mem
Udp
InuseNumber of UCP connections in use
Raw
InuseNumber of RAW connections in use
Frag
Inuse
Mem

TCP, collectl -st

# TCP SUMMARY (/sec)
# PureAcks HPAcks   Loss FTrans
PureAcksACKs/sec that only contain acks (ie no data).
HPAcksFast-path acks/sec.
LossPackets/sec TCP thinks have been lost coming in.
FTransFast retransmissions/sec.