Nilesh Ghogre

Net-SNMP

Nilesh Ghogre — Thu, 26 Nov 2009 12:42:11 +0000

Net-SNMP is provided in Solaris 10 release now, and how to use it is descripted in this article.

To disable Sun’s SNMP
#svcadm disable svc:/application/management/snmpdx:default

the Net-SNMP daemon program locates at /usr/sfw/sbin/snmpd, and configure file of snmpd saved with the name snmpd.conf at /etc/sma/snmp. Use svcadm to start of stop snmpd:
#svcadm enable svc:/application/management/sma:default
#svcadm disable svc:/application/management/sma:default

Log File
if there’s any error or warning messages, see the file /var/svc/log/application-management-sma\:default.log. The default log file of snmpd is /var/log/snmpd.log.

Configuration file
We may use /etc/sma/snmp/snmpd.conf.save as the working configure file to test wheather our snmpd daemon is ok.
#cp /etc/sma/snmp/snmpd.conf.save /etc/sma/snmp/snmpd.conf
#svcadm restart sma

P.S. There’s another file also named as snmpd.conf stored in /var/sma_snmp directory, and which is called persistent storage file. The persistent storage file, /var/sma_snmp/snmpd.conf, contains USM security information and any MIB components that are set for persistent storage. This file also contains the engineID and the engineID boots. This persistent storage file is automatically updated when the System Management Agent starts. When the System Management Agent stops, the snmpusm and snmpvacm utilities write user security information to this storage file.

Using the AgentX Protocol
The AgentX protocol is supported in the System Management Agent. By default, the System Management Agent ships with a secure profile, that is, read–only access. AgentX allows interaction with third party subagents, provided that these subagents support AgentX over Unix Domain Sockets. For security reasons, AgentX is not supported over TCP/UDP. For more information on the AgentX protocol, see http://www.ietf.org/rfc/rfc2741.txt Edit the main /etc/sma/snmp/snmpd.conf configuration file to configure the System Management Agent to use the AgentX protocol. By default, the AgentX protocol is disabled. The following procedure describes how to enable the AgentX protocol. To enable the AgentX protocol add the following to /etc/sma/snmp/snmpd.conf file:

master agentx

restart the sma (system management agent).
# svcadm restart svc:/application/management/sma:default

Using the SMA
To view the status of the agent
# svcs svc:/application/management/sma:default

To See Which MIBs Are Initialized
# /usr/sfw/sbin/snmpd -Dregister_mib -Dmib_init -L

To Check the Disk Space and Network on a Local or Remote Machine
# /usr/sfw/bin/snmpwalk -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageIndex

# /usr/sfw/bin/snmpget -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageSize.1

# /usr/sfw/bin/snmpwalk -v1 -c public localhost HOST-RESOURCES-MIB::hrStorageUsed

# /usr/sfw/bin/snmpnetstat -v 2c -c public -a localhost

if all these works, congratulations, the sma is configured and brought on properly.

Logrotate

Nilesh Ghogre — Tue, 24 Nov 2009 12:56:12 +0000

Logrotate is a Linux utility that renames and reuses system error log files on a periodic basis so that they don’t occupy excessive disk space.

The /etc/logrotate.conf File

This is logrotate’s general configuration file in which you can specify the frequency with which the files are reused.

o You can specify either “weekly” or “daily” rotation parameter. In the case below the weekly option is “commented out” with a “#”, allowing for daily updates.

o The “rotate” parameter specifies the number of copies of log files logrotate will maintain. In the case below the 4 copy option is “commented out” with a “#”, while allowing 7 copies.

o The “create” parameter creates a new log file after each rotation

Therefore our sample configuration file will create daily archives of ALL the logfiles and store them for seven days. The files will have the following names with “logfile” being current active version:

logfile

logfile.0

logfile.1

logfile.2

logfile.3

logfile.4

logfile.5

logfile.6

Sample contents of /etc/logrotate.conf

# rotate log files weekly
#weekly

# rotate log files daily
daily

# keep 4 weeks worth of backlogs
#rotate 4

# keep 7 days worth of backlogs
rotate 7

# create new (empty) log files after rotating old ones
create

The /etc/logrotate.d Directory

Most Linux applications that use syslog will put an additional configuration file in this directory to specify the names of the log files to be rotated. It is a good practice to verify that all new applications that you want to use the syslog log have configuration files in this directory. Here are some sample files which define the specific files to be rotated for each application.

The /etc/logrotate.d/syslog File (For General System Logging)

/var/log/cisco/* /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {

sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript

}

The /etc/logrotate.d/apache File (For Apache)

/var/log/httpd/access_log /var/log/httpd/agent_log /var/log/httpd/error_log /var/log/httpd/referer_log {

missingok
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
endscript

}

The /etc/logrotate.d/samba File (for SAMBA)

/var/log/samba/*.log {

notifempty
missingok
sharedscripts
copytruncate
postrotate
/bin/kill -HUP `cat /var/lock/samba/*.pid 2> /dev/null` 2> /dev/null || true
endscript

}

Activating logrotate

The above logrotate settings will not take effect until you issue the following command to do so:

[root@bigboy tmp]# logrotate –f

If you want logrotate to reload only a specifc configuration file, and not all of them, then issue the logrotate command with just that filename as the argument like this:

[root@bigboy tmp]# logrotate -f /etc/logrotate.d/syslog

Solaris Commands

Nilesh Ghogre — Mon, 23 Nov 2009 12:26:28 +0000

apptrace – for tracing library calls
dtrace – debugger, new in version 10
pargs – get list of arguments and environment variables with which process was started
pfiles – list of file descriptors, associated with process
pgrep – get the PID’s of processes by name i.e. Something like ps -efl|grep -v grep|grep process_name
pkill – send signal to process. For example pkill -9 init :-P
pldd – list dynamic libraries, associated with process, similar to ldd for executable
plockstat – see list of locked by process files. Lock can be mutex i.e. exclusive and reader/writer for shared access
pmap – get memory map (segments) of process
preap – try to kick-off zombie process
prstat – fullscreen view of processes sorted by different criteria, similar to Linux top command
prun – continue hold with pstop process
ps – print process information and status. In Solaris exist SYSV and BSD variants, respectively /usr/bin/ps and /usr/ucb/ps
psig – list signals that can be handled by process
pstack – get backtrace stack of process for debugging purposes
pstop – temporary hold process
ptree – print the tree of processes
pwait – wait till process finish
pwdx – list working directory for process, like pwd command
truss – for tracing user/library/system calls and signals

Examples

apptrace

# apptrace man
-> man      -> libc.so.1:int atexit(int (*)() = 0xd27d6bfc)
<- man      -> libc.so.1:atexit()
-> man      -> libc.so.1:int atexit(int (*)() = 0x8054e8c)
<- man      -> libc.so.1:atexit()
-> man      -> libc.so.1:void __fpstart(void)
<- man      -> libc.so.1:__fpstart() = 0xd251fc94
-> man      -> libc.so.1:access(0x8055984, 0x1, 0x8047e30) ** NR
-> man      -> libc.so.1:char * setlocale(int = 0x6, const char * = 0x8055980 "")
<- man      -> libc.so.1:setlocale() = 0xd25002ed
-> man      -> libc.so.1:char * setlocale(int = 0x5, const char * = 0x0 )
<- man      -> libc.so.1:setlocale() = 0xd25002ed
-> man      -> libc.so.1:strcpy(0x80669e8, 0xd25002ed, 0x8047e30) ** NR
-> man      -> libc.so.1:int strcmp(const char * = 0x805597c "C", const char * = 0x80669e8 "C")
<- man      -> libc.so.1:strcmp()
-> man      -> libc.so.1:char * textdomain(const char * = 0x8055968 "SUNW_OST_OSCMD")
<- man      -> libc.so.1:textdomain() = 0x80683e0
-> man      -> libc.so.1:strcpy(0x80665e8, 0x8055960, 0x8047e30) ** NR
-> man      -> libc.so.1:char * getenv(const char * = 0x8054fc0 "MANPATH")
<- man      -> libc.so.1:getenv()
-> man      -> libc.so.1:strrchr(0x8047e98, 0x2f, 0x8047e30) ** NR
-> man      -> libc.so.1:int strcmp(const char * = 0x8047e98 "man", const char * = 0x8054fb8 "apropos")
<- man      -> libc.so.1:strcmp() = 0xc
-> man      -> libc.so.1:int strcmp(const char * = 0x8047e98 "man", const char * = 0x8055958 "whatis")
<- man      -> libc.so.1:strcmp() = 0xfffffff6
-> man      -> libc.so.1:int strcmp(const char * = 0x8047e98 "man", const char * = 0x8055944 "catman")
<- man      -> libc.so.1:strcmp() = 0xa
-> man      -> libc.so.1:int getopt(int = 0x1, char *const * = 0x8047de4, const char * = 0x8066420 "FfkrP:M:T:ts:lad")
<- man      -> libc.so.1:getopt() = 0xffffffff
-> man      -> libc.so.1:char * gettext(const char * = 0x8055838 "usage:        man [-] [-adFlrt] [-M path] [-T macro-package ] [ -s  section ] name ...
        man [-M path] -k keyword ...
        man [-M path] -f file ...")
<- man      -> libc.so.1:gettext() = 0x8055838
-> man      -> libc.so.1:int fprintf(FILE * = 0x8067610, const char * = 0x8054f7c "%s
", void * = 0x8055838, ...)
usage:  man [-] [-adFlrt] [-M path] [-T macro-package ] [ -s section ] name ...
        man [-M path] -k keyword ...
        man [-M path] -f file ...
<- man      -> libc.so.1:fprintf() = 0x88
-> man      -> libc.so.1:exit(0x2, 0x8047e30, 0x8047da0) ** NR

in the above example you can clearly see library calls as fprintf(), strcmp(), getenv(), etc.

pargs

# pargs 558
558:    /usr/lib/sendmail -Ac -q15m
argv[0]: sendmail: Queue runner@00:15:00 for /var/spool/clientmqueue

argv[1]: 
argv[2]: @00:15:00 for /var/spool/clientmqueue

in above example you see the arguments for starting sendmail as a daemon

pfiles

# pfiles 558
558:    /usr/lib/sendmail -Ac -q15m
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_RDONLY
      /devices/pseudo/mm@0:null
   1: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY
      /devices/pseudo/mm@0:null
   2: S_IFCHR mode:0666 dev:270,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_WRONLY
      /devices/pseudo/mm@0:null
   3: S_IFDOOR mode:0444 dev:279,0 ino:53 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[366]
      /var/run/name_service_door
   4: S_IFCHR mode:0666 dev:270,0 ino:11010052 uid:0 gid:3 rdev:21,0
      O_WRONLY FD_CLOEXEC
      /devices/pseudo/log@0:conslog
   5: S_IFREG mode:0600 dev:102,0 ino:189384 uid:25 gid:25 size:32
      O_WRONLY|O_CREAT|O_EXCL
      /var/spool/clientmqueue/sm-client.pid

Here you see a list of files as NULL, console log, some PID files. On the top you can see the maximum number of file handlers for this process, with info that can be helpful for troubleshooting of applications

pgrep

# pgrep named
586

Something like ps -efl|grep -v grep|grep process_name|awk ‘{print $1}’

# ps -e|grep -v grep|grep named|awk '{print $1}'
586

pkill

# pgrep named
628
# pkill named
# pgrep named
636
#

As you see, the named daemon was killed and respawned (with new PID)

pldd

# pldd 636
636:    /usr/sbin/named
/usr/lib/dns/liblwres.so.1.1.2
/usr/lib/dns/libdns.so.16.0.0
/usr/lib/dns/libisccfg.so.0.0.11
/usr/sfw/lib/libcrypto.so.0.9.7
/usr/lib/dns/libisccc.so.0.1.0
/usr/lib/dns/libisc.so.7.1.5
/lib/libnsl.so.1
/lib/libsocket.so.1
/lib/libscf.so.1
/lib/libpthread.so.1
/lib/libthread.so.1
/lib/libc.so.1
/lib/libdoor.so.1
/lib/libuutil.so.1
# ldd /usr/sbin/named
        liblwres.so.1 =>         /usr/lib/dns/liblwres.so.1
        libdns.so.16 =>  /usr/lib/dns/libdns.so.16
        libisccfg.so.0 =>        /usr/lib/dns/libisccfg.so.0
        libcrypto.so.0.9.7 =>    /usr/sfw/lib/libcrypto.so.0.9.7
        libisccc.so.0 =>         /usr/lib/dns/libisccc.so.0
        libisc.so.7 =>   /usr/lib/dns/libisc.so.7
        libnsl.so.1 =>   /lib/libnsl.so.1
        libsocket.so.1 =>        /lib/libsocket.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libpthread.so.1 =>       /lib/libpthread.so.1
        libthread.so.1 =>        /lib/libthread.so.1
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd5.so.1 =>   /lib/libmd5.so.1
        libdoor.so.1 =>  /lib/libdoor.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libm.so.2 =>     /lib/libm.so.2

Results from both commands are almost the same :-) Why? Ask SUN

pmap

# pmap 636
636:    /usr/sbin/named
08046000       8K rw----    [ stack ]
08050000     232K r-x--  /usr/sbin/named
08099000      16K rwx--  /usr/sbin/named
0809D000     544K rwx--    [ heap ]
D21ED000       4K rw--R    [ stack tid=4 ]
D21F0000      64K rwx--    [ anon ]
D2218000       4K rw--R    [ stack tid=3 ]
D221B000       4K rwxs-    [ anon ]
D2236000      16K rw--R    [ stack tid=2 ]
D223C000      64K rw----    [ anon ]
D224E000      64K rw----    [ anon ]
D2260000      24K rwx--    [ anon ]
D2270000       4K rwx--    [ anon ]
D2280000      24K r-x--  /lib/libuutil.so.1
D2296000       4K rw----  /lib/libuutil.so.1
D22A0000       4K r-x--  /lib/libdoor.so.1
D22B1000       4K rw----  /lib/libdoor.so.1
D22C0000       4K rwx--    [ anon ]
D22D0000     740K r-x--  /lib/libc.so.1
D2399000      24K rw----  /lib/libc.so.1
D239F000       8K rw----  /lib/libc.so.1
D23B0000      16K r-x--  /lib/libthread.so.1
D23C0000      12K r-x--  /lib/libpthread.so.1
D23D0000      88K r-x--  /lib/libscf.so.1
D23F6000       4K rw----  /lib/libscf.so.1
D2400000       4K rwx--    [ anon ]
D2410000      44K r-x--  /lib/libsocket.so.1
D242B000       4K rw----  /lib/libsocket.so.1
D2430000     512K r-x--  /lib/libnsl.so.1
D24B0000      20K rw----  /lib/libnsl.so.1
D24B5000      32K rw----  /lib/libnsl.so.1
D24C0000     228K r-x--  /usr/lib/dns/libisc.so.7.1.5
D2508000       8K rwx--  /usr/lib/dns/libisc.so.7.1.5
D2510000      24K r-x--  /usr/lib/dns/libisccc.so.0.1.0
D2525000       4K rwx--  /usr/lib/dns/libisccc.so.0.1.0
D2530000       4K rwx--    [ anon ]
D2540000     924K r-x--  /usr/sfw/lib/libcrypto.so.0.9.7
D2637000      84K rw----  /usr/sfw/lib/libcrypto.so.0.9.7
D264C000       8K rw----  /usr/sfw/lib/libcrypto.so.0.9.7
D2650000      48K r-x--  /usr/lib/dns/libisccfg.so.0.0.11
D266B000      12K rwx--  /usr/lib/dns/libisccfg.so.0.0.11
D2670000    1024K r-x--  /usr/lib/dns/libdns.so.16.0.0
D277F000      20K rwx--  /usr/lib/dns/libdns.so.16.0.0
D2790000       4K rwx--    [ anon ]
D27A0000      56K r-x--  /usr/lib/dns/liblwres.so.1.1.2
D27BD000       4K rwx--  /usr/lib/dns/liblwres.so.1.1.2
D27CA000     132K r-x--  /lib/ld.so.1
D27FB000       4K rwx--  /lib/ld.so.1
D27FC000       8K rwx--  /lib/ld.so.1
 total      5192K

This command helps in the process of fine-tuning large applications, especially for resolving emory leak issues

preap

# ps -efl|grep 551
 0 Z     root   551   550   0   0  -        -      0        -        - ?           0:00 
# preap 551
551: exited with status 0
# ps -efl|grep 551

In above example, this tool successfully removes zombie process. Otherwise the only way to kill them is to reboot

prstat

# prstat
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
   744 root     4376K 4080K cpu0    39    0   0:00:00 0.4% prstat/1
   560 root     6988K 1944K sleep   59    0   0:00:01 0.1% sshd/1
   468 root       31M   13M sleep   59    0   0:00:04 0.1% Xorg/1
   550 root     9576K 6856K sleep   59    0   0:00:01 0.0% dtgreet/1
   577 root     1160K  892K sleep   59    0   0:00:00 0.0% ksh/1
   366 root     3476K 2472K sleep   59    0   0:00:00 0.0% nscd/27
   699 root     6984K 1948K sleep   59    0   0:00:00 0.0% sshd/1
   506 root     6672K 4776K sleep   59    0   0:00:00 0.0% snmpd/1
   476 root     2724K 1552K sleep   59    0   0:00:00 0.0% dmispd/1
   481 root     3268K 1964K sleep   59    0   0:00:00 0.0% snmpXdmid/2
   532 root     2492K 1928K sleep   59    0   0:00:00 0.0% vold/5
   538 root     2352K 1564K sleep   59    0   0:00:00 0.0% devfsadm/6
   457 root     4448K 1668K sleep   59    0   0:00:00 0.0% dtlogin/1
   458 root     2040K 1240K sleep   59    0   0:00:00 0.0% snmpdx/1
   466 root     1468K  724K sleep   59    0   0:00:00 0.0% fbconsole/1
   413 root     1700K  700K sleep   59    0   0:00:00 0.0% smcboot/1
   562 root     1076K  824K sleep   59    0   0:00:00 0.0% sh/1
   412 root     1700K  700K sleep   59    0   0:00:00 0.0% smcboot/1
   356 root     3336K 1176K sleep   59    0   0:00:00 0.0% sshd/1
   533 root     7480K 2860K sleep   59    0   0:00:00 0.0% dtlogin/1
   411 root     1704K 1032K sleep   59    0   0:00:00 0.0% smcboot/1
   263 root     1064K  620K sleep   59    0   0:00:00 0.0% utmpd/1
   359 root     4400K 2212K sleep   59    0   0:00:00 0.0% syslogd/16
   261 root     3936K 3004K sleep   59    0   0:00:02 0.0% inetd/4
   264 root     1888K 1084K sleep   59    0   0:00:00 0.0% ttymon/1
   241 daemon   2340K 1520K sleep   59    0   0:00:00 0.0% statd/1
   122 root     1328K  904K sleep   59    0   0:00:00 0.0% powerd/3
   202 root     2228K  884K sleep   59    0   0:00:00 0.0% cron/1
    96 root     2128K 1288K sleep   59    0   0:00:00 0.0% syseventd/14
   554 root     7200K 3664K sleep   59    0   0:00:00 0.0% sshd/1
   361 root     9272K 5640K sleep   59    0   0:00:00 0.0% fmd/14
   244 daemon   2072K 1312K sleep   60  -20   0:00:00 0.0% lockd/2
    55 root     1952K  924K sleep   59    0   0:00:00 0.0% dhcpagent/1
   253 root     1700K  892K sleep   59    0   0:00:00 0.0% sac/1
   232 daemon   2288K 1012K sleep   59    0   0:00:00 0.0% rpcbind/1
   123 daemon   3908K 2016K sleep   59    0   0:00:00 0.0% kcfd/3
   117 root     2468K 1544K sleep   59    0   0:00:00 0.0% picld/4
   339 root     4252K 1592K sleep   59    0   0:00:00 0.0% automountd/3
   175 root     1852K  952K sleep   59    0   0:00:00 0.0% in.routed/1
     9 root     9000K 8068K sleep   59    0   0:00:11 0.0% svc.configd/18
     7 root     8888K 7940K sleep   59    0   0:00:05 0.0% svc.startd/13
   260 root     1948K 1168K sleep   59    0   0:00:00 0.0% ttymon/1

The program in default view is less informative that top, but can give you info for zones, projects and even for processes bind to specific processor. Use the -u option to limit display to specific users. Use the -p option to limit display to specific processes

prun – see example of pstop

# /usr/bin/ps
   PID TTY         TIME CMD
   562 pts/2       0:00 sh
   577 pts/2       0:00 ksh
   757 pts/2       0:00 ps
# /usr/ucb/ps
   PID TT       S  TIME COMMAND
   260 console  S  0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -T sun-color -m ldterm,ttcompat -h -p unknown console lo
   562 pts/2    S  0:00 -sh
   577 pts/2    S  0:00 ksh -o ksh
   759 pts/2    O  0:00 /usr/ucb/ps
   701 pts/3    S  0:00 -sh

psig

# psig 636
636:    /usr/sbin/named
HUP     default
INT     default
QUIT    default
ILL     default
TRAP    default
ABRT    default
EMT     default
FPE     default
KILL    default
BUS     default
SEGV    default
SYS     default
PIPE    ignored
ALRM    default
TERM    default
USR1    default
USR2    default
CLD     default                 NOCLDSTOP
PWR     default
WINCH   default
URG     default
POLL    default
STOP    default
TSTP    ignored
CONT    default
TTIN    ignored
TTOU    ignored
VTALRM  default
PROF    default
XCPU    default
XFSZ    ignored
WAITING default
LWP     default
FREEZE  default
THAW    default
CANCEL  default
LOST    default
XRES    default
JVM1    default
JVM2    default
RTMIN   default
RTMIN+1 default
RTMIN+2 default
RTMIN+3 default
RTMAX-3 default
RTMAX-2 default
RTMAX-1 default
RTMAX   default

pstack

# pstack 636
636:    /usr/sbin/named
----------------------  lwp# 1 / thread# 1  --------------------------
 d23700a5 sigtimedwait (8047ce0, 8047cf0, 0)
 d23655a6 sigwait  (8047dc0) + 1c
 d235e206 __posix_sigwait (8047dc0, 8047e60) + 2e
 d24db3dd isc_app_run (8047ec8, 8047e60, d27fb840, 0, 8047e98, 805c33e) + 158
 08066da7 main     (1, 8047ea4, 8047eac) + fe
 0805c33e ???????? (1, 8047f30, 0, 8047f40, 8047f47, 8047f5f)
----------------------  lwp# 2 / thread# 2  --------------------------
 d236fc99 lwp_park (0, 0, 0)
 d236a501 cond_wait_queue (80a2a7c, 80a2a48, 0, 0) + 3b
 d236a9fa _cond_wait (80a2a7c, 80a2a48) + 66
 d236aa3c cond_wait (80a2a7c, 80a2a48) + 21
 d236aa75 pthread_cond_wait (80a2a7c, 80a2a48) + 1b
 d24d871b dispatch (80a2a40, d2239fec, d236f93e, 80a2a40, 0, 0) + a0
 d24d8ba2 run      (80a2a40) + c
 d236f93e _thr_setup (d2262400) + 4e
 d236fc20 _lwp_start (d2262400, 0, 0, d2239ff8, d236fc20, d2262400)
----------------------  lwp# 3 / thread# 3  --------------------------
 d236fc99 lwp_park (0, d2218ec8, 0)
 d236a501 cond_wait_queue (80a2ae8, 80a2ab8, d2218ec8, 0) + 3b
 d236a8a1 cond_wait_common (80a2ae8, 80a2ab8, d2218ec8) + 1df
 d236aad5 _cond_timedwait (80a2ae8, 80a2ab8, d2218f28) + 51
 d236ab40 cond_timedwait (80a2ae8, 80a2ab8, d2218f28) + 24
 d236ab7c pthread_cond_timedwait (80a2ae8, 80a2ab8, d2218f28) + 1e
 d24e7601 isc_condition_waituntil (80a2ae8, 80a2ab8, 80a2ae0) + 81
 d24da800 run      (80a2ab0) + a2
 d236f93e _thr_setup (d21f0000) + 4e
 d236fc20 _lwp_start (d21f0000, 0, 0, d2218ff8, d236fc20, d21f0000)
----------------------  lwp# 4 / thread# 4  --------------------------
 d2370cd5 pollsys  (d21edce0, 6, 0, 0)
 d231ecca pselect  (1c, d21ede40, d21edec0, d239a868, 0, 0) + 18e
 d231efad select   (1c, d21ede40, d21edec0, 0, 0) + 6f
 d24e355c watcher  (80a5bd0) + bd
 d236f93e _thr_setup (d21f0400) + 4e
 d236fc20 _lwp_start (d21f0400, 0, 0, d21edff8, d236fc20, d21f0400)

This command can sometimes be very helpful for debugging daemons. You can see the function call stack by thread, in real-time

pstop

# ps -efl|grep ssh
 0 S     root   554   356   0  40 20 d419b870   1800 d4c87e72 21:03:52 ?           0:00 /usr/lib/ssh/sshd
 0 S     root   356     1   0  40 20 d4195320    834 d49f3432 21:02:46 ?           0:00 /usr/lib/ssh/sshd
 0 S     root   560   554   0  40 20 d41931a0   1747 d4c87ab2 21:05:03 ?           0:02 /usr/lib/ssh/sshd
 0 S     root   812   809   0  40 20 d418a270   1746 d50402f2 22:09:00 ?           0:00 /usr/lib/ssh/sshd
 0 S     root   809   356   0  40 20 d4189a10   1801 d4c874f2 22:08:47 ?           0:00 /usr/lib/ssh/sshd
# pstop 356
# ssh localhost
(here I stop with ctrl-C ssh client after 5 minutes of waiting)
# prun 356
# ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is da:14:db:bc:2f:e0:0f:ef:f6:3f:12:0e:f4:77:0c:00.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password:
Last login: Wed Jun 13 22:09:00 2007 from gateway
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
#

Command is interesting, but I don’t see much benefit for daily tasks

ptree

# ptree
7     /lib/svc/bin/svc.startd
  253   /usr/lib/saf/sac -t 300
    264   /usr/lib/saf/ttymon
  260   /usr/lib/saf/ttymon -g -d /dev/console -l console -T sun-color -m ldterm,ttcomp
9     /lib/svc/bin/svc.configd
55    /sbin/dhcpagent
96    /usr/lib/sysevent/syseventd
117   /usr/lib/picl/picld
122   /usr/lib/power/powerd
123   /usr/lib/crypto/kcfd
175   /usr/sbin/in.routed
202   /usr/sbin/cron
232   /usr/sbin/rpcbind
241   /usr/lib/nfs/statd
244   /usr/lib/nfs/lockd
261   /usr/lib/inet/inetd start
263   /usr/lib/utmpd
339   /usr/lib/autofs/automountd
356   /usr/lib/ssh/sshd
  554   /usr/lib/ssh/sshd
    560   /usr/lib/ssh/sshd
      562   -sh
        577   ksh -o vi
          771   ptree
  696   /usr/lib/ssh/sshd
    699   /usr/lib/ssh/sshd
      701   -sh
359   /usr/sbin/syslogd
361   /usr/lib/fm/fmd/fmd
366   /usr/sbin/nscd
411   /usr/sadm/lib/smc/bin/smcboot
  412   /usr/sadm/lib/smc/bin/smcboot
  413   /usr/sadm/lib/smc/bin/smcboot
457   /usr/dt/bin/dtlogin -daemon
  466   /usr/openwin/bin/fbconsole -n -d :0
  468   /usr/X11/bin/Xorg :0 -depth 24 -nobanner -auth /var/dt/A:0-Zgai5a
  533   /usr/dt/bin/dtlogin -daemon
    550   dtgreet -display :0
458   /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
476   /usr/lib/dmi/dmispd
481   /usr/lib/dmi/snmpXdmid -s unknown
506   /usr/sfw/sbin/snmpd
532   /usr/sbin/vold -f /etc/vold.conf
538   devfsadmd
636   /usr/sbin/named

No need for comments :-)

pwait

# ps -efl|grep name
 0 S     root   875     1   0  45 20 d4188950   1298 d553ab16 22:17:58 ?           0:00 /usr/sbin/named
# pwait 875
(in this moment from another terminal I execute svcadm disable dns/server)
#

Maybe this command will be great in some specific scripts

pwdx

# pwdx 636
636:    /var/named

Can be helpful in case of strange errors when unmount some filesystem

truss

# truss ls
execve("/usr/bin/ls", 0x08047E40, 0x08047E48)  argc = 1
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
resolvepath("/usr/bin/ls", "/usr/bin/ls", 1023) = 11
sysconfig(_CONFIG_PAGESIZE)                     = 4096
xstat(2, "/usr/bin/ls", 0x08047C38)             = 0
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
xstat(2, "/lib/libsec.so.1", 0x080474D0)        = 0
resolvepath("/lib/libsec.so.1", "/lib/libsec.so.1", 1023) = 16
open("/lib/libsec.so.1", O_RDONLY)              = 3
mmap(0x00010000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_ALIGN, 3, 0) = 0xD27C0000
mmap(0x00010000, 139264, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xD2790000
mmap(0xD2790000, 49071, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_TEXT, 3, 0) = 0xD2790000
mmap(0xD27AC000, 12169, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_INITDATA, 3, 49152) = 0xD27AC000
mmap(0xD27AF000, 8536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xD27AF000
munmap(0xD279C000, 65536)                       = 0
memcntl(0xD2790000, 8776, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)                                        = 0

mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xD2780000
xstat(2, "/lib/libc.so.1", 0x080474D0)          = 0
resolvepath("/lib/libc.so.1", "/lib/libc.so.1", 1023) = 14
open("/lib/libc.so.1", O_RDONLY)                = 3
mmap(0xD27C0000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xD27C0000
mmap(0x00010000, 856064, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xD26A0000
mmap(0xD26A0000, 754581, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_TEXT, 3, 0) = 0xD26A0000
mmap(0xD2769000, 24239, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_INITDATA, 3, 757760) = 0xD2769000
mmap(0xD276F000, 5656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xD276F000
munmap(0xD2759000, 65536)                       = 0
memcntl(0xD26A0000, 120300, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)                                        = 0
xstat(2, "/lib/libavl.so.1", 0x080474D0)        = 0
resolvepath("/lib/libavl.so.1", "/lib/libavl.so.1", 1023) = 16
open("/lib/libavl.so.1", O_RDONLY)              = 3
mmap(0xD27C0000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xD27C0000
mmap(0x00010000, 73728, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xD2680000
mmap(0xD2680000, 2788, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_TEXT, 3, 0) = 0xD2680000
mmap(0xD2691000, 204, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_INITDATA, 3, 4096) = 0xD2691000
munmap(0xD2681000, 65536)                       = 0
memcntl(0xD2680000, 1056, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)                                        = 0
munmap(0xD27C0000, 4096)                        = 0
mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xD27C0000
getcontext(0x080479D0)
getrlimit(RLIMIT_STACK, 0x080479C8)             = 0
getpid()                                        = 786 [785]
lwp_private(0, 1, 0xD27C2000)                   = 0x000001C3
setustack(0xD27C2060)
sysi86(SI86FPSTART, 0xD276FCD8, 0x0000133F, 0x00001F80) = 0x00000001
brk(0x080651E8)                                 = 0
brk(0x080671E8)                                 = 0
time()                                          = 1181761288
ioctl(1, TCGETA, 0x08047B9C)                    = 0
ioctl(1, TIOCGWINSZ, 0x08064460)                = 0
brk(0x080671E8)                                 = 0
brk(0x080711E8)                                 = 0
lstat64(".", 0x08046A70)                        = 0
openat(-3041965, ".", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3
fcntl(3, F_SETFD, 0x00000001)                   = 0
fstat64(3, 0x08047AC0)                          = 0
getdents64(3, 0xD27C4000, 8192)                 = 992
getdents64(3, 0xD27C4000, 8192)                 = 0
close(3)                                        = 0
ioctl(1, TCGETA, 0x08045E44)                    = 0
fstat64(1, 0x08045DB0)                          = 0
TT_DB       cdrom       etc         kernel      mnt         platform    system      var
write(1, " T T _ D B              ".., 88)      = 88
bin         dev         export      lib         net         proc        tmp         vol
write(1, " b i n                  ".., 88)      = 88
boot        devices     home        lost+found  opt         sbin        usr
write(1, " b o o t                ".., 76)      = 76
_exit(0)

Shows you system calls made by a process along with parameters of the function – very good for program execution traces. Use -u a.out to see user-level function calls. Note “a.out” is typed literally as “a.out” – not replaced with the name of the program

IP network multipathing

Nilesh Ghogre — Tue, 22 Sep 2009 10:17:59 +0000

IP network multipathing provides your system with the following capabilities:

Recovery from single-point failures with network adapters
Increased traffic throughput

To use IP network multipathing, you must have an alternate adapter connected to the same IP link. Consequently, if a failure occurs in the network adapter, the system switches all the network accesses automatically from the failed adapter to the alternate adapter. This process ensures uninterrupted access to the network. Also, when you have multiple network adapters that are connected to the same IP link, you achieve increased traffic throughput. The traffic is spread across multiple network adapters

The Solaris implementation of IP network multipathing provides the following features:

Failure Detection – Ability to detect when a network adapter has failed and automatic switching (failover) of the network access to an alternate network adapter. Failure detection can only occur when you have configured an alternate network adapter. See Detecting Physical Interface Failures for more information.
Repair Detection – Ability to detect when a network adapter that failed previously has been repaired. The network access is switched back automatically (failback) to an alternate network adapter. Repair detection assumes that you have enabled failbacks. See Detecting Physical Interface Repairs for more information.
Outbound Load Spreading – Outbound network packets are spread across multiple network adapters without affecting the ordering of packets in order to achieve higher throughput. Load spreading occurs only when the network traffic is flowing to multiple destinations that use multiple connections.

IP Network Multipathing Components


Component	Description
IP Link	A communication facility or medium over which nodes can communicate at the link layer. The link layer is the layer immediately following IPv4 or IPv6. Examples include Ethernets, simple or bridged, or ATM networks. One or more IPv4 subnet numbers or prefixes are assigned to an IP link. A subnet number or prefix cannot be assigned to more than one IP link. In ATM LANE, an IP link is a single emulated LAN. When using ARP, the scope of the ARP protocol is a single IP link.
Network Interface Card (NIC)	Network adapter that is either internal or a separate card that serves as an interface to a link.
Physical interface	A node’s attachment to a link. This attachment is often implemented as a device driver plus a network adapter. Some network adapters can have multiple points of attachment, for example, qfe. The usage of Network adapter in this document refers to a “Single Point of Attachment.”
Physical interface group	The set of physical interfaces on a system that is connected to the same link. The set is identified by assigning the same (non-null) character string name to all the physical interfaces in the group.
Physical interface group name	A name that is assigned to a physical interface that identifies the group. The name is local to a system. Multiple physical interfaces, sharing the same group name, form a physical interface group.
Failure detection	The process of detecting when a NIC or the path from the NIC to a layer 3 device no longer works.
Repair detection	The process of detecting when a NIC or the path from the NIC to some layer 3 device starts operating correctly after a failure.
Failover	The process of switching network access from a failed interface to a good physical interface. Network access includes IPv4 unicast, multicast, and broadcast traffic, as well as IPv6 unicast and multicast traffic.
Failback	The process of switching back network access to an interface that is detected to have been repaired.
Standby Interface	A physical interface that is not used to carry data traffic unless some other physical interface in the group has failed.

The in.mpathd daemon can detect interface failure and repair

Grouping Physical Interfaces

You use the ifconfig command to configure groups. This command uses a new group parameter that requires a group name. The ifconfig command places both the IPv4 and IPv6 instances of the interface in that group. The group parameter has the following syntax:

ifconfig interface-name group group-name

You can have any number of network adapters that you can configure in the same

multipathing group. You cannot use the group parameter with logical interfaces.

For example, you can use the parameter with hme0, but not with hme0:1.

Configuring Test Addresses

You must configure all physical interfaces of a multipathing group with a test address. You need test addresses to detect failures and repairs. If a test address is not configured, the test address is not chosen for failover. Only in.mpathd uses test addresses. Normal applications should not use this address. This address is not chosen for failover when the interface fails. In IPv4, you should configure the test address in such a way that normal applications do not use the test address. The in.mpathd multipathing daemon requires a test IP address for detecting failures and repairs. You must use a routeable address for this IP address. The subnet prefix of the address must be known to any routers present on the link. You use the ifconfig command’s new -failover option to configure a test address. Use the following syntax to configure a test address:

# ifconfig interface-name addif ip-address  -failover up
EX:
# ifconfig hme0 addif 19.16.85.21 netmask + broadcast + -failover up

Preventing Applications From Using Test Addresses

After you have configured a test address, you need to ensure that this address is not used by normal applications. If you let applications use the test address, applications fail, because test addresses do not fail over during the failover operation. To ensure that IP does not pick the test address for normal applications, you mark the test address deprecated by using the ifconfig command. This parameter has the following syntax:

#ifconfig interface-name deprecated

Using the `hostname` File to Configure Groups and Test Addresses

You can use the /etc/hostname.interface files to configure multipathing groups and test addresses. To configure a multipathing group by using the /etc/hostname.interface file, you can add a line to the file by using the following syntax:

interface-address group group-name up \
addif logical-interface-address up

EX:#
19.16.85.19 netmask + broadcast + group test up \
addif 19.16.85.21 deprecated -failover netmask + broadcast + up
How to Configure a Multipathing Interface Group With Two Interfaces

Place each physical interface into a multipathing group by typing the following command.

# ifconfig interface-name group group-name

For example, to place hme0 and hme1 under group test, you type the following commands:

# ifconfig hme0 group test
# ifconfig hme1 group test

Configure a test address for all the physical interfaces.

# ifconfig interface-name addif ip-address -failover deprecated up

For example, if you wanted to configure a test address on hme0 with the following configuration:
§ Address set to 19.16.85.21
§ Netmask and broadcast address set to the default value
§ -failover and deprecated options set
You type the following command:

# ifconfig hme0 addif 19.16.85.21 netmask + broadcast + -failover deprecated up

You can check the configuration by typing the following:

# ifconfig hme0:1
hme0:1: flags=9000843
mtu 1500 index 2 inet 19.16.85.21 netmask ffffff00 broadcast 19.16.85.255

For example, if you want to configure a test address on hme1 with the following configuration:
§Address set to 19.16.85.22
§Netmask and broadcast address set to the default value
§-failover and deprecated options set
Type the following command:

# ifconfig hme1 addif 19.16.85.22 netmask + broadcast + -failover deprecated up

Configuring Standby Interfaces
You can configure multipathing groups with standby interfaces. As the name implies, the interface is considered a standby interface. The standby interface is not used unless some other interface in the group fails. A standby interface has an IFF_INACTIVE flag when the interface is not hosting any failover IP address. Consequently, when an active interface fails, the standby interface is always chosen for failover. After the standby interface is chosen, the IFF_INACTIVE flag is cleared on the standby interface. From that instant, the active standby interface is treated the same as other active interfaces. Some failures might not choose a standby interface. Instead, some failures might choose an active interface that hosts fewer IP addresses than the standby interface.
The standby interface is not used to send normal data packets. Consequently, limited traffic flows on a standby interface. You must configure standby interfaces with a test address to ensure that probes are sent to determine if the interface is functional. If you do not configure standby interfaces with a test address, the interface is not chosen for failovers when another interface in the group fails. A standby interface might carry traffic under the following conditions:

If another host on the network communicates with a host by using the standby interface address, the standby interface is subsequently used for incoming packets.
Applications can bind to an address by using bind or IP_ADD_MEMBERSHIP. Applications that bind to the address that is hosted on the standby interface might continue to generate traffic by using the standby interface.

Thus, the system does not normally select a standby interface, except for probes, unless the standby interface is explicitly chosen by an application. If some interface in the group fails, all network access is switched to the standby interface. To configure a standby interface, you use the ifconfig command's new standby parameter by using the following syntax:

# ifconfig interface-name standby group group-name