PURPOSE:

This page contains useful information on administering Solaris. Most of this information was acquired as a result of some need on our system, so may also be helpful to other people administering a small Solaris installation.

Topics-
  • Automatically applying Solaris patches with PCA
  • Automatically applying Solaris patches - My own script (before finding PCA)
  • Handling /var when it is too full
  • Setting up metadevices

    Automatically applying Solaris patches with PCA

    Martin Paul from the University of Vienna has created the Patch Check Advanced (PCA) perl
    script to make the process of patching Solaris much much easier.  Here's the 
    website for PCA.  This has the 
    script as well as information on using it. 
    

    Automatically applying Solaris patches

    Solaris provides some really great ways to keep your OS up-to-date in terms
    of patches to the kernel, libraries and applications.  It all revovles around
    the smpatch command.  The first thing to do is read the smpatch manpage.
    
    Done reading?  Ok.
    
    To repeat some man info: 
         The smpatch command can be used to download, apply, and 
         remove patches. Also, use the smpatch command to configure 
         the patch management environment for your system.
    
         The system on which you run Sun Patch Manager must  be  run-
         ning at least Solaris 8 and have the Developer Software Sup-
         port Group installed. If  your  system  runs  Solaris  8  or
         Solaris  9,  it  must  also  have  the Sun Patch Manager 2.0
         software installed. If your system runs Solaris 10  and  has
         the  Developer  Software  Support  Group  installed, the Sun
         Patch Manager 2.0 software is included.
    
    You need to configure the Patch Manager a little before you can use it.
    This is done with the smpatch set command.  Use the smpatch get
    command to see how things are currently configured.  Here's a list of 
    how things are configured on josie:
    [dfoster@josie admin_info]% smpatch get patchpro.backout.directory - "" patchpro.download.directory - /var/sadm/spool patchpro.install.types - rebootafter:reconfigafter:standard patchpro.patch.source - https://getupdates1.sun.com/solaris/ patchpro.patchset - current patchpro.proxy.host - "" patchpro.proxy.passwd **** **** patchpro.proxy.port - 8080 patchpro.proxy.user - "" Please note that the download directory is set to /var/sadm/spool (this is the default location for the download directory) and the backout directory is not set to anything, but will default to /var/sadm/pkg. By default, backout data is stored in the default locations used by patchadd. This is the save directory of each package that was modified by the patch. For example, if a patch modifies the SUNWcsr package, the backout data for that package is stored in the /var/sadm/pkg/SUNWcsr/save directory. Normally the default locations are fine, but be aware that if you keep backout info on all your patches, the backout directory can grow quite big. (see this about keeping the size of /var manageable) The install.types parameter determines the type of patches that smpatch can install.
    patchpro.install.types Your policy for applying patches. The value is a list of zero or more colon-separated patch properties that are permitted to be applied by an update operation (smpatch update). By default, patches that have the standard, rebootafter, and reconfigafter properties can be applied. Patches are classified as being standard or nonstandard. A standard patch can be applied by smpatch update. Such a patch is associated with the standard patch property. A non- standard patch has one of the following characteristics: o A patch that is associated with the rebootafter, rebootimmediate, reconfigafter, reconfigimmediate, or singleuser properties. Such a nonstandard patch can be applied during an update operation if permitted by the policy. o A patch that is associated with the interactive pro- perty. Such a patch cannot be applied by using the smpatch command. interactive A patch that cannot be applied by running the usual patch management tools (pprosvc, smpatch, or patchadd). Before this patch is applied, the user must perform spe- cial actions. Such actions might include checking the serial number of a disk drive, stopping a critical daemon, or reading the patch's README file. rebootafter The effects of this patch are not visible until after the system is rebooted. rebootimmediate When this patch is applied, the sys- tem becomes unstable until the sys- tem is rebooted. An unstable system is one in which the behavior is unpredictable and data might be lost. reconfigafter The effects of this patch are not visible until after a reconfigura- tion reboot (boot -r). See the boot(1M) man page. reconfigimmediate When this patch is applied, the sys- tem becomes unstable until the sys- tem gets a reconfiguration reboot (boot -r). An unstable system is one in which the behavior is unpredict- able and data might be lost. singleuser Do not apply this patch while the system is in multiuser mode. You must apply this patch on a quiet system with no network traffic and with extremely restricted I/O activity. standard This patch can be applied while the system is in multiuser mode. The effects of the patch are visible as soon as it is applied unless the application being patched is running while the patch is applied. In this case, the effects of the patch are visible after the affected applica- tion is restarted. You can see that josie has patch install.types of rebootafter:reconfigafter:standard This gets all the important patches we should need and the cron script we use alerts us if one of the installed patches requires a reboot. If you're running Solaris 10, set patchset parameter to "current". If you're running Solaris 8/9, set patchset paramenter to "patchdb" patch_josie.csh - This is the shell script I use to check for, download and install patches automatically. It's not really commented, but should be pretty self-explainitory. It creates a log of all the patches that are needed and all that are installed (or why they aren't installed). This is the root crontab entry that I use to run the script daily: # Automate checking for, downloading & installing patches every morning at 7:30am 30 7 * * * /root/bin/patch_josie.csh 2>&1 > /dev/null

    Handling /var when it is too full

    Important information about the /var directory and what to do if it gets too full
    
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    There is now a script ( clean_var_sadm_obsolete.csh) 
    that will automatically clean out /var/crash and all the obsolete.Z files in /var/sadm/pkg
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    
    Clean out /var/crash  This can contain large files from crashes that are just
    taking up space.
    
    The software registry is located in the /var file system. /var may vary widely in its
    disk space consumption because it contains the /var/tmp directory where users can
    place temporary files, as well as /var/adm and /var/log, which collect accounting
    and logging information. The software registry requires space to grow as software is
    installed. Any patches that are installed without explicitly using the .do not backup
    the files. option will store the files needed for back out in the /var/sadm/pkg
    directory. Enough space should be available in the /var directory to hold user
    temporary files plus software installation requirements and patch management. For
    a large server, the software registry may require 50MB or more and will grow over
    time as patches are added.
    
    ----------------------------------------------------------------
    If /var/sadm/pkg is really big (it holds patch backout info):
    ----------------------------------------------------------------
    Check for the existence of "obsolete.Z" files in
    the /var/sadm/pkg hierarchy. If you've done a lot of patching these
    files can take up a lot of space. They can be removed without ill
    effect. For example:
    
    # find /var/sadm/pkg -type f -name obsolete.Z -exec rm {} +
    
    This can recover several hundreds of megabytes depending on the number
    of patches applied.
    
    If you don't want to backout the patches again you can just get rid of the
    undo backup files:
    
    find  /var/sadm/pkg -name '1?????-??' -type d -exec rm -rf {} + 
    
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    There is now a csh script (clean_old_patch_backouts.csh) 
    that will search the /var/sadm/pkg directory and remove ONLY patch backout files that are older
    than the most recent backout file.  If there is only one backout file for a patch/pkg, then it will
    not be deleted.  You would be surprised how much space this can clean up.  I just got back 1.3GB of 
    space running this on my system, which has been installing regular patches for close to two years.
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    
    Optionally, you could change the download/backout directory for smpatch:
    
    One thing you can do in this case is to clean out /var/sadm/spool, and
    then set up a patch download directory on another partition that does
    have the space.  This only helps if /var/sadm has sufficient space for
    the patch and package files.
    
    ------------------------------------------------------------------------
    You can use `smpatch set' to specify a different download directory,
    which you have to first create.  Then `smpatch get' should show it,
    like this:
    
            $ smpatch get
            patchpro.backout.directory      -               ""
            patchpro.download.directory     /opt/sadm/spool /var/sadm/spool
    
    Changing this setting adds a line to /etc/patch/patch.conf, like this:
    
            patchpro.download.directory=/opt/sadm/spool
    
    The real solution is to ensure that /var is large enough.  5 GB should
    be sufficient.
    
    -----------------------------------------------------------------------------
    Do NOT create a symlink for /var/sadm/pkg or /var/sadm to somewhere else on a 
    bigger partition.  This will cause problems with patches as they are looking 
    for a directory, not a symlink.  If you want to do something like this, do
    it this way with a loopback mount.
    
    Therefor I prefer (if I cannot avoid them) loopback mounts on system
    directories. Say you want to migrate /var/sadm to /space/sadm:
    
    put an entry of the form
    
    /space/sadm     -       /var/sadm       lofs    -       yes     -
    
    in your /etc/vfstab somewhere after the line which mounts /space.  
    

    Setting up metadevices

    ********************************************************************************
    --------------------
    Disk setup on josie 
    --------------------
    Josie is setup for RAID 1.  This means all data is mirrored to two separate 
    drives.  In Josie's case, there are two IDE drives both installed as masters on 
    the separate IDE channels (/dev/dsk/c0t0d0 and /dev/dsk/c0t2d0 respectively).  
    Two IDE drives, each master of an independent IDE bus, are required in order 
    for the RAID-1 mirror to boot if either drive fails.  This is VERY important.  
    If they aren't both masters, random kernel panics/core dumps/reboots can also 
    occur as IDE bus errors occur during normal mirroring (I learned this the hard 
    way)
    c0t0d0 is 80GB and c0t2d0 is 120GB.  Each disk is "broken" into 7 "slices" 
    (which we can think of as partitions) and the disks must be partitioned 
    identically (or the second mirror has to have larger partitions than the master 
    mirror).  For josie, the smaller drive determines how much is mirrored (80GB, 
    actually around 75GB after formatted).  This means the 120GB drive will have an 
    extra ~40GB of space (slice 4 of c0t2d0) that will not be mirrored, but can be 
    mounted and used as a regular directory.  
    For josie the disks were partitioned as follows:
    c0t0d0
    slice  name    	      cylinders      size	blocks
    --------------------------------------------------------------------------------
    0       root    wm  1062 - 38306    72.46GB    (37245/0/0) 151959600
    1       swap    wu     0 -   513  1023.98MB    (514/0/0)     2097120
    2     backup    wm     0 - 38306    74.53GB    (38307/0/0) 156292560
    3        var    wm   548 -  1061  1023.98MB    (514/0/0)     2097120
    4 unassigned    wm     0             0         (0/0/0)             0
    5 unassigned    wm   538 -   547    19.92MB    (10/0/0)        40800
    6 unassigned    wm   528 -   537    19.92MB    (10/0/0)        40800
    7 unassigned    wm   518 -   527    19.92MB    (10/0/0)        40800
    --------------------------------------------------------------------------------
    c0t2d0
    slice  name    	      cylinders      size	blocks
    --------------------------------------------------------------------------------
    0       root    wm  2091 - 39335    72.46GB    (37245/0/0) 151959600
    1       swap    wu     0 -   513  1023.98MB    (514/0/0)     2097120
    2     backup    wm     0 - 57458   111.79GB    (57459/0/0) 234432720
    3        var    wm   548 -  2090     3.00GB    (1543/0/0)    6295440
    4 unassigned    wm 39336 - 57458    35.26GB    (18123/0/0)  73941840
    5 unassigned    wm   538 -   547    19.92MB    (10/0/0)        40800
    6 unassigned    wm   528 -   537    19.92MB    (10/0/0)        40800
    7 unassigned    wm   518 -   527    19.92MB    (10/0/0)        40800
    
    --------------------------------------------------------------------------------
    slice 2 is generally reserved and spans the entire disk, so is never mounted 
    (but provides an easy way to back up an entire disk to tape)
    Only slice 0, 1 and 3 are mirrored.  
    Slices 5, 6 & 7 are identical on both disks, but didn't need to be.  Each of 
    those slices (6 in all) are used to hold a state database replica, which is 
    basically what the system uses to keep the mirrors all in sync.  It is important 
    to have multiple state database replicas on each disk, since if one disk fails, 
    it still needs at least three to keep going (or something like that).  These 
    slices should be at least 4MB in size.
    
    Mirroring uses the concept of metadevice, where we create a metadevice for 
    every partition we want to mirror.  Then the two partitions that will actually 
    be mirrored are made into sub-mirrors and attched to the metadevice as such.   
    Below shows how the metadevices are setup on josie.
    
    d0 is /
     d10 is disk0 /	(c0t0d0s0)
     d20 is disk1 / (c0t2d0s0)
    d1 is swap
     d11 is disk0 swap  (c0t0d0s1)
     d21 is disk1 swap  (c0t2d0s1)
    d3 is /var
     d13 is disk0 /var  (c0t0d0s3)
     d23 is disk1 /var  (c0t2d0s3)
    
    -------------------------------
    To Create and setup metadevices
    -------------------------------
    // Assumption: Solaris is initially installed on disk 0 before setting up RAID
    
    metadb -c 2 -a -f /dev/dsk/c0t0d0s5   //Create copies of state database 
    metadb -c 2 -a -f /dev/dsk/c0t0d0s6   //replica, The "-c 2" will create 
    metadb -c 2 -a -f /dev/dsk/c0t0d0s7   //two replicas per slice (That 
    metadb -c 2 -a -f /dev/dsk/c0t2d0s5   //really isn't necessary)
    metadb -c 2 -a -f /dev/dsk/c0t2d0s6
    metadb -c 2 -a -f /dev/dsk/c0t2d0s7
    
    metainit -f d10 1 1 c0t0d0s0	      //Initialize slices as "sub-mirrors"
    metainit -f d20 1 1 c0t2d0s0	      //"-f" will force initialization of
    metainit -f d11 1 1 c0t0d0s1	      //mounted devices
    metainit -f d21 1 1 c0t2d0s1
    metainit -f d13 1 1 c0t0d0s3
    metainit -f d23 1 1 c0t2d0s3
    
    metainit d0 -m d10	 	      //Set one of the submirrors as the
    metainit d1 -m d11		      //master mirror for the metadevice
    metainit d3 -m d13
    
    metaroot d0			      //Updates /etc/vfstab to use metadevices
    
    The /etc/vfstab file must be updated at this point to reflect the changes made 
    to the system. The / partition will have already been updated through the 
    metaroot command run earlier, but the system needs to know about the new 
    devices for swap and /var. The entries in the file will look something like the 
    following:
    
    /dev/md/dsk/d1  -       	-       swap    -       no      -
    /dev/md/dsk/d3  /dev/md/rdsk/d3 /var    ufs     1       yes      -
    
    Notice that the device paths for the disks have changed from the normal 
    style /dev/dsk/c#t#d#s# and /dev/rdsk/c#t#d#s# to the new metadevice paths, 
    /dev/md/dsk/d# and /dev/md/rdsk/d#.
    
    REBOOT
    
    metattach d0 d20	//Attach the other submirror to the metadevice
    metattach d1 d21	//Now we should have a functioning RAID 1 system and
    metattach d3 d23	//it should begin syncing up the drives at this point
    			//Do not do anything during initial resyncing
    
    // Make mirrored disk bootable
    installboot /usr/platform/'uname -i'/lib/fs/ufs/bootblk /dev/rdsk/c0t2d0s0
    
    //Optional
    //To enable easy booting from second mirror device (in case first disk fails), 
    //we can add an alias to the Open Boot Prom
    ls -l /dev/dsk/c0t0d0s0	  //Returned value is full device name for first disk
    ls -l /dev/dsk/c0t1d0s0	  //Returned value is full device name for second disk
    
    //at OBP prompt:
    ok nvalias  disk0 /pci@1f,0/pci@1,1/ide@3/disk@0,0   //For disk0 in josie 
    ok nvalias  disk1 /pci@1f,0/pci@1,1/ide@3/disk@1,0   //For disk1 in josie
    (For some reason the ls -l command above returns 
    /pci@1f,0/pci@1,1/ide@3/dad@0,0:a, but OBP won't seem to boot from that.)
    //Now you can boot the second device at the OBP prompt by 
    //just typing "boot disk1"
    
    // To enable the system to boot from the other disk is one disk fails, 
    // do the following at the OBP:
    ok setenv boot-device disk0 disk1 
    
    // Remember that the secondary disk needs to be made bootable using the 
    // "installboot" command above.  Now the system can boot from any disk, but 
    // the problem will be that if one disk fails, only half the database replicas 
    // will be available, and the system complains about insufficient database
    // replicas, write permissions, read permissions, etc.  To overcome this 
    // problem, modify the file /etc/system by appending the line:
    
    set md:mirrored_root_flag=1
    
    // That way, it will allow the system to boot even if just half the 
    // database replicas are available (Since we include 3 database replicas 
    // on each disk, this may not be necessary...
    
    // Alternatively, if a disk dies and Solaris wants a quorum of DB replicas, 
    // boot into single user mode, delete the database replicas on the failed disk 
    // and reboot:
    boot -s 
    metadb -d /dev/dsk/c0t?d0s5
    metadb -d /dev/dsk/c0t?d0s6
    metadb -d /dev/dsk/c0t?d0s7
    reboot
     
    ---------------------------------
    Checking Metadevice (RAID) status
    ---------------------------------
    Use the metastat command.  
    To see the status of all metadevices, use it without arguments.
    	metastat 
    To see the status of an individual metatdevice, 
    specify the metadevice as an argument:
    	metadevice d0
    
    -----------------------------------------------------
    To detach a submirror (disk) [i.e. for replacement]
    -----------------------------------------------------
    // Assumption: We're replacing disk 1, do detach all submirrors on it
    metadetach d0 d20	//Detaches submirror for metadevice
    metadetach d1 d21	//May need to use the -f flag if the disk
    metadetach d3 d23	//"needs maintenance"
    
    metadb -d c0t1d0s5	//Remove state database replicas from the disk 
    metadb -d c0t1d0s6	//(use metastat -i to verify)
    metadb -d c0t1d0s7
    
    metaclear d20		//Remove submirror definition/instantiation 
    metaclear d21		//(use metastat to verify)
    metaclear d23
    
    ---------------------
    To Attach a submirror
    ---------------------
    // Assumption: We just replaced disk 1 with a new disk and partitioned and 
    // formatted it correctly (see above)
    metadb -a -f /dev/dsk/c0t1d0s5		//Add database state replicas to new 
    metadb -a -f /dev/dsk/c0t1d0s6		//hard drive (3 copies)
    metadb -a -f /dev/dsk/c0t1d0s7
    
    metainit -f d20 1 1 c0t1d0s0		//Initialize new submirrors
    metainit -f d21 1 1 c0t1d0s1
    metainit -f d23 1 1 c0t1d0s3
    
    metattach d0 d20			//Attach submirrors to metadevice 
    metattach d1 d21			//(disks are resynced at this point)
    metattach d3 d23			//Do nothing while disks are resyncing
    
    metaroot d0				//Updates /etc/vfstab to use metadevices
    
    //Make new mirrored disk bootable
    installboot /usr/platform/'uname -i'/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0
    
    // If both submirrors are replaced with bigger drives (replacing one at a 
    // time, resyncing data in between), and some of the slices are bigger now, 
    // use the growfs command to enable the system to recognize the additional 
    // space available.
    
    ------------------------------------------------------
    To be able to boot with only 50% of metaDBs available 
    (one disk failure in a two-disk mirror setup)
    ------------------------------------------------------
    You can choose to force Solstice DiskSuite software to start if half of 
    the state database replicas are available by setting the tunable 
    mirrored_root_flag to 1 in /etc/system. The default value of this tunable 
    is disabled, which requires that a majority of all replicas are available 
    and are in sync before Solstice DiskSuite software will start. To enable 
    this tunable, type the following: 
    
    echo set md:mirrored_root_flag=1 >> /etc/system
    
    Caution   Enable the mirrored_root_flag only if the following prerequisites 
    are met: 1) The configuration only has two disks. 2) Unattended reboots 
    of a system with only two disks are a requirement and the risk of booting 
    from a stale replica, and therefore, stale volume management system state, 
    is acceptable. This means that in your setup, system availability is more 
    important than data consistency and integrity. Setting this tunable risks 
    data corruption or data loss in the event of a transient disk failure, but 
    allows unattended reboots. See the Solaris Volume Manager documentation on 
    docs.sun.com for more information about state database replicas (metadbs). 
    
    ********************************************************************************