Introduction:
Upgrading Solaris is never very simple work. It is not like just
running one command and everything finishes. Especially when you are going from
Solaris 11.4.22 to 11.4.89, it becomes more complicated and needs more careful
thinking. In this situation, it is not only about updating packages but also
about understanding how Global Zone (GZ) and Non-Global Zones (NGZs) are
working together and how this relation will change.
One of the main problems in this upgrade is something called a
"Sync-Linked" exception. This happens when dependencies between
packages are very tight, and the system does not allow upgrading all zones
together in one step. Because of this, we cannot follow the normal upgrade
path. We need to separate the process and let Global Zone go first, independent
of others.
In this blog series, I will go through the complete upgrade process
from Solaris 11.4.22 to 11.4.89 and share the steps I followed in my
environment. I will explain how to upgrade the Global Zone and Non-Global
Zones, what issues I faced during the process, and how I solved them. I will
also cover some mistakes that can happen during the upgrade and the recovery
steps that helped me bring the system back when things did not go as expected.
My hope is that this real-world experience can help others who are planning a
similar Solaris upgrade.
Solaris 11.4 Upgrade Series:
- Part 1: Pre-Upgrade Steps and Upgrading
the Solaris Global Zone (Current)
- Part 2: Upgrading Solaris Non-Global Zones
- Part 3: Lessons Learned – Using the Wrong
Zone Image
- Part 4: Recovering Solaris Oracle Mount
Points During Upgrade
In this first part, we focus on the basic and very important steps. We check the system before the upgrade, prepare ZFS snapshots for safety, and then perform the upgrade on the physical host to reach the 11.4.89 kernel. These steps are very critical because if something goes wrong, recovery will be difficult.
1. Pre-Maintenance (System Online)
These steps should be done before the maintenance window starts. It is very important not to skip anything here, because later it can cause a big problem.
- Centralized
Backup Storage:
First, we need one ZFS pool with a size bigger than the sum of the sizes of all databases in all the Non-Global Zones and mount it to all Non-Global Zones. For example, we can use a path like /u99/backups. This will be used for keeping all backups in one place.
- Oracle
RMAN Backups:
Take a full backup of all databases in all Non-Global Zones using RMAN. Save everything inside the shared mount /u99/backups. Make sure the backup is completed successfully, not partial or failed.
- Collect information like IP/Adress and all the mount points related to the global zone and all non-global zones.
- Collect System Information:
We need to collect all important information, like IP addresses, hostname, and all mount points for the Global Zone, and also all Non-Global Zones. This will help later if something goes wrong or needs recovery.
- Space Verification:
for zone in $(zoneadm list -cp | grep -v "global" | cut -d: -f2); do
zfs list "os_zone_$zone"
doneThis is the sample output for above command
- System Protection (Snapshots):
To avoid using too much space inside each non-global zone OS mount point, I changed the way backups are stored. Instead of saving metadata inside each zone, this script sends all backup data to a central location /var/share in the Global Zone.
In this way, we don’t fill up the limited disk space of each zone
during upgrade time. Also, by using ZFS snapshots at the pool level (not
copying files inside zones), we can still have a full recovery point. This is a
better approach because it does not need extra space inside the active
filesystem of the zones and keeps the system more stable during upgrades.
BACKUP_DIR="/var/share/backup/solaris_upgrade_$(date +%F)"
mkdir -p $BACKUP_DIR
# Use -cp to get the parsable list (including configured/installed zones)
for z in $(zoneadm list -cp | cut -d: -f2 | grep -v global); do
echo "--- Preparing: $z ---"
# 1. Export the 'Recipe' to the CENTRAL GZ directory
zonecfg -z $z export > $BACKUP_DIR/$z.zonecfg.backup
# 2. FIX: Identify the ZFS DATASET name, not just the path
# We query the 'dataset' property of the zone specifically
ZDATASET=$(zonecfg -z $z info dataset | grep name | awk '{print $2}')
# If the above returns empty (common if not using delegated datasets),
# we find the dataset based on the Zone Path
if [ -z "$ZDATASET" ]; then
ZPATH=$(zoneadm -z $z list -p | cut -d: -f4)
ZDATASET=$(df -h $ZPATH | tail -1 | awk '{print $1}')
fi
# 3. Take a Recursive 'Data Snapshot'
echo "Creating snapshot for dataset: $ZDATASET"
zfs snapshot -r ${ZDATASET}@pre_upgrade_full
# 4. Safety Check
# We check for the dataset name in the snapshot list
zfs list -t snapshot | grep "${ZDATASET}@pre_upgrade_full"
echo "Done with $z. Metadata saved to $BACKUP_DIR"
echo "--------------------------------------"
doneAfter running the script, verify your snapshots using zfs list (the following commands).
This command shows the pre_upgrade_full snapshot that we made for each Non-Global Zone.
zfs list -t snapshot -o name,creation | grep pre_upgrade_full
This command shows the list of all snapshots related to a specific Non-Global Zone.
zfs list -t snapshot -o name,used,refer,creation | grep <Non-Global Zone Name> Example of output of the second command for a specific Non-Global
Zone
You will notice multiple entries for a single zone; this is the
expected result of a recursive snapshot. It ensures that not only the
root OS but also all delegated datasets—such as Oracle /u01 or /u02—are
captured at the exact same timestamp. This creates a consistent state across
the entire environment, allowing for a total rollback if the version jump encounters
issues.
2. Global Zone (GZ) Upgrade
After all backups are completed and verified, it is finally time to
start the actual upgrade. This is the stage where we move from preparation to
execution.
Before touching anything, I always like to verify the current state
of the system. It is important to know exactly where we are starting from
before making any major changes to the operating system. For this reason, I
first check the current Solaris release and then verify the active Boot
Environment (BE).
root@sun3:~# pkg info entireThe output confirms that the server is currently running Solaris 11.4.22.
Next, verify the available Boot Environments using the beadm list
command:
From the output, we can see that the active Boot Environment is S11.4SRU22. The flags N and R indicate that this BE is currently active and will also be used on the next reboot. At this point, we know exactly where the system stands before starting the upgrade.
2.1. Understanding the Sync-Linked Issue
Before shutting down the zones, I wanted to see what would happen
if I performed a normal upgrade validation.
To do this safely, I executed the following dry-run command:
pkg update -nvThe -n option tells Solaris to simulate the update without making
any actual changes. I usually do this before major upgrades because it helps
identify dependency problems before the maintenance window becomes critical.
When I executed the command, the update process failed with a
Sync-Linked exception.
The output clearly shows that Solaris was unable to complete the
linked image operation for the local zone. The package dependency chain could
not be resolved because the Global Zone was attempting to move to a newer
package level while the Non-Global Zone remained on older package versions.
This was the confirmation I needed. A direct upgrade of the entire
server was not going to work. The Global Zone and Non-Global Zones had to be
treated separately.
After seeing this output, I changed the upgrade approach and
decided to upgrade the Global Zone first while keeping all Non-Global Zones
offline. Once the Global Zone was upgraded successfully, I could then focus on
upgrading and synchronizing the zones separately.
2.2. Stopping the Workloads Cleanly
Before doing any operating system upgrade, make sure all
applications and databases are stopped correctly. Upgrading while workloads are
still active can create unnecessary risk and recovery work later.
First stop application services and Oracle databases inside each
zone.
zloging <Non-GlobalZone-Name>
su - oracle
# Inside Oracle zones
lsnrctl stop
sqlplus / as sysdba
shutdown immediate;After all application workloads are stopped, halt the Non-Global
Zones from the Global Zone.
for zone in $(zoneadm list -p | grep -v "global" | cut -d: -f2); do
echo "Shutting down zone: $zone"
zoneadm -z $zone halt
doneVerify all zones are stopped:
zoneadm list -cv
2.3. Updating the Solaris Global Zone
With all zones safely shut down, we can start the operating system
upgrade.
Before running the upgrade, I opened a Screen session:
screen -S sru89
pkg update --accept --be-name SRU89_Update 2>&1 | tee /var/tmp/update_sru89.logI always do this for major upgrades because some updates can take a
long time to finish. If the SSH session disconnects or my MobaXterm window
closes by mistake, the upgrade process will continue running inside the Screen
session. Later, I can reconnect and resume the session without interrupting the
upgrade.
The -S option allows us to give the session a name. In this case, I
used sru89 so it is easy to identify. If needed, I can reconnect to the
session using:
screen -r sru89After creating the Screen session, I started the upgrade using the pkg update command that I wrote above. Here is the end of the output of the above-mentioned command.
I prefer to create a clearly named Boot Environment during upgrades because it makes troubleshooting and rolling back much easier later. Instead of using an automatically generated name, I created a Boot Environment called SRU89_Update.
I also redirected both standard output and error messages to a log
file using tee. This allows me to monitor the upgrade in real time while
keeping a complete record of the process in /var/tmp/update_sru89.log.
2.4. Verify the New Boot Environment
After the upgrade was completed and before rebooting, I verified that
the new Boot Environment had been created successfully using beadm list command:
From the output, we can see that the newly created Boot Environment
SRU89_Update is marked with the R flag. This means Solaris will
use this Boot Environment on the next reboot.
The existing Boot Environment S11.4SRU22 is still present
and marked with the N flag, indicating it is the environment currently
running. This is one of the major advantages of Solaris Boot Environments. The
upgrade does not modify the active operating system directly. Instead, it
creates a separate bootable copy that can be tested safely.
If anything goes wrong after rebooting into SRU89_Update, it
is possible to roll back and boot from the previous environment without
reinstalling the operating system.
At this stage, the upgrade packages had already been applied to the
new Boot Environment, but the server was still running on the old S11.4SRU22
environment. The final step was simply to reboot the server and activate the
upgraded image.
2.5. Activating the New Boot Environment
Reboot the physical server:
init 6The server will boot into the newly created Boot Environment.
uname -a
The output should show the updated Solaris 11.4.89 kernel.
Check the status of all zones:
zoneadm list -cv
At this stage, the Global Zone is running Solaris 11.4.89 while the Non-Global Zones are still using the older software level. This is expected behavior.
If you try to boot one of the zones now, it will fail because the
zone software has not yet been upgraded to match the Global Zone version.
Conclusion
At this stage, the Global Zone upgrade is complete and the physical
server is successfully running Solaris 11.4.89.
Our ZFS snapshots are available as rollback points, the backup
metadata has been stored safely, and the new Boot Environment is active.
However, the work is not finished yet.
The Non-Global Zones are still running older binaries and cannot be
started until they are upgraded and aligned with the new Global Zone version.
In Part 2, I will walk through the complete process of upgrading
the Non-Global Zones, dealing with zone attachment issues, package
synchronization, and making sure Oracle databases and applications come back
online safely after the migration.
.png)
