Cross-Version Oracle Database Automation on Oracle Linux (7, 8, 9) with systemd, dbstart/dbshut, and Email Alerts on Failure

 

Introduction

In Oracle environments that do not use Oracle ASM, databases are not configured to start automatically after a server reboot. To automate database startup and shutdown, administrators commonly use Oracle's standard dbstart and dbshut utilities.

Although these tools provide basic automation, they do not always verify that the database has successfully reached the OPEN state, and they do not provide built-in notification when startup failures occur.

This article demonstrates how to combine dbstart/dbshut, systemd, startup validation checks, and automated email alerts to create a reliable Oracle database auto-start framework. The approach is compatible with Oracle Linux 7, 8, and 9 and can be used with Oracle database versions that support the standard dbstart/dbshut utilities. It also includes a safe testing method for validating the email alert system without affecting a production database.

In this blog post, I design:

• An Oracle auto-start service using systemd
• Proper shutdown control
• An email alert system for service failures
• A safe test service to validate the alerting mechanism without production risk

This design is simple, stable, and suitable for Oracle Linux production environments.

Part 3: Lessons Learned – Upgrading a Solaris Non-Global Zone Using the Wrong Zone Image

 

Introduction

In Part 2 of this series, I described how I upgraded several Solaris Non-Global Zones using an out-of-place upgrade method. Because the zones did not have enough temporary working space for a direct IPS upgrade, I copied the zone filesystems to a larger storage area, performed the package update there, and then synchronized the updated files back to the original zones.

The approach worked, but it also introduced additional manual steps compared to a standard upgrade. As with many administrative tasks, the technical procedure itself was not the biggest challenge. The real challenge was making sure each step was performed against the correct zone and the correct filesystem.

During the upgrade process, I made two mistakes that led to some unexpected troubleshooting. First, I accidentally synchronized files from the wrong upgraded zone image, causing one Non-Global Zone to boot with another zone's hostname, IP address, and listener configuration. Later, while correcting the network settings, I mistakenly executed an IP configuration command in the Global Zone instead of the target Non-Global Zone, which immediately disconnected my SSH session and required recovery through the server's ILOM console.

Fortunately, neither issue resulted in data loss, and both were recoverable. This article documents what happened, how the problems were identified, the recovery steps I followed, and the lessons I took away from the experience.

Part 2: Upgrading Solaris Non-Global Zones

 

Introduction

In Part 1 of this series, I covered the process of upgrading the Global Zone from Solaris 11.4 SRU22 to SRU90. Once the Global Zone upgrade was completed successfully, the next task was upgrading the Non-Global Zones running on the same server.

According to Oracle's recommended approach, upgrading Non-Global Zones is usually a straightforward process. If the zone has sufficient free space available, the update can normally be completed by mounting the zone, updating the packages, and then booting the zone.

In many environments, this method works without any issues. However, during this project I encountered an unexpected problem. Even though several Non-Global Zones had more than 12 GB of free space available, the standard upgrade procedure repeatedly failed. This article documents the issue I encountered, the troubleshooting process, and the workaround that ultimately allowed the upgrades to complete successfully.

Part 1: Pre-Upgrade Steps and Upgrading the Solaris Global Zone

 

Introduction:

Upgrading Solaris is never very simple work. It is not like just running one command and everything finishes. Especially when you are going from Solaris 11.4.22 to 11.4.90, it becomes more complicated and needs more careful thinking. In this situation, it is not only about updating packages but also about understanding how Global Zone (GZ) and Non-Global Zones (NGZs) are working together and how this relation will change.

One of the main problems in this upgrade is something called a "Sync-Linked" exception. This happens when dependencies between packages are very tight, and the system does not allow upgrading all zones together in one step. Because of this, we cannot follow the normal upgrade path. We need to separate the process and let Global Zone go first, independent of others.

In this blog series, I will go through the complete upgrade process from Solaris 11.4.22 to 11.4.90 and share the steps I followed in my environment. I will explain how to upgrade the Global Zone and Non-Global Zones, what issues I faced during the process, and how I solved them. I will also cover some mistakes that can happen during the upgrade and the recovery steps that helped me bring the system back when things did not go as expected. My hope is that this real-world experience can help others who are planning a similar Solaris upgrade.