Fixing a DBCS Startup Failure Caused by ORA-07445

 

Introduction:

Running Oracle databases on Oracle Database Cloud Service (DBCS) usually provides a stable and well-automated environment, but unexpected platform-level issues can still occur—especially when critical OS-level components are modified or removed. Recently, I encountered a serious problem where a DBCS compute node failed to start the database, and the stack repeatedly threw an ORA-07445 error during the startup process.

At first glance, the issue looked like a typical software or patch conflict, but after deeper investigation, it turned out to be something far more fundamental: the swap mount point and related configuration had been cleared or removed on the compute instance. Without an active swap device, several Oracle background processes failed during initialization, resulting in the ORA-07445 crash.

In this blog, I’ll walk you through the symptoms, diagnostic steps, and the exact solution that restored the instance.

Identifying the Error in the Alert Log

Upon examining the log, I immediately noticed multiple occurrences of the ORA-07445, ORA-27102, and ORA-27103 errors. This type of error typically indicates an unexpected exception at the OS or process level, which suggests that the issue was not simply a database configuration problem but something deeper affecting Oracle’s ability to initialize its background processes

Here is what I saw in the alert log.

[oracle@Eclipsys diag]$ tail -100 ./rdbms/devbase_oci/devbase/trace/alert_devbase.log
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.249.3.4)(PORT=10898))
2025-11-05T09:55:05.725758-05:00
WARNING: inbound connection timed out (ORA-3136)
2025-11-05T09:55:06.629264-05:00
Errors in file /u01/app/oracle/diag/rdbms/devbase_oci/devbase/trace/devbase_cjq0_15179.trc (incident=3238875):
ORA-00445: background process "J000" did not start after 30 seconds
Incident details in: /u01/app/oracle/diag/rdbms/devbase_oci/devbase/incident/incdir_3238875/devbase_cjq0_15179_i3238875.trc
2025-11-05T09:55:08.998626-05:00
Errors in file /u01/app/oracle/diag/rdbms/devbase_oci/devbase/incident/incdir_3239010/devbase_j007_15593_i3239010.trc:
ORA-27103: erreur interne
Additional information: 11818
Additional information: 9
ORA-27102: m▒moire insuffisante
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 10548
Additional information: 1114112

Finding the root cause of the problem.

When I executed the top command, I found that the kswapd process was consuming a high percentage of CPU, which occurs when the OS is unable to allocate memory to processes.

As shown in the image above, the system’s memory is nearly fully utilized, and the OS is not using any swap space. Therefore, I decided to check the swap configuration using the swapon --show command, which returned no results. I then used additional commands to confirm that the OS was not using any swap space and that swap had not been configured on this system.

[root@Eclipsys ~]# swapon --show
[root@Eclipsys ~]# sar -S 2 10
10:15:45 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
10:15:47 AM         0         0      0.00         0      0.00
10:15:49 AM         0         0      0.00         0      0.00
10:15:51 AM         0         0      0.00         0      0.00
10:15:54 AM         0         0      0.00         0      0.00
10:15:56 AM         0         0      0.00         0      0.00
10:15:58 AM         0         0      0.00         0      0.00
10:16:00 AM         0         0      0.00         0      0.00
10:16:06 AM         0         0      0.00         0      0.00
10:16:08 AM         0         0      0.00         0      0.00
10:16:10 AM         0         0      0.00         0      0.00
Average:            0         0      0.00         0      0.00

The output of the lsblk command indicated that the vg00-swap  volume was not mounted and had no corresponding mount point on the system.

[root@Eclipsys ~]# lsblk
NAME                   MAJ:MIN    RM  SIZE RO TYPE MOUNTPOINT
sdf                      8:80      0   64G  0 disk
asm!commonstore-231    251:118273  0    5G  0 disk /opt/oracle/dcs/commonstore
sdd                      8:48      0  200G  0 disk /u01
sdb                      8:16      0   64G  0 disk
sdi                      8:128     0   64G  0 disk
sdg                      8:96      0   64G  0 disk
sde                      8:64      0   64G  0 disk
sdc                      8:32      0   64G  0 disk
sda                      8:0       0   75G  0 disk
├─sda4                   8:4       0 25.4G  0 part
│ ├─vg00-opt           252:4       0   35G  0 lvm  /opt
│ └─vg00-var           252:3       0   10G  0 lvm  /var
├─sda2                   8:2       0  512M  0 part /boot
├─sda3                   8:3       0   49G  0 part
│ ├─vg00-var_log       252:1       0    4G  0 lvm  /var/log
│ ├─vg00-root          252:6       0   10G  0 lvm  /
│ ├─vg00-opt           252:4       0   35G  0 lvm  /opt
│ ├─vg00-var_tmp       252:2       0    1G  0 lvm
│ ├─vg00-var_log_audit 252:0       0    2G  0 lvm  /var/log/audit
│ ├─vg00-swap          252:7       0   10G  0 lvm
│ ├─vg00-home          252:5       0    1G  0 lvm  /home
│ └─vg00-var           252:3       0   10G  0 lvm  /var
└─sda1                   8:1       0  128M  0 part /boot/efi
sdj                      8:144     0   64G  0 disk
sdh                      8:112     0   64G  0 disk
asm!exportvol-217      251:111105  0   75G  0 disk /acfsmounts/exports

Then I checked the /etc/fstab file, and it showed that no mount point for swap was defined. Below was the fstab configuration on the OS:

tmpfs /var/tmp tmpfs nodev,nosuid,noexec 0 0
tmpfs /tmp tmpfs nodev,nosuid 0 0
tmpfs /dev/shm tmpfs nodev,nosuid 0 0
swap none    swap    sw,_netdev,comment=cloudconfig  0   0

Solution

I added the swap path for DBCS to the /etc/fstab file as the last line.

tmpfs /var/tmp tmpfs nodev,nosuid,noexec 0 0
tmpfs /tmp tmpfs nodev,nosuid 0 0
tmpfs /dev/shm tmpfs nodev,nosuid 0 0
###swap none    swap    sw,_netdev,comment=cloudconfig  0 0
/dev/mapper/vg00-swap   none    swap    sw,_netdev,comment=cloudconfig  0 0

I executed the command mount -a to make the swap space immediately usable. Then I executed swapon command to activate the swap on the OS.

[root@Eclipsys ~]# mount -a
[root@Eclipsys ~]# mount
[root@Eclipsys ~]# swapon /dev/mapper/vg00-swap
[root@Eclipsys ~]# cat /proc/swaps
Filename                  Type            Size    Used    Priority
/dev/dm-7                 partition       10485756        989440  -2

Now, when I checked the OS using the top command, it indicates that the swap is being used by the system.

Conclusion

This incident highlights the critical role of the OS-level swap configuration in Oracle Database Cloud Service environments. Even in a stable and automated cloud setup, removing or misconfiguring fundamental components like the swap mount point can prevent background processes from starting, leading to ORA-07445 and related errors.

By carefully examining the alert logs and system metrics, it became clear that the root cause was not a software bug or patch conflict, but a lack of available virtual memory. Restoring the swap configuration resolved the issue and allowed the database to start normally.