今天周末，一大早开发经理发消息说项目上有个数据库连不上了，让上去排查一下。连上数据库之后，监听是启动状态，但是并没有注册数据库，看日志早上8点多的时候数据库出错，自动关掉了，错误日志如下
1
| tail -n 1000 /u01/app/oracle/diag/rdbms/orcl/orcl/trace/alert_orcl.log
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
| Sat Apr 18 23:50:29 2020
Thread 1 cannot allocate new log, sequence 11779
Private strand flush not complete
Current log# 3 seq# 11778 mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
Thread 1 advanced to log sequence 11779 (LGWR switch)
Current log# 1 seq# 11779 mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Sun Apr 19 01:29:06 2020
Thread 1 advanced to log sequence 11780 (LGWR switch)
Current log# 2 seq# 11780 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log
Sun Apr 19 02:00:00 2020
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
Sun Apr 19 03:02:45 2020
Thread 1 advanced to log sequence 11781 (LGWR switch)
Current log# 3 seq# 11781 mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
Sun Apr 19 04:49:42 2020
Thread 1 advanced to log sequence 11782 (LGWR switch)
Current log# 1 seq# 11782 mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Sun Apr 19 06:00:00 2020
Setting Resource Manager plan SCHEDULER[0x32DF]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Apr 19 06:00:00 2020
Starting background process VKRM
Sun Apr 19 06:00:00 2020
VKRM started with pid=58, OS id=17023
Sun Apr 19 06:00:02 2020
Begin automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Sun Apr 19 06:00:07 2020
Thread 1 advanced to log sequence 11783 (LGWR switch)
Current log# 2 seq# 11783 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log
Sun Apr 19 06:00:51 2020
End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Sun Apr 19 07:10:49 2020
Thread 1 advanced to log sequence 11784 (LGWR switch)
Current log# 3 seq# 11784 mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
Sun Apr 19 08:00:49 2020
KCF: read, write or open error, block=0xe0 online=1
file=3 '/u01/app/oracle/oradata/orcl/undotbs01.dbf'
error=27072 txt: 'Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 224
Additional information: -1'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_dbw1_19450.trc:
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_dbw1_19450.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 3 (block # 224)
ORA-01110: data file 3: '/u01/app/oracle/oradata/orcl/undotbs01.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 224
Additional information: -1
DBW1 (ospid: 19450): terminating the instance due to error 63999
Sun Apr 19 08:00:49 2020
System state dump requested by (instance=1, osid=19450 (DBW1)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_19440_20200419080049.trc
Dumping diagnostic data in directory=[cdmp_20200419080049], requested by (instance=1, osid=19450 (DBW1)), summary=[abnormal instance termination].
Instance terminated by DBW1, pid = 19450
|
第一反应是磁盘空间是不是满了，查看了系统空间，/u01分区剩余还很多
经查看，这台机器是虚拟机环境，非实体机，可能存在存储超分的情况（分配出去的存储空间大于实际可用的物理空间，倒是物理空间被耗尽后，系统中依然看到存在富裕剩余空间）
1
2
| [root@db-23-200 ~]# dmidecode |grep -i prod
Product Name: Bochs
|
为了进一步验证是否存在超分，删除了一个本地的备份文件，然后启动oracle数据库服务，居然拉起来了。原来这tmd就叫惊喜╰(°▽°)╯
参考链接：https://blog.csdn.net/weixin_39133690/article/details/102799091