Mongo错误修复 - WiredTiger metadata corruption detected

免责声明

本文提供的所有信息仅供参考，作者不对使用本文所含信息而导致的任何直接或间接损失负责。在进行MongoDB数据恢复操作之前，请务必备份您的数据并仔细阅读MongoDB官方文档。数据恢复操作具有风险，可能导致数据的永久丢失或损坏。作者建议在执行任何数据恢复步骤之前，先在测试环境中进行尝试，并咨询有经验的数据库管理员或技术专家。

背景

系统: macOS 13.6
Mongo: 5.0.3 (Docker 容器部署)

原因: 由于Docker Desktop在Mac上运行消耗资源非常高，所以又安装了OrbStack进行docker管理。在OrbStack运行时，意外的打开了Docker Desktop（噩耗的开始）。刚开始没在意，中午吃饭发现系统刷新失败了！？果然还是出问题了。

782c0bd0dfb4   mongo:5.0.3   "docker-entrypoint.s…"   6 weeks ago    Restarting (14) 46 seconds ago     mongodb

mongodb的容器一直在重启，于是看看log:

{"t":{"$date":"2024-08-01T06:15:22.352+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2024-08-01T06:15:22.352+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2024-08-01T06:15:22.354+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"2f51c27fb1c7"}}
{"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.3","gitVersion":"657fea5a61a74d7a79df7aff8e4bcf0bc742b748","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
{"t":{"$date":"2024-08-01T06:15:22.357+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2024-08-01T06:15:22.357+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3455M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776094][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 637: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776189][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 643: WiredTiger has failed to open its metadata: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776210][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 646: This may be due to the database files being encrypted, being from an older version or due to corruption on disk: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776232][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 649: You should confirm that you have opened the database with the correct options including all encryption and compression options: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783621][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 637: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783671][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 643: WiredTiger has failed to open its metadata: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783683][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 646: This may be due to the database files being encrypted, being from an older version or due to corruption on disk: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783688][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 649: You should confirm that you have opened the database with the correct options including all encryption and compression options: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790000][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 637: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790045][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 643: WiredTiger has failed to open its metadata: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790057][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 646: This may be due to the database files being encrypted, being from an older version or due to corruption on disk: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790063][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 649: You should confirm that you have opened the database with the correct options including all encryption and compression options: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"W",  "c":"STORAGE",  "id":22348,   "ctx":"initandlisten","msg":"WiredTiger metadata corruption detected"}
{"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"F",  "c":"STORAGE",  "id":50944,   "ctx":"initandlisten","msg":"Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair"}
{"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50944,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":685}}
{"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

关键信息：

 Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade.
 
 WiredTiger metadata corruption detected
 
 Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair

1、"WiredTiger metadata corruption detected"
原因清楚了，存储引擎的元数据坏掉了。

在 MongoDB 中，WiredTiger 是一个用于存储引擎的名字。WiredTiger 引擎提供了高效的数据压缩和并发处理能力，从 MongoDB 3.2 版本开始成为默认的存储引擎。WiredTiger 文件实际上包含了数据文件和索引文件。下面是一些有关 WiredTiger 文件的关键点：

存储引擎：WiredTiger 是 MongoDB 的存储引擎，它负责管理数据库的数据存储、索引、并发控制和缓存等方面。

数据文件：在 MongoDB 数据库目录中，WiredTiger 存储引擎会创建一系列文件，用于存储数据和索引。这些文件包括 .wt 后缀的文件。

配置文件：WiredTiger.wt 是一个主要的配置文件，它包含了有关存储引擎配置和元数据的信息。

日志文件：WiredTiger 还使用一组日志文件（如 WiredTigerLog 开头的文件）来进行写入操作的日志记录，以便在发生崩溃时进行数据恢复。

压缩和并发：WiredTiger 提供了数据压缩功能，可以显著减少存储空间需求。此外，它还支持多线程并发写入，从而提高数据库性能。

简而言之，WiredTiger 是 MongoDB 用于管理和存储数据的核心组件，提供了高效的数据存储和管理能力。

2、"http://dochub.mongodb.org/core/repair"
问题发生时系统没有数据写入，所以我们数据应该没问题，根据提示我们进行修复:

mongod --dbpath /data/db --repair

修复

由于在容器中部署的Mongo，不可能在这个容器中进行修复，所以重新起一个同版本的容器进行修复。

步骤：

备份数据
确定Mongo 版本
启动修复Mongo的容器"fix-mongo"
拷贝数据到"fix-mongo"并修复
从"fix-mongo"导出修复好的数据

数据备份

zip -r bad_mongodata_yyyyMMdd.zip /your/mount/path/mongodata

确定Mongo版本

 docker inspect  your_container |grep 'MONGO_VERSION'
 
  "MONGO_VERSION=5.0.3"

我这里是"5.0.3"

启动修复Mongo的容器"fix-mongo"

docker run --name fix-mongo -d mongo:your_mongo_version

拷贝数据到"fix-mongo"并修复

将整个数据目录拷贝到"fix-mongo"的"/tmp"

## 拷贝数据到容器
docker cp /your/mount/path/mongodata fix-mongo:/tmp

## 修复 - 等待结束
mongod --dbpath /tmp/mongodata --repair

## 修复的部分log
...
{"t":{"$date":"2024-08-01T06:47:54.435+00:00"},"s":"I",  "c":"STORAGE",  "id":22319,   "ctx":"initandlisten","msg":"Finished shutting down session sweeper thread"}
{"t":{"$date":"2024-08-01T06:47:54.451+00:00"},"s":"I",  "c":"STORAGE",  "id":4795902, "ctx":"initandlisten","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,"}}
{"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I",  "c":"STORAGE",  "id":4795901, "ctx":"initandlisten","msg":"WiredTiger closed","attr":{"durationMillis":242}}
{"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I",  "c":"STORAGE",  "id":22279,   "ctx":"initandlisten","msg":"shutdown: removing fs lock..."}
{"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I",  "c":"FTDC",     "id":20626,   "ctx":"initandlisten","msg":"Shutting down full-time diagnostic data capture"}
{"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":0}}


## 退出容器
exit

从"fix-mongo"导出修复好的数据

## 移除修复前的数据，确保已做好数据备份
rm -rf /your/mount/path/mongodata

## 取回修复好的数据
docker cp fix-mongo:/tmp/mongodata /your/mount/path/mongodata

## 重启你的容器

docker restart your_container

结束！