搜索到
2
篇与
MongoDB
的结果
-
Mongo错误修复 - WiredTiger metadata corruption detected 免责声明本文提供的所有信息仅供参考，作者不对使用本文所含信息而导致的任何直接或间接损失负责。在进行MongoDB数据恢复操作之前，请务必备份您的数据并仔细阅读MongoDB官方文档。数据恢复操作具有风险，可能导致数据的永久丢失或损坏。作者建议在执行任何数据恢复步骤之前，先在测试环境中进行尝试，并咨询有经验的数据库管理员或技术专家。背景系统: macOS 13.6Mongo: 5.0.3 (Docker 容器部署)原因: 由于Docker Desktop在Mac上运行消耗资源非常高，所以又安装了OrbStack进行docker管理。 在OrbStack运行时，意外的打开了Docker Desktop（噩耗的开始）。 刚开始没在意，中午吃饭发现系统刷新失败了！？ 果然还是出问题了。782c0bd0dfb4 mongo:5.0.3 "docker-entrypoint.s…" 6 weeks ago Restarting (14) 46 seconds ago mongodbmongodb的容器一直在重启，于是看看log:{"t":{"$date":"2024-08-01T06:15:22.352+00:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}} {"t":{"$date":"2024-08-01T06:15:22.352+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"} {"t":{"$date":"2024-08-01T06:15:22.354+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"} {"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."} {"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"} {"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}} {"t":{"$date":"2024-08-01T06:15:22.355+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}} {"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"2f51c27fb1c7"}} {"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.3","gitVersion":"657fea5a61a74d7a79df7aff8e4bcf0bc742b748","openSSLVersion":"OpenSSL 1.1.1f 31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}} {"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}} {"t":{"$date":"2024-08-01T06:15:22.356+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}} {"t":{"$date":"2024-08-01T06:15:22.357+00:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}} {"t":{"$date":"2024-08-01T06:15:22.357+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3455M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}} {"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776094][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 637: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776189][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 643: WiredTiger has failed to open its metadata: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776210][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 646: This may be due to the database files being encrypted, being from an older version or due to corruption on disk: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.776+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:776232][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 649: You should confirm that you have opened the database with the correct options including all encryption and compression options: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783621][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 637: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783671][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 643: WiredTiger has failed to open its metadata: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783683][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 646: This may be due to the database files being encrypted, being from an older version or due to corruption on disk: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.783+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:783688][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 649: You should confirm that you have opened the database with the correct options including all encryption and compression options: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790000][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 637: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790045][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 643: WiredTiger has failed to open its metadata: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790057][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 646: This may be due to the database files being encrypted, being from an older version or due to corruption on disk: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.790+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1722492922:790063][1:0x7f18c7605c80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 649: You should confirm that you have opened the database with the correct options including all encryption and compression options: WT_ERROR: non-specific WiredTiger error"}} {"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"W", "c":"STORAGE", "id":22347, "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."} {"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"W", "c":"STORAGE", "id":22348, "ctx":"initandlisten","msg":"WiredTiger metadata corruption detected"} {"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"F", "c":"STORAGE", "id":50944, "ctx":"initandlisten","msg":"Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair"} {"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50944,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":685}} {"t":{"$date":"2024-08-01T06:15:22.791+00:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"} 关键信息： Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade. WiredTiger metadata corruption detected Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair1、"WiredTiger metadata corruption detected" 原因清楚了，存储引擎的元数据坏掉了。在 MongoDB 中，WiredTiger 是一个用于存储引擎的名字。WiredTiger 引擎提供了高效的数据压缩和并发处理能力，从 MongoDB 3.2 版本开始成为默认的存储引擎。WiredTiger 文件实际上包含了数据文件和索引文件。下面是一些有关 WiredTiger 文件的关键点： 存储引擎：WiredTiger 是 MongoDB 的存储引擎，它负责管理数据库的数据存储、索引、并发控制和缓存等方面。 数据文件：在 MongoDB 数据库目录中，WiredTiger 存储引擎会创建一系列文件，用于存储数据和索引。这些文件包括 .wt 后缀的文件。 配置文件：WiredTiger.wt 是一个主要的配置文件，它包含了有关存储引擎配置和元数据的信息。 日志文件：WiredTiger 还使用一组日志文件（如 WiredTigerLog 开头的文件）来进行写入操作的日志记录，以便在发生崩溃时进行数据恢复。 压缩和并发：WiredTiger 提供了数据压缩功能，可以显著减少存储空间需求。此外，它还支持多线程并发写入，从而提高数据库性能。 简而言之，WiredTiger 是 MongoDB 用于管理和存储数据的核心组件，提供了高效的数据存储和管理能力。2、"http://dochub.mongodb.org/core/repair"问题发生时系统没有数据写入，所以我们数据应该没问题，根据提示我们进行修复:mongod --dbpath /data/db --repair修复由于在容器中部署的Mongo，不可能在这个容器中进行修复，所以重新起一个同版本的容器进行修复。步骤：备份数据确定Mongo 版本启动修复Mongo的容器"fix-mongo"拷贝数据到"fix-mongo"并修复从"fix-mongo"导出修复好的数据数据备份zip -r bad_mongodata_yyyyMMdd.zip /your/mount/path/mongodata确定Mongo版本 docker inspect your_container |grep 'MONGO_VERSION' "MONGO_VERSION=5.0.3"我这里是"5.0.3"启动修复Mongo的容器"fix-mongo"docker run --name fix-mongo -d mongo:your_mongo_version拷贝数据到"fix-mongo"并修复将整个数据目录拷贝到"fix-mongo"的"/tmp"## 拷贝数据到容器 docker cp /your/mount/path/mongodata fix-mongo:/tmp ## 修复 - 等待结束 mongod --dbpath /tmp/mongodata --repair ## 修复的部分log ... {"t":{"$date":"2024-08-01T06:47:54.435+00:00"},"s":"I", "c":"STORAGE", "id":22319, "ctx":"initandlisten","msg":"Finished shutting down session sweeper thread"} {"t":{"$date":"2024-08-01T06:47:54.451+00:00"},"s":"I", "c":"STORAGE", "id":4795902, "ctx":"initandlisten","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,"}} {"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I", "c":"STORAGE", "id":4795901, "ctx":"initandlisten","msg":"WiredTiger closed","attr":{"durationMillis":242}} {"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I", "c":"STORAGE", "id":22279, "ctx":"initandlisten","msg":"shutdown: removing fs lock..."} {"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I", "c":"-", "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"} {"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I", "c":"FTDC", "id":20626, "ctx":"initandlisten","msg":"Shutting down full-time diagnostic data capture"} {"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I", "c":"CONTROL", "id":20565, "ctx":"initandlisten","msg":"Now exiting"} {"t":{"$date":"2024-08-01T06:47:54.693+00:00"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":0}} ## 退出容器 exit从"fix-mongo"导出修复好的数据## 移除修复前的数据，确保已做好数据备份 rm -rf /your/mount/path/mongodata ## 取回修复好的数据 docker cp fix-mongo:/tmp/mongodata /your/mount/path/mongodata ## 重启你的容器 docker restart your_container结束！数据勤备份、数据勤备份、数据勤备份 -
Mongo 关联表查询 摘要Mongo支持与Mysql类似的关联表查询，但比较麻烦。通过与Mysql实现进行对比，便于理解Mongo如何实现。MysqlOrder表关联Product表的id, 实现查询订单支持搜索产品名称。# Order CREATE TABLE `order` ( `id` bigint(20) DEFAULT NULL, `no` varchar(255) DEFAULT NULL, `amount` int(11) DEFAULT NULL COMMENT '单位- 分\n', `productId` bigint(20) DEFAULT NULL, `count` int(11) DEFAULT NULL, `userId` bigint(20) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; -- ---------------------------- -- Records of product -- ---------------------------- INSERT INTO `order` (`id`, `no`, `amount`, `productId`, `count`, `userId`) VALUES (1, '2024062701', 10000, 1, 1, 1); INSERT INTO `order` (`id`, `no`, `amount`, `productId`, `count`, `userId`) VALUES (1, '2024062702', 20000, 1, 2, 1); # Product CREATE TABLE `product` ( `id` bigint(20) DEFAULT NULL, `name` varchar(255) DEFAULT NULL, `price` int(11) DEFAULT NULL COMMENT '单位-分' ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; -- ---------------------------- -- Records of product -- ---------------------------- INSERT INTO `product` (`id`, `name`, `price`) VALUES (1, '测试产品', 10000); 关联查询查询订单、产品名称，通过left join搜索产品名称SELECT o,p.name from `order` as o left JOIN `product` as p on o.productId = p.id where p.name like '%测试%'; ## 结果 #id no amount productId count userId name 1 2024062701 10000 1 1 1 测试产品 1 2024062702 20000 1 2 1 测试产品Mongo同样在mongo中准备Order和Product的集合## insert to product db.product.insertOne({ name: '测试产品', price: 10000 }); ## result { "_id" : ObjectId("667d7fbe04cfea555206351f"), "name" : "测试产品", "price" : 10000 } ## insert to orders db.order.insertMany([ { no: '2024062701', amount: 10000, productId: "667d7fbe04cfea555206351f", count: 1, userId: 1 }, { no: '2024062702', amount: 20000, productId: "667d7fbe04cfea555206351f", count: 2, userId: 1 } ]); # result { "_id" : ObjectId("667d80b804cfea5552063520"), "no" : "2024062701", "amount" : 10000, "productId" : "667d7fbe04cfea555206351f", "count" : 1, "userId" : 1 } { "_id" : ObjectId("667d80b804cfea5552063521"), "no" : "2024062702", "amount" : 20000, "productId" : "667d7fbe04cfea555206351f", "count" : 2, "userId" : 1 }关联查询使用Mongo提供的$lookup来实现关联查询{ "$lookup": { "from": { "connectionName": "<registered-atlas-connection>", "db": "<registered-database-name>", "coll": "<atlas-collection-name>" }, "localField": "<field-in-source-messages>", "foreignField": "<field-in-from-collection>", "as": "<output-array-field>" } }那么我们使用Mongo查询，应该这样写db.order.aggregate( { $lookup: { from: "product", # 需要关联的product表 localField: "proudctId", # order中的productId foreignField: '_id', # 与proudct中的_id关联 as: "p" # product的别名，出现在查询结果中，默认是一个数组 } }, { $unwind: "$p" }, # 将p拉平，如果p是多个，那么会出现多条记录 # 此时得到 order + p:{} { $match: { "p.name": { # 过滤p.name $regex: ".*测试.*", # 使用正则，模糊查询= '%测试%' $options: "i" # 忽略大小写 } } }) 但是，没有查询到结果！？因为 在order中productId是String,而product._id是ObjectId,他俩不是同一个类型，所以是不等的。解决他俩类型不一样，那么把他俩类型弄成一样再比较！db.order.aggregate( { $lookup: { from: "product", let: { pid: "$productId" }, ## 取order.productId到变量pid中 pipeline: [ ## 在product表操作，将pid变成ObjectId后与order._id进行等于比较 { $match: { $expr: { $eq: ["$_id", { $toObjectId: "$$pid" }] } } } ], as: "p" } }, { $unwind: "$p" }, { $match: { "p.name": { $regex: ".*测试.*", $options: "i" } } }) ## Result // collection: order { "_id" : ObjectId("667d80b804cfea5552063520"), "no" : "2024062701", "amount" : 10000, "productId" : "667d7fbe04cfea555206351f", "count" : 1, "userId" : 1, "p" : { "_id" : ObjectId("667d7fbe04cfea555206351f"), "name" : "测试产品", "price" : 10000 } } // collection: order { "_id" : ObjectId("667d80b804cfea5552063521"), "no" : "2024062702", "amount" : 20000, "productId" : "667d7fbe04cfea555206351f", "count" : 2, "userId" : 1, "p" : { "_id" : ObjectId("667d7fbe04cfea555206351f"), "name" : "测试产品", "price" : 10000 } } lookup 解释1、将order.productId声明一个新的变量：pid2、将order._id与 pid比较，此时的pid转换成了obejctIdAI 解释这段代码是MongoDB聚合管道($aggregate)中的一部分，使用了$lookup阶段来实现集合之间的关联查询。这里是将当前集合（假设为订单或其他与产品相关的集合）与"product"集合进行关联，以获取与每个文档中productId字段对应的产品详细信息。我将逐步解释每一部分的含义： $lookup: 这是MongoDB聚合管道的一个操作符，用于在当前集合的文档中连接来自另一个集合的文档。它基于指定的键执行类似SQL中的左外连接。 - from: "product": 指定要连接的另一个集合名为"product"。这是你想要获取额外数据的集合。 - let: { pid: "$productId" }: 定义了一个变量pid，其值为当前文档中的productId字段。这在后续的pipeline中作为查找依据。 - pipeline: 这是一个数组，包含了在"product"集合上执行的聚合管道操作。这里仅有一个操作： - $match: 用于过滤"product"集合中的文档。它只选择那些满足特定条件的文档。 - $expr: 允许在$match中使用聚合表达式。 - $eq: 检查两个表达式是否相等。在这个例子中，检查"product"集合文档的_id字段是否等于通过$toObjectId转换后的$$pid（即来自外部文档的productId，已经转换为ObjectId类型）。 - as: "p": 指定将连接结果存储在当前文档的新字段p中，作为数组形式存在，包含匹配到的所有"product"集合的文档。 Spring Data Mongo Tempalte 实现 Pageable pageable= "Spring data Pageable'; String name= "测试" // Template不支持let,所以自己实现 AggregationOperation l = context -> new Document("$lookup", new Document("from", "product") .append("let", new Document("pid", "$projectId")) .append("pipeline", Arrays.asList( new Document("$match", new Document("$expr", new Document("$eq", Arrays.asList("$_id", new Document("$toObjectId", "$$pid")))) ) )) .append("as", "p")); UnwindOperation unwindOperation = Aggregation.unwind("p"); Criteria c = new Criteria(); if (StringUtils.isNotBlank(name)) { c.and("p.name").regex(".*" + name + ".*", "i"); } Aggregation aggregation = Aggregation.newAggregation( l, unwindOperation, Aggregation.match(c), Aggregation.count().as("c") ); Document count = MongoUtils.template(dbName).aggregate(aggregation, Order.class, Document.class) .getUniqueMappedResult(); // 没有搜索到结果，这里的count会是null if (count == null) { return PageResult.empty(); } Aggregation pageAggregation = Aggregation.newAggregation( // 这有顺序关联： 先跳过，在limit l, unwindOperation, Aggregation.match(c), Aggregation.skip(pageable.getOffset()), Aggregation.limit(pageable.getPageSize()) ); List<Order> data = MongoUtils.template(dbName).aggregate(pageAggregation, Order.class, Order.class) .getMappedResults(); return Page.of(count.getInteger("c"), data);
