MongoDB
| 功能 | 说明 |
|---|---|
全量数据迁移 | 逻辑迁移，通过顺序扫描表数据，将数据分批写入到对端数据库，支持的 _id 类型为 ObjectId、Long、Integer |
增量实时同步 | 支持 INSERT、UPDATE、DELETE 同步 |
数据校验和订正 | 全量数据校验，并可选根据校验结果订正差异数据，支持定时，文档：创建定时校验订正任务 |
修改订阅 | 新增、删除、修 改订阅表，支持历史数据迁移，文档：修改订阅 |
增量位点回溯 | 支持按照 时间戳 回溯位点，重新消费过去一段时间的 oplog |
部署形态支持 | 支持 主备、副本集、分片集群 |
限制和注意点
| 限制项 | 说明 |
|---|---|
Oplog Size and Retention Settings | By default, the value of replication.oplogSizeMB or storage.oplogMinRetentionHours in MongoDB is too small. If data synchronization latency is significant, unconsumed oplogs may be removed. In this case, it is necessary to increase these parameters. |
Parameter Configuration for MongoDB Master-Slave Architecture | For MongoDB master-slave architecture, set the Source parameter oplogCollection to oplog.$main. |
ChangeStream Mode | MongoDB 3.6 and above support changeStream for capturing incremental data changes. Set the Source parameter captureMode to CHANGE_STREAM. For sharded clusters, use the MongoDB connection string for synchronization. |
Oplog Mode | When using oplog mode for data synchronization from a MongoDB instance, ensure the access to the local database. |
任务参数
| 参数名称 | 说明 |
|---|---|
captureMode | Configure the MongoDB incremental data sync mode, supporting OP_LOG and CHANGE_STREAM modes. |
changeStreamBatchSize | Set the maximum number of change events per batch for MongoDB Change Stream. |
oplogCollection | Specify the collection name for MongoDB oplog. The default name is oplog.rs. |
timezone | Source time zone (the default time zone is UTC). |
Tips: 通用参数配置请参考 通用参数及功能
前置条件
| 条件 | 说明 |
|---|---|
Permissions for Account | |
Port Preparation | Allow the migration and sync node (Worker) to connect to the MongoDB ports. |
任务参数
| 参数名称 | 说明 |
|---|---|
totalDataInMemMb | Maximum data size allowed in memory when writing in batches; If the data size exceeds the memory limit, or the wait time exceeds asyncFlushIntervalSec, then data is flushed to the write queue. |
asyncFlushIntervalSec | Interval to wait for flushing when writing in batches; If the wait time exceeds asyncFlushIntervalSec, or the data size exceeds totalDataInMemMb, then data is flushed to the write queue. |
flushBatchMb | Maximum batch size per table; If the batch size exceeds this limit, then data is flushed to the write queue. |
realFlushPauseSec | Wait time to flush data to MongoDB. 0 means no wait is needed. |
enableTimeZoneProcess | Enable time zone conversion for time fields. |
timezone | Timezone in the Target (by default, UTC). |
enableBatchApply | Enable batch write mode (it is disabled by default). |
Tips: 通用参数配置请参考 通用参数及功能