OceanBase 源码学习: 3.4 高可用(HA)与数据恢复系统
P
digoal约 11 分钟
digoal约 11 分钟OceanBase 源码学习: 3.4 高可用(HA)与数据恢复系统
本文介绍 OceanBase 的高可用性 (HA) 和数据恢复系统，该系统提供日志流(LS)迁移、Tablet group 恢复、物理数据复制以及全面的调试(debug)基础架构(Infra)等机制。这些系统通过基于 DAG(有向无环图) 的协调工作流来处理跨分布式集群的迁移、恢复和传输操作，从而确保数据的持久性和可用性。
HA 系统主要包括这几点: 以 ObStorageHATabletsBuilder 创建/更新 Tablet、ObMigrationDagNet 迁移工作流、ObTabletGroupRestoreDagNet 实现恢复操作以及用于测试和验证的广泛调试(debug)同步点。
系统架构
OceanBase 的 HA 和数据恢复系统围绕基于 DAG 的工作流构建，通过专门的管理器类协调 tablet 的构建、迁移和恢复操作。
HA核心架构
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L113-L142 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L228-L304 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L267-L404 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L24-L764
关键组件
ObStorageHATabletsBuilder | create_or_update_tablets()build_tablets_sstable_info() | ||
ObMigrationDagNet | start_running_for_migration_()clear_dag_net_ctx() | ||
ObTabletGroupRestoreDagNet | start_running_for_restore_()init_by_param() | ||
ObPhysicalCopyTask | fetch_macro_block_with_retry_()process() | ||
ObBackupSetTaskMgr | process()backup_sys_meta_() | ||
Debug Sync Framework | DEBUG_SYNC() |
传输(Transfer)操作
传输操作在同一集群内的日志流之间移动 Tablet，由 ObTransferHandler 管理。
传输处理程序状态机
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_transfer_handler.cpp#L441-L576 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_transfer_handler.cpp#L615-L749
传输流程实现
ObTransferHandler 协调程序通过以下主要方法完成 Tablet 传输：
start_transfer() | INIT | |
process_start_transfer_() | abort_transfer() | |
process_doing_transfer_() | ||
finish_transfer() | ||
abort_transfer() |
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_transfer_handler.cpp#L441-L502 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_transfer_handler.cpp#L577-L614
恢复过程
恢复过程遵循由 ObLSRestoreHandler 管理的明确定义的状态机。
Tablet group 恢复状态机
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L407-L456 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L458-L502 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L549-L582
Tablet 恢复上下文管理
恢复过程通过跟踪 Tablet 状态和恢复进度的专门类来管理上下文：
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L65-L145 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L147-L246 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L314-L404
备份集成和数据源
恢复系统与备份基础架构(backup Infra)集成，从外部存储中获取 Tablet 元数据和数据：
ObBackupMetaIndexStoreWrapper | init()get_backup_meta_index() | |
ObCopyTabletInfoRestoreReader | init()fetch_tablet_info() | |
ObCopySSTableInfoRestoreReader | get_next_tablet_sstable_header() | |
ObCopyMacroBlockRestoreReader | get_next_macro_block() |
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L358-L374 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L485-L514 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L725-L771
迁移过程
OceanBase 通过 ObLSMigrationHandler 和 DAG 框架在 OceanBase 服务器之间迁移整个日志流(LogStream)。
迁移操作类型
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_struct.cpp#L27-L64 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L273-L305 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L320-L383
迁移上下文和状态管理
ObMigrationCtx 跟踪迁移状态并提供跨 DAG 操作的上下文：
arg_ | ls_id、type、src、dst） | ObMigrationOpArg |
local_clog_checkpoint_scn_ | Palf | |
src_ls_rebuild_seq_ | ||
ha_table_info_mgr_ | ObStorageHATableInfoMgr | |
tablet_group_mgr_ | ObStorageHATabletGroupMgr |
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L41-L134 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L288-L302
迁移期间的 Tablet 构建
ObStorageHATabletsBuilder 处理迁移操作期间的 tablet 创建和更新：
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L144-L202 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L568-L613 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L463-L512
重建(rebuild)服务
ObRebuildService 检测并协调整个集群的数据不一致。
重建检测和处理
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_rebuild_service.cpp#L27-L91 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_rebuild_service.cpp#L596-L677 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ls/ob_ls.cpp#L597-L604
物理复制机制
物理复制使用 ObPhysicalCopyTask 在存储位置之间传输宏块。
复制任务架构
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_physical_copy_task.cpp#L82-L143 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_physical_copy_task.cpp#L146-L178 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_reader.cpp#L31-L114
复制任务重试逻辑
复制过程通过重试机制实现了强大的错误处理：
fetch_macro_block_with_retry_() | |||
get_remote_logical_sstable_meta_() | |||
read_backup_macro_block_data_() |
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_physical_copy_task.cpp#L146-L178 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_reader.cpp#L180-L237
存储 RPC 接口
HA 操作依赖 ObStorageRpc 进行远程数据访问和协调。
存储 RPC 操作
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.h#L1482-L1527 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.cpp#L4890-L4954 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.cpp#L5044-L5108
RPC 参数结构
HA 操作中使用的关键 RPC 参数结构(struct)：
copy_tablet_info | ObCopyTabletInfoArg | tenant_id_tablet_id_list_，need_check_seq_，ls_id_ |
copy_macro_block | ObCopyMacroBlockListArg | tenant_id_table_key_，arg_list_，ls_id_ |
fetch_ls_meta_info | ObFetchLSMetaInfoArg | tenant_id_version_，ls_id_ |
notify_restore_tablets | ObNotifyRestoreTabletsArg | tenant_id_tablet_id_array_，restore_status_，ls_id_ |
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.h#L233-L266 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.h#L92-L133 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.h#L460-L481 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/ob_storage_rpc.h#L692-L714
错误处理和恢复
HA 操作通过多层实现全面的错误处理。
HA 上下文错误管理
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_struct.cpp#L692-L759 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L120-L134
迁移状态转换
迁移状态遵循 ObMigrationStatusHelper 中明确定义的故障状态转换：
OB_MIGRATION_STATUS_ADD | OB_MIGRATION_STATUS_ADD_FAIL | |
OB_MIGRATION_STATUS_MIGRATE | OB_MIGRATION_STATUS_MIGRATE_FAIL | |
OB_MIGRATION_STATUS_REBUILD | OB_MIGRATION_STATUS_REBUILD_FAIL | |
OB_MIGRATION_STATUS_CHANGE | OB_MIGRATION_STATUS_NONE |
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_struct.cpp#L176-L236 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_struct.cpp#L238-L305
Tablet 状态恢复
在 HA 操作期间，tablet 可能处于需要恢复的不一致状态：
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L370-L460 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_tablet_group_restore.cpp#L28-L54
调试同步基础设施(Debug Synchronization Infrastructure)
OceanBase 提供了 764 个调试同步点，用于对 HA 操作进行全面的测试和调试，以 ObDebugSyncPoint 枚举形式定义。
调试同步点类别, 参考代码: src/share/ob_debug_sync_point.h
这部分可参考: 《何为MySQL 调试同步基础设施(Debug Synchronization Infrastructure)?》
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L66-L71 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L277-L287 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L465-L582 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L416-L438 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L696-L702
HA 操作中的调试同步使用情况
调试同步系统在整个 HA 代码中被广泛用于测试特定场景：
AFTER_MIGRATION_FETCH_TABLET_INFO | ||
BEFORE_PREPARE_MIGRATION_TASK | ||
BEFORE_RESTORE_START | ||
BEFORE_BACKUP_META | ||
START_TRANSFER_TRANS |
调试同步实现模式(pattern)
调试同步基础设施在 HA 操作中遵循一致的模式(pattern)：
调试同步给 HA 测试带来的好处
广泛的调试同步可实现复杂的测试场景：
故障注入：在特定点暂停以注入故障并测试错误处理 竞争条件测试：控制并发 HA 操作之间的时间 状态验证：在复杂操作期间暂停执行以检查中间状态 性能分析：测量迁移/恢复工作流程中各个阶段的延迟 集成测试：协调不同 HA 子系统之间的时间
源文件:
https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_storage_ha_tablet_builder.cpp#L186-L192 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/storage/high_availability/ob_ls_migration.cpp#L706-L710 https://github.com/oceanbase/oceanbase/blob/8e2580cf/src/share/ob_debug_sync_point.h#L24-L764
小结
OceanBase 的高可用性和数据恢复系统提供了一个全面的框架，用于确保数据的持久性和可用性。该系统采用状态驱动的方法，并通过有向无环图 (DAG) 管理分阶段操作，以处理复杂的恢复和迁移场景。关键机制包括：
tablet 级修复，优先修复系统 tablet 具有重试机制的物理块复制 用于高效组织数据的表存储管理 使用智能重试策略进行错误处理
这些机制协同工作，即使在遇到硬件故障或其他灾难性事件时也能提供强大的数据保护。
更多详细内容请关注我的github: https://github.com/digoal/blog