穷鬼还想搭PolarDB RAC？结局显而易见
digoal约 26 分钟参考文档点击文末阅读原文打开; 推荐《最好的PostgreSQL学习镜像》;
穷鬼福音: 笔记本搭建PolarDB RAC
为了让穷鬼爱好者能在笔记本装上PolarDB RAC环境, 我最近在把弄这样的方法: 把宿主机块设备共享给多个docker容器, 可以用来构建PolarDB RAC一写多读集群. 但是macOS不支持, 前面文章有详细介绍:
《在 macOS 中将文件模拟为块设备》 《在 macOS 中用docker --device挂载块设备报错》
本文将换一种方式, 把文件放在宿主机, 在容器内使用loop设备把文件模拟为块设备. 这个方法不管你是window,macOS还是Linux都可以适用. 真正的穷鬼爱好者福音.
DEMO
使用polardb_pg_devel:ubuntu22.04 镜像, 自带pfs, 无需再安装pfs.
1、在宿主机(macOS)中生成1个文件(以下文件20GB), 后面在两个容器中将使用这个文件作为loop块设备.
mkdir ~/data_volumn
cd ~/data_volumn
dd if=/dev/zero of=./VirtualDisk.img bs=1m count=20480 oflag=direct
2、启动容器pb1
cd ~/data_volumn
PWD=`pwd`
docker run -d -it -v $PWD:/data -P --shm-size=1g --cap-add=SYS_PTRACE --cap-add SYS_ADMIN --privileged=true --name pb1 registry.cn-hangzhou.aliyuncs.com/polardb_pg/polardb_pg_devel:ubuntu22.04 bash
后面要确保2个容器的网络可以互通
进入容器pb1
docker exec -ti pb1 bash ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
link/tunnel6 :: brd :: permaddr ca4f:9c91:a9ad::
104: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
安装ping 客户端
sudo apt update
sudo apt-get install -y inetutils-ping
等2个容器都启动后检查确认网络可以互通
172.17.0.2
ping 172.17.0.3
64 bytes from 172.17.0.3: icmp_seq=0 ttl=64 time=0.050 ms
3、启动容器pb2
cd ~/data_volumn
PWD=`pwd`
docker run -d -it -v $PWD:/data -P --shm-size=1g --cap-add=SYS_PTRACE --cap-add SYS_ADMIN --privileged=true --name pb2 registry.cn-hangzhou.aliyuncs.com/polardb_pg/polardb_pg_devel:ubuntu22.04 bash
确保2个容器的网络可以互通
进入容器pb2
docker exec -ti pb2 bash ip addr show
172.17.0.3
安装ping 客户端
sudo apt update
sudo apt-get install -y inetutils-ping
等2个容器都启动后检查确认网络可以互通
ping 172.17.0.2
64 bytes from 172.17.0.2: icmp_seq=0 ttl=64 time=0.050 ms
4、在pb1中使用共享的VirtualDisk.img文件创建loop设备, 需要找到一个空闲的loop设备, 从loop0开始找, 逐渐增加数字. 直到创建loopN成功.
postgres@e065fac6f0b4:~$ sudo losetup /dev/loop0 /data/VirtualDisk.img
losetup: /data/VirtualDisk.img: failed to set up loop device: Device or resource busy 或者可以使用 losetup -f 找第一个未使用的loop设备.
$ losetup -f
/dev/loop1
postgres@e065fac6f0b4:~$ sudo losetup /dev/loop1 /data/VirtualDisk.img
postgres@e065fac6f0b4:~$ ll /dev/loop1
brw-rw-rw- 1 root root 7, 1 Dec 16 13:52 /dev/loop1
将loop设备软链到nvme1n1方便使用, 如果你的容器内已经有这个名字, 可以换一个例如nvme2n1.
sudo ln -s /dev/loop1 /dev/nvme1n1
losetup用法:
$ losetup --helpUsage:
losetup [options] [<loopdev>]
losetup [options] -f | <loopdev> <file>
Set up and control loop devices.
Options:
-a, --all list all used devices
-d, --detach <loopdev>... detach one or more devices
-D, --detach-all detach all used devices
-f, --find find first unused device
-c, --set-capacity <loopdev> resize the device
-j, --associated <file> list all devices associated with <file>
-L, --nooverlap avoid possible conflict between devices
-o, --offset <num> start at offset <num> into file
--sizelimit <num> device is limited to <num> bytes of the file
-b, --sector-size <num> set the logical sector size to <num>
-P, --partscan create a partitioned loop device
-r, --read-only set up a read-only loop device
--direct-io[=<on|off>] open backing file with O_DIRECT
--show print device name after setup (with -f)
-v, --verbose verbose mode
-J, --json use JSON --list output format
-l, --list list info about all or specified (default)
-n, --noheadings don't print headings for --list output
-O, --output <cols> specify columns to output for --list
--output-all output all columns
--raw use raw --list output format
-h, --help display this help
-V, --version display version
Available output columns:
NAME loop device name
AUTOCLEAR autoclear flag set
BACK-FILE device backing file
BACK-INO backing file inode number
BACK-MAJ:MIN backing file major:minor device number
MAJ:MIN loop device major:minor number
OFFSET offset from the beginning
PARTSCAN partscan flag set
RO read-only device
SIZELIMIT size limit of the file in bytes
DIO access backing file with direct-io
LOG-SEC logical sector size in bytes
For more details see losetup(8).
可以跳过步骤 : 5.1 和 5.2
5.1、在pb2中使用共享的VirtualDisk.img文件创建loop设备, 需要找到一个空闲的loop设备, 从loop0开始找, 逐渐增加数字. 直到创建loopN成功.
注意, 在pb1创建的loop设备, 在pb2能看到, 并且在pb2不能再使用这个loop1名字来进行创建(已占用)
lsblk|grep loop
loop0 7:0 0 658.4M 1 loop
loop1 7:1 0 20G 0 loop // pb1创建的
loop2 7:2 0 20G 0 loop // pb2创建的
loop3 7:3 0 40G 0 loop // pb1创建的
loop4 7:4 0 40G 0 loop // pb2创建的
在pb1创建的loop设备, 在pb2可以直接使用. 猜测可能是macOS中docker底层基于HyperKit有关. 参看:《在 macOS 中用docker --device挂载块设备报错》
postgres@a73af821619f:~$ sudo losetup /dev/loop1 /data/VirtualDisk.img
losetup: /data/VirtualDisk.img: failed to set up loop device: Device or resource busy
postgres@a73af821619f:~$ sudo losetup /dev/loop0 /data/VirtualDisk.img
losetup: /data/VirtualDisk.img: failed to set up loop device: Device or resource busy
postgres@a73af821619f:~$ sudo losetup /dev/loop2 /data/VirtualDisk.img
postgres@a73af821619f:~$ ll /dev/loop2
brw-rw-rw- 1 root root 7, 2 Dec 16 13:57 /dev/loop2
5.2、将loop2设备软链到nvme1n1方便使用, 如果你的容器内已经有这个名字, 可以换一个例如nvme2n1.
sudo ln -s /dev/loop2 /dev/nvme1n1
5.3、由于我这个环境任何一个容器创建了loop设备其他容器都能看到, 所以pb2直接使用pb1创建的loop1即可.
将loop设备软链到nvme1n1方便使用, 如果你的容器内已经有这个名字, 可以换一个例如nvme2n1.
sudo ln -s /dev/loop1 /dev/nvme1n1
6、格式化共享盘, 在pb1 或 pb2中, 任意一个容器中执行一次即可. 注意有报错, 可以改成使用:sudo pfs -C disk mkfs -f nvme1n1
sudo pfs -C disk mkfs nvme1n1
以下是日志:
pfs tool cmd record:mkfs nvme1n1
[PFS_LOG] Dec 16 14:02:16.459203 INF [48] pfs build version:libpfs_version_("pfsd-build-desc-_-Wed Sep 4 17:22:25 CST 2024")
[PFS_LOG] Dec 16 14:02:16.459428 INF [48] pid: 47, caller: sudo pfs -C disk mkfs nvme1n1
[PFS_LOG] Dec 16 14:02:16.459501 INF [48] pid: 46, caller: sudo pfs -C disk mkfs nvme1n1
[PFS_LOG] Dec 16 14:02:16.459551 INF [48] pid: 14, caller: bash
[PFS_LOG] Dec 16 14:02:16.460965 INF [48] open device cluster disk, devname nvme1n1, flags 0x13
[PFS_LOG] Dec 16 14:02:16.461028 INF [48] disk dev path: /dev/nvme1n1
[PFS_LOG] Dec 16 14:02:16.461041 INF [48] open local disk: open(/dev/nvme1n1, 0x10002)
[PFS_LOG] Dec 16 14:02:16.461138 INF [48] ioctl status 0
[PFS_LOG] Dec 16 14:02:16.461146 INF [48] pfs_diskdev_info get pi_pbdno 0, pi_rwtype 1, pi_unitsize 4194304, pi_chunksize 10737418240, pi_disksize 21474836480
[PFS_LOG] Dec 16 14:02:16.461149 INF [48] pfs_diskdev_info waste size: 0
[PFS_LOG] Dec 16 14:02:16.461155 INF [48] disk size 0x500000000, chunk size 0x280000000
[PFS_LOG] Dec 16 14:02:16.461888 ERR [48] chunk 0 pfs magic mismatch 0 vs 0x5046534348
[PFS_LOG] Dec 16 14:02:16.461903 INF [48] mkfs PBD nvme1n1 isn't formatted
Init chunk 0
metaset 0/1: sectbda 0x1000, npage 80, objsize 128, nobj 2560, oid range [ 0, a00)
metaset 0/2: sectbda 0x51000, npage 64, objsize 128, nobj 2048, oid range [ 0, 800)
metaset 0/3: sectbda 0x91000, npage 64, objsize 128, nobj 2048, oid range [ 0, 800) Init chunk 1
metaset 1/1: sectbda 0x280001000, npage 80, objsize 128, nobj 2560, oid range [ 1000, 1a00)
metaset 1/2: sectbda 0x280051000, npage 64, objsize 128, nobj 2048, oid range [ 800, 1000)
metaset 1/3: sectbda 0x280091000, npage 64, objsize 128, nobj 2048, oid range [ 800, 1000)
Inited filesystem(21474836480 bytes), 2 chunks, 2560 blktags, 2048 direntries, 2048 inodes per chunk
making paxos file
init paxos lease
making journal file
pfs mkfs succeeds!
我用上面那条命令, 日志中出现了报错, 但是命令成功结束.
[PFS_LOG] Dec 16 14:02:16.461888 ERR [48] chunk 0 pfs magic mismatch 0 vs 0x5046534348
发现有一个issue是同样的问题. 回复: “确实是磁盘太小引起的这个问题，改成大于10G的盘后，问题解决。” 但是我开始报错的是20G, 改成了40G, 还是一样的报错.
https://github.com/ApsaraDB/PolarDB-FileSystem/issues/13
我这里加-f就不报错了
sudo pfs -C disk mkfs -f nvme1n1
7、在pb1, pb2都启动pfsd,-w为工作进程数量, 数字越大IO性能越好, 这里受制于容器资源限制设置为1.
pb1
sudo /usr/local/polarstore/pfsd/bin/start_pfsd.sh -p nvme1n1 -w 1 option workers `
option pbdname nvme1n1
option server id 0
option logconf /usr/local/polarstore/pfsd/bin/../conf/pfsd_logger.conf
starting pfsd[68] nvme1n1
pfsdaemon nvme1n1 start success
查看该共享盘内, 当前无数据
sudo pfs -C disk ls /nvme1n1/
File 1 4194304 Mon Dec 16 14:02:16 2024 .pfs-paxos
File 1 1073741824 Mon Dec 16 14:02:45 2024 .pfs-journal
total 2105344 (unit: 512Bytes)
pb2
sudo /usr/local/polarstore/pfsd/bin/start_pfsd.sh -p nvme1n1 -w 1 option workers `
option pbdname nvme1n1
option server id 0
option logconf /usr/local/polarstore/pfsd/bin/../conf/pfsd_logger.conf
starting pfsd[86] nvme1n1
pfsdaemon nvme1n1 start success
查看该共享盘内, 当前无数据
sudo pfs -C disk ls /nvme1n1/
File 1 4194304 Mon Dec 16 14:02:16 2024 .pfs-paxos
File 1 1073741824 Mon Dec 16 14:02:45 2024 .pfs-journal
total 2105344 (unit: 512Bytes)
停止pfsd使用如下方式
sudo /usr/local/polarstore/pfsd/bin/stop_pfsd.sh nvme1n1
8、在pb1编译polardb 15.
cd /data # 克隆PolarDB 15 稳定分支代码
git clone --depth 1 -b POLARDB_15_STABLE https://github.com/ApsaraDB/PolarDB-for-PostgreSQL
# 编译安装PolarDB 15, 不要初始化实例
cd /data/PolarDB-for-PostgreSQL
./build.sh --prefix=/data/polardb --jobs=4 --debug=off --port=5432 --ni --ec="--with-pfsd"
# 注意一定要加上 with-pfsd, 否则启动数据库时会报错.
# 之前 ./build.sh --prefix=/data/polardb --jobs=4 --debug=off --port=5432 --ni 启动数据库就遇到了第13步的报错
# 参考 : https://github.com/ApsaraDB/polardb-pg-docker-images/blob/main/Dockerfile-binary
以下是日志:
Begin clean existing installation, may raising errors, ignore them
Makefile:42: warning: overriding recipe for target 'check'
../../src/makefiles/pgxs.mk:451: warning: ignoring old recipe for target 'check'
Begin configure, flag: --enable-depend --with-uuid=e2fs --disable-rpath --with-segsize=128 --with-openssl --enable-nls --with-libxml --with-libxslt --with-icu --with-pam --with-gssapi --with-ldap --with-perl --with-python --with-tcl --with-llvm --with-lz4 --with-zstd --with-system-tzdata=/usr/share/zoneinfo --with-libunwind -q --prefix=/data/polardb/tmp_polardb_pg_15_base --with-pgport=5432
Begin compile and install PolarDB, flag: -j4 -s
Makefile:42: warning: overriding recipe for target 'check'
../../src/makefiles/pgxs.mk:451: warning: ignoring old recipe for target 'check'
Skip init PolarDB cluster
9、将编译好的二进制拷贝到pb1, pb2的HOME目录, 便于调用:
$ cp -r /data/polardb/tmp_polardb_pg_15_base ~/
$ which psql
/home/postgres/tmp_polardb_pg_15_base/bin/psql
版本
$ psql -V
psql (PostgreSQL) 15.10
10、将pb1作为primary 节点, 在pb1执行创建polardb实例共享数据目录.
sudo pfs -C disk mkdir /nvme1n1/shared_data
11、初始化primary节点
initdb -D $HOME/primary sudo /home/postgres/tmp_polardb_pg_15_base/bin/polar-initdb.sh $HOME/primary/ /nvme1n1/shared_data/ primary
以下是日志
....
copy file from /home/postgres/primary//pg_multixact/offsets/0000 to /nvme1n1/shared_data///pg_multixact/offsets/0000 succeeded
copy file from /home/postgres/primary//pg_multixact/members/0000 to /nvme1n1/shared_data///pg_multixact/members/0000 succeeded
list base dir file 1
13572
4
5
delete /home/postgres/primary/base/1/*
delete /home/postgres/primary/base/13572/*
delete /home/postgres/primary/base/4/*
delete /home/postgres/primary/base/5/*
delete /home/postgres/primary/global/*
delete /home/postgres/primary/pg_wal
delete /home/postgres/primary/pg_logindex
delete /home/postgres/primary/pg_twophase
delete /home/postgres/primary/pg_xact
delete /home/postgres/primary/pg_commit_ts
delete /home/postgres/primary/pg_multixact
init PolarDB data dir success
可以看到当前pfs里共享数据目录中已经有数据了. 实际上是初始化过程中把一些initdb的目录move过去的, 剩下的就是本地实例文件.
$ sudo pfs -C disk ls /nvme1n1/shared_data
Dir 1 512 Mon Dec 16 14:34:48 2024 base
Dir 1 7808 Mon Dec 16 14:34:48 2024 global
Dir 1 0 Mon Dec 16 14:34:48 2024 pg_tblspc
Dir 1 256 Mon Dec 16 14:35:19 2024 pg_wal
Dir 1 0 Mon Dec 16 14:35:19 2024 pg_logindex
Dir 1 0 Mon Dec 16 14:35:19 2024 pg_twophase
Dir 1 128 Mon Dec 16 14:35:19 2024 pg_xact
Dir 1 0 Mon Dec 16 14:35:19 2024 pg_commit_ts
Dir 1 256 Mon Dec 16 14:35:20 2024 pg_multixact
total 0 (unit: 512Bytes)
12、配置pb1本地实例postgresql.conf,pg_hba.conf
echo"
huge_pages=off
port=5432
polar_hostid=1
polar_enable_shared_storage_mode=on
polar_disk_name='nvme1n1'
polar_datadir='/nvme1n1/shared_data/'
polar_vfs.localfs_mode=off
shared_preload_libraries='\$libdir/polar_vfs,\$libdir/polar_worker'
polar_storage_cluster_name='disk'
logging_collector=on
log_line_prefix='%p\t%r\t%u\t%m\t'
log_directory='pg_log'
listen_addresses='0.0.0.0'
max_connections=200
" >> ~/primary/postgresql.conf
echo"
host all all 0.0.0.0/0 md5
host replication postgres 172.17.0.0/16 trust
" >> ~/primary/pg_hba.conf
13、在pb1启动primary实例
pg_ctl start -D $HOME/primary
遇到报错如下:
waiting for server to start....36762 2024-12-16 15:35:02.906 CST LOG: polar_vfs loaded in postmaster 36762
36762 2024-12-16 15:35:02.906 CST LOG: Database will be in readwrite mode
36762 2024-12-16 15:35:02.906 CST LOG: init pangu cluster disk
36762 2024-12-16 15:35:02.906 CST LOG: begin mount pfs name nvme1n1 id 1 pid 36762 backendid -1
36762 2024-12-16 15:35:02.906 CST LOG: mount pfs nvme1n1 readwrite (force) mode success
36762 2024-12-16 15:35:02.906 CST LOG: polar_vfs init done
stopped waiting
pg_ctl: could not start server
Examine the log output.
数据库无日志文件, 该目录不存在
~/primary/pg_log
pfs进程显示正常, 日志如下, 即使重启pfs后, 再启动primary还是报错.
$ ps -ewf|grep pfs
root 36592 1 6 15:32 ? 00:00:16 /usr/local/polarstore/pfsd/bin/../bin/pfsdaemon -p nvme1n1 -w 1 -c /usr/local/polarstore/pfsd/bin/../conf/pfsd_logger.conf $ cat /usr/local/polarstore/pfsd/bin/../conf/pfsd_logger.conf
# Copyright (c) 2017-2021, Alibaba Group Holding Limited
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[global]
strict init = true
buffer min = 1024
buffer max = 2MB
default format = "%d.%us|%6V|%p|%U:%f:%L|%m%n"
file perms = 644
[formats]
simple = "%d.%us|%-5V|%f:%L|%m%n"
trace = "%m%n"
[rules]
pfsd_cat.* "/var/log/pfsd-%E(PFSD_PBDNAME)/pfsd.log", 100MB * 10 ~ "/var/log/pfsd-%E(PFSD_PBDNAME)/pfsd.log.#r";simple
original_cat.* "/var/log/pfsd-%E(PFSD_PBDNAME)/pfsd.log", 100MB * 10 ~ "/var/log/pfsd-%E(PFSD_PBDNAME)/pfsd.log.#r";trace
test_cat.* >stdout; simple
# pfs日志内容未看到错误
postgres@e065fac6f0b4:~$ cat /var/log/pfsd-nvme1n1/pfsd.log
# pfs重启后日志
sudo /usr/local/polarstore/pfsd/bin/stop_pfsd.sh nvme1n1
sudo /usr/local/polarstore/pfsd/bin/start_pfsd.sh -p nvme1n1 -w 1
2024-12-16 15:37:18.106717|INFO |pfsd_main.cc:142|[pfsd]pthread_join 0
2024-12-16 15:37:18.106783|INFO |pfsd_main.cc:148|[pfsd]byebye
2024-12-16 15:37:27.899529|INFO |pfsd_main.cc:101|starting pfsd[36806] nvme1n1
2024-12-16 15:37:27.899613|INFO |pfsd_shm.cc:113|ch 32, unit 4096, sizeof pfsd_shm 11.2539M
2024-12-16 15:37:27.899701|INFO |pfsd_shm.cc:113|ch 32, unit 8192, sizeof pfsd_shm 19.2539M
2024-12-16 15:37:27.899775|INFO |pfsd_shm.cc:113|ch 32, unit 16384, sizeof pfsd_shm 35.2539M
2024-12-16 15:37:27.899841|INFO |pfsd_shm.cc:113|ch 16, unit 65536, sizeof pfsd_shm 65.6289M
2024-12-16 15:37:27.899896|INFO |pfsd_shm.cc:113|ch 4, unit 262144, sizeof pfsd_shm 64.4102M
2024-12-16 15:37:27.899933|INFO |pfsd_shm.cc:113|ch 1, unit 1048576, sizeof pfsd_shm 64.1055M
2024-12-16 15:37:27.899959|INFO |pfsd_shm.cc:113|ch 1, unit 4194304, sizeof pfsd_shm 64.1055M
2024-12-16 15:37:27.899973|INFO |pfsd_shm.cc:134|total pfsd_shm 324.012M
2024-12-16 15:37:27.900041|INFO |pfsd_worker.cc:227|skip busy cpu 0
2024-12-16 15:37:27.900049|INFO |pfsd_worker.cc:227|skip busy cpu 1
2024-12-16 15:37:27.900053|INFO |pfsd_worker.cc:227|skip busy cpu 2
2024-12-16 15:37:27.900058|INFO |pfsd_worker.cc:227|skip busy cpu 3
2024-12-16 15:37:27.900063|ERROR|pfsd_worker.cc:260|may be too many workers or not enough cpu: workes 1, cpu 4
2024-12-16 15:37:27.900139|WARN |pfsd_worker.cc:309|can't find cpuidx for worker 0
2024-12-16 15:37:27.900227|INFO |pfsd_chnl_shm.cc:912|prepare done 0, notify listen thread
2024-12-16 15:37:27.900253|INFO |pfsd_chnl_shm.cc:389|listen thread waiting to work.
2024-12-16 15:37:27.900285|INFO |pfsd_chnl_shm.cc:391|listen thread start working.
数据库core文件如下
postgres@e065fac6f0b4:~$ gdb --exec=/home/postgres/tmp_polardb_pg_15_base/bin/postgres --core=./core
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty"for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration"for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>. For help, type"help".
Type "apropos word" to search for commands related to "word".
[New LWP 36819]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/postgres/tmp_polardb_pg_15_base/bin/postgres -D /home/postgres/primary'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x0000ffff9b2c6010 in vfs_stat (path=0xffffef7080d8 "/nvme1n1/shared_data//global/polar_settings.conf", buf=0xffffef707838) at polar_vfs_interface.c:673
#2 0x0000aaaab780f12c in ?? ()
#3 0x0000ffffef7080d8 in ?? ()
#4 0x6663438ba95dfce6 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
polar_settings.conf的内容如下, 看起来是个空文件:
$ sudo pfs -C disk read /nvme1n1/shared_data/global/polar_settings.conf
# Do not edit this file manually!
# It will be overwritten by the ALTER SYSTEM FOR CLUSTER command.
有解决办法吗?
回复: 编译polardb时缺少了with-pfsd选项, 加上即可:
./build.sh --prefix=/data/polardb --jobs=4 --debug=on --port=5432 --ni --ec="--with-pfsd"
参考: https://github.com/ApsaraDB/polardb-pg-docker-images/blob/main/Dockerfile-binary
$ pg_ctl start -D ~/primary
waiting for server to start....63754 2024-12-17 13:46:10.076 CST LOG: polar_vfs loaded in postmaster 63754
63754 2024-12-17 13:46:10.076 CST LOG: Database will be in readwrite mode
63754 2024-12-17 13:46:10.076 CST LOG: init pangu cluster disk
63754 2024-12-17 13:46:10.076 CST LOG: begin mount pfs name nvme1n1 id 1 pid 63754 backendid -1
[PFSD_SDK INF Dec 17 13:46:10.076311][63754]pfs_mount_prepare 103: begin prepare mount cluster(disk), PBD(nvme1n1), hostid(1),flags(0x200013)
[PFSD_SDK INF Dec 17 13:46:10.076391][63754]pfs_mount_prepare 165: pfs_mount_prepare success for nvme1n1 hostid 1
[PFSD_SDK INF Dec 17 13:46:10.180610][63754]chnl_connection_poll_shm 1238: ack data update s_mount_epoch 1
[PFSD_SDK INF Dec 17 13:46:10.180623][63754]chnl_connection_poll_shm 1266: connect and got ack data from svr, err = 0, mntid 0
[PFSD_SDK INF Dec 17 13:46:10.180767][63754]pfsd_sdk_init 191: pfsd_chnl_connect success
[PFSD_SDK INF Dec 17 13:46:10.180776][63754]pfs_mount_post 208: pfs_mount_post err : 0
63754 2024-12-17 13:46:10.180 CST LOG: mount pfs nvme1n1 readwrite (force) mode success
63754 2024-12-17 13:46:10.180 CST LOG: polar_vfs init done
63754 2024-12-17 13:46:10.198 CST LOG: "polar_settings.conf" is copied from sharedstorage success, additional load it.
63754 2024-12-17 13:46:10.200 CST LOG: polardb read control file "/nvme1n1/shared_data//global/pg_control" success
63754 2024-12-17 13:46:10.200 CST LOG: PolarDB load controlfile in postmaster
63754 2024-12-17 13:46:10.204 CST LOG: clog buffer max slot size is 256, and it will be set to 256
63754 2024-12-17 13:46:10.204 CST LOG: commit timestamp buffer max slot size is 512, and it will set to 256
63754 2024-12-17 13:46:10.204 CST LOG: clog buffer max slot size is 256, and it will be set to 256
63754 2024-12-17 13:46:10.204 CST LOG: commit timestamp buffer max slot size is 512, and it will set to 256
63754 2024-12-17 13:46:10.207 CST LOG: clog buffer max slot size is 256, and it will be set to 256
63754 2024-12-17 13:46:10.207 CST LOG: commit timestamp buffer max slot size is 512, and it will set to 256
63754 2024-12-17 13:46:10.212 CST LOG: initialized faultinjector shmem
63754 2024-12-17 13:46:10.224 CST LOG: PolarIOStatArray share memory total size is 1220464
63754 2024-12-17 13:46:10.226 CST LOG: redirecting log output to logging collector process
63754 2024-12-17 13:46:10.226 CST HINT: Future log output will appear in directory "pg_log".
done
server started
14、在 pb1 Primary 节点上，为pb2的 Replica 节点创建相应的复制槽，用于 Replica 节点的物理复制：
psql -p 5432 -d postgres -c "SELECT pg_create_physical_replication_slot('replica1');"
pg_create_physical_replication_slot
-------------------------------------
(replica1,)
(1 row)
15、replica节点部署, 在pb2节点本地磁盘的~/replica1 路径下创建一个空目录，然后通过polar-initdb.sh 脚本使用共享存储上的数据目录来初始化 Replica 节点的本地目录。初始化后的本地目录中没有默认配置文件，所以还需要使用initdb 创建一个临时的本地目录模板，然后将所有的默认配置文件拷贝到 Replica 节点的本地目录下：
# 进入pb2容器
docker exec -ti pb2 bash mkdir -m 0700 $HOME/replica1
sudo /home/postgres/tmp_polardb_pg_15_base/bin/polar-initdb.sh $HOME/replica1/ /nvme1n1/shared_data/ replica
# 日志如下:
init replica
list base dir file 5
1
4
13572
mkdir /home/postgres/replica1/base/5
mkdir /home/postgres/replica1/base/1
mkdir /home/postgres/replica1/base/4
mkdir /home/postgres/replica1/base/13572
init PolarDB replica mode dir success
修改replica数据目录及子目录及文件的权限和owner
sudo chmod -R 700 $HOME/replica1
sudo chown -R postgres:postgres $HOME/replica1
生成初始配置文件
initdb -D /tmp/replica1
cp /tmp/replica1/*.conf $HOME/replica1/
16、编辑pb2 replica节点配置文件
echo"
port=5432
polar_hostid=2
polar_enable_shared_storage_mode=on
polar_disk_name='nvme1n1'
polar_datadir='/nvme1n1/shared_data/'
polar_vfs.localfs_mode=off
shared_preload_libraries='\$libdir/polar_vfs,\$libdir/polar_worker'
polar_storage_cluster_name='disk'
logging_collector=on
log_line_prefix='%p\t%r\t%u\t%m\t'
log_directory='pg_log'
listen_addresses='0.0.0.0'
max_connections=200 # replication
primary_slot_name='replica1'
primary_conninfo='host=172.17.0.2 port=5432 user=postgres dbname=postgres application_name=replica1'
"
>> ~/replica1/postgresql.conf
echo"
host all all 0.0.0.0/0 md5
host replication postgres 172.17.0.0/16 trust
" >> ~/replica1/pg_hba.conf
标识节点以 Replica 模式启动：
touch $HOME/replica1/replica.signal
启动 Replica 节点：
pg_ctl start -D $HOME/replica1
检查 Replica 节点能否正常运行：
psql -p 5432 -d postgres -c 'SELECT version();'
version
--------------------------------------------------------------------------------
PostgreSQL 15.10 (PolarDB 15.10.2.0 build d4f5477d debug) on aarch64-linux-gnu
(1 row)
17、检查PolarDB基于共享存储的复制是否正常
pb1 检查到replica复制槽正常连接
postgres=# select * from pg_stat_replication;
-[ RECORD 1 ]----+------------------------------
pid | 63834
usesysid | 10
usename | postgres
application_name | replica1
client_addr | 172.17.0.3
client_hostname |
client_port | 49322
backend_start | 2024-12-17 14:06:06.335694+08
backend_xmin |
state | streaming
sent_lsn | 0/40963428
write_lsn | 0/40963428
flush_lsn | 0/40963428
replay_lsn | 0/40963428
write_lag |
flush_lag |
replay_lag |
sync_priority | 0
sync_state | async
reply_time | 2024-12-17 14:06:16.373174+08
在pb1 primary 节点建表, 写入测试数据库
postgres@d648efb76b69:~/primary$ psql
psql (PostgreSQL 15.10 (PolarDB 15.10.2.0 build d4f5477d debug) on aarch64-linux-gnu)
Type "help"forhelp.
postgres=# create table tbl (id int primary key, info text, ts timestamp);
CREATE TABLE
postgres=# insert into tbl select generate_series(1,10), md5(random()::text), clock_timestamp();
INSERT 0 10
在pb2 replica 节点, 查询到数据正常, 确认PolarDB基于共享存储的复制正常
postgres@d99291973fcc:~/replica1$ psql
psql (PostgreSQL 15.10 (PolarDB 15.10.2.0 build d4f5477d debug) on aarch64-linux-gnu)
Type "help"forhelp. postgres=# select * from tbl;
id | info | ts
----+----------------------------------+----------------------------
1 | 8ddf7a3ddf9d9a348637dba4bc2d9308 | 2024-12-17 14:09:44.805661
2 | 374171ae8e839d595112570bbf6e6a5d | 2024-12-17 14:09:44.808097
3 | 36c83070ecef903804f843d8b6f542a5 | 2024-12-17 14:09:44.808106
4 | eeb9fb43ed70b98bc1e3cac7a8591268 | 2024-12-17 14:09:44.80811
5 | 7d24eb72eee5dc5abd73337f55b3bf91 | 2024-12-17 14:09:44.808114
6 | 4c2910dc1939dc57a7cc91334a3546ce | 2024-12-17 14:09:44.808118
7 | e59739763a6137b05f9b85c6f4daf83d | 2024-12-17 14:09:44.808121
8 | 36183814d28784a1dd8e145daada9e75 | 2024-12-17 14:09:44.808124
9 | 1757ba67ab4c49af449e56ee0f70485c | 2024-12-17 14:09:44.808127
10 | 4b2a4e2849c9ace23c21258c6e95842b | 2024-12-17 14:09:44.808131
(10 rows)
参考
《PolarDB PG 15 编译安装 & pg_duckdb 插件 + OSS 试用》
https://apsaradb.github.io/PolarDB-for-PostgreSQL/zh/deploying/fs-pfs.html
https://github.com/ApsaraDB/PolarDB-FileSystem/blob/master/Readme-CN.md
https://github.com/ApsaraDB/PolarDB-FileSystem/blob/master/docs/PFS_Tools-CN.md
https://www.jamescoyle.net/how-to/2096-use-a-file-as-a-linux-block-device
《在 macOS 中将文件模拟为块设备》
《在 macOS 中用docker --device挂载块设备报错》
文末彩蛋:国产数据库周边生态
当然一款数据库要流行起来, 除了自己要强大, 还离不开生态. 用好周边生态工具, 管理水平战胜90%老司机!!! 下面简单介绍一下国产数据库周边生态.
1、管控软件
鸣嵩(前阿里云数据库总经理 / 研究员)等大佬们创业创办的云猿生, 核心产品是KubeBlocks. 他们的理念是让管理数据库和搭积木一样简单, 如果你要管理很多套并且种类(OLTP\OLAP\NoSQL\KV\TS\MQ等)很多的数据库产品, 推荐首选.
https://github.com/apecloud/kubeblocks
PG中文社区核心委员唐成老师的公司乘数开源的Clup, 专用管理PostgreSQL和PolarDB的集群管理软件, 如果你要管理很多套数据库, 推荐选择. 并且Clup还提供了企业版、自研的连接池、分布式存储、一体机、备份平台等, 是企业用户推荐之选.
https://www.csudata.com/
若航老司机开源的pigsty, 集成了300多个PG插件的PG集群和PolarDB集群管理软件, 如果你要管理很多套PG或PolarDB数据库, 且对插件有特别多的需求, 推荐选择.
https://pigsty.cc/zh/
2、审计监控诊断优化
翟总(曾经是我背后的男人)到海信聚好看后研发的 DBdoctor, 采用ebpf技术, 在对数据库几乎没有影响的情况下实时监控数据库和服务器的各项指标, 发现和诊断问题根因非常方便.
https://www.dbdoctor.cn/
天舟老哥的核心产品Bytebase 是位于您和数据库之间的中间件。它是数据库 DevOps 的 GitLab/GitHub，专为开发人员、DBA 和平台工程师打造。
https://bytebase.cc/docs/introduction/what-is-bytebase/
PawSQL, SQL优化和诊断产品.
D-Smart, Oracle老前辈白老大出品, 专注企业级市场, 将业界顶级DBA经验的产品化作品, 产品功能包括数据库监控、诊断、优化等.
https://www.modb.pro/db/567140
3、国产数据库IDE
IDE是开发者的必备工具，例如社区有pgAdmin, 国产IDE则可以看看老程序猿达刚老师的DeskUI:
https://www.deskui.com
4、数据同步&迁移&备份恢复
NineData, 老领导出去创业做的产品, 产品涵盖了数据同步、迁移、备份、比对、devops、chatDBA等.
https://www.ninedata.cloud/home
DSG, 非常老牌的数据库同步迁移企业级产品, 支持各种数据库的异构和同构迁移, 用他们的话说, 没有dsg搞不定的迁移, 比goldengate还牛.
https://www.dsgdata.com/
公开课
如果你对PolarDB学习感兴趣可以阅读这个公开课系列:
除了PolarDB还非常值得关注的几款PG栈国产数据库:
HaloDB(基于PG兼容PostgreSQL、Oracle、MySQL. http://www.halodbtech.com/ )、 IvorySQL(基于开源PG兼容PG、Oracle. https://www.ivorysql.org/zh-cn/ )、 ProtonBase(云原生分布式数仓. https://protonbase.com/ )、 成都文武数据库（https://ww-it.cn）
参考文档点击阅读原文获得
感谢关注我的github (https://github.com/digoal/blog) 及视频号: