ReaderMX.
PostgreSQL码农集散地

MySQL气数已尽？靠DuckDB能续命吗？

Pdigoal约 10 分钟

参考文档点击文末阅读原文打开; 推荐《最好的PostgreSQL学习镜像》;

MySQL气数已尽，Oracle无力回天，云猿生借DuckDB还魂

MySQL是非常流行的开源数据库, 可能也是史上最惨开源数据库, 从2016年以来一路下跌颓势不减，就这样气数将尽了？

Image

可能是受制于其优化器过于简单单一MySQL数据库仅适合简单的存取场景, 从5.7到8和9的两个大更新几乎聊胜于无，数据量一旦过亿依旧要上分库分表架构, 依旧需要结合其他产品才能处理复杂查询和分析请求. 这也是为什么MySQL基于binlog复制的生态特别发达. 将从MySQL流到搜索引擎、大数据平台的ETL产品相当丰富, 兼容MySQL语法的中间件、分析型产品很多, 这些产品也挺难的，都是为了迎合习惯了MySQL的大量开发者，你们辛苦了. 

虽然生态丰富, 但也着实存在一些问题, 例如同步数据总会遇到延迟、不一致、错误、卡住等问题，管理更多产品必然带来的运维、开发、资源成本提升的等. 大量用户倒戈到PostgreSQL这个功能更加丰富、单体性能更强、扩展可以玩出花来的开源数据库. 就连ProxySQL都倒戈了: <靠MySQL活不下去了! ProxySQL投向PostgreSQL>

Oracle不争气啊，MySQL在它手里怕是要玩完！好在MySQL救命稻草可能来了: 它就是DuckDB , 短短几年已经飙到24K star. 集成到MySQL的话至少能解决大量数据(数十亿级别)实时分析性能问题. 预计带来1000x性能提升！而且DuckDB可不仅仅是分析查询快, 它还支持数据湖架构、支持向量索引(用于火热的AI RAG场景)、插件化等, 这些都是它真正迅速崛起的原因.  

Image

这是我看到的对MySQL用户最具价值的产品组合创新, PG凭借扩展插件早已经用上DuckDB的能力(duckdb_fdw, pg_duckdb, pg_arrow, pg_parquet, aliyun rds pg, aliyun polardb等产品/开源项目), 这回MySQL终于跟上了. 

杭州云猿生已将MySQL+DuckDB集成到开源项目myduckserver中，快上车快上车快上车 :

myduckserver: MySQL+DuckDB power by Apecloud

MyDuck Server

MyDuck Server unlocks serious power for your MySQL analytics. Imagine the simplicity of MySQL’s familiar interface fused with the raw analytical speed of DuckDB. Now you can supercharge your MySQL queries with DuckDB’s lightning-fast OLAP engine, all while using the tools and dialect you know.

❓ Why MyDuck ❓

While MySQL is a popular go-to choice for OLTP, its performance in analytics often largely lags. DuckDB, on the other hand, is built for fast, embedded analytical processing. MyDuck Server lets you enjoy DuckDB's high-speed analytics without leaving the MySQL ecosystem.

With MyDuck Server, you can:

  • Accelerate MySQL analytics by running analytical queries on your MySQL data at speeds several orders of magnitude faster 🚀
  • Keep familiar tools—there’s no need to change your existing MySQL-based data analysis toolchains 🛠️
  • Go beyond MySQL syntax through DuckDB’s full power to expand your analytics potential 💥
  • Run DuckDB in server mode to share a DuckDB instance with your team or among your applications 🌩️
  • and much more! See below for a full list of feature highlights.

MyDuck Server isn’t here to replace MySQL — it’s here to help MySQL users do more with their data. This open-source project gives you a convenient way to integrate high-speed analytics into your MySQL workflow, all while embracing the flexibility and efficiency of DuckDB.

✨ Key Features

  • Blazing Fast OLAP with DuckDB: MyDuck stores data in DuckDB, an OLAP-optimized database known for lightning-fast analytical queries. With DuckDB, MyDuck executes queries up to 1000x faster than traditional MySQL setups, enabling complex analytics that were impractical with MySQL alone.

  • MySQL-Compatible Interface: MyDuck speaks MySQL wire protocol and understands MySQL syntax, so you can connect to it with any MySQL client and run MySQL-style SQL. MyDuck translates your queries on the fly and executes them in DuckDB. Connect your favorite data visualization tools and BI platforms to MyDuck without any changes, and enjoy the speed boost.

  • Zero-ETL: Just START REPLICA and go! MyDuck replicates data from your primary MySQL server in real-time, so you can start querying immediately. There’s no need to set up complex ETL pipelines.

  • Consistent and Efficient Replication: Thanks to DuckDB's solid ACID support, we’ve carefully managed transaction boundaries in the replication stream to ensure a consistent data view — you’ll never see dirty data mid-transaction. Plus, MyDuck’s transaction batching collects updates from multiple transactions and applies them to DuckDB in batches, significantly reducing write overhead (since DuckDB isn’t designed for high-frequency OLTP writes).

  • Raw DuckDB Power: MyDuck also offers a Postgres-compatible port, allowing you to send DuckDB SQL directly. This opens up DuckDB’s full analytical capabilities, including friendly SQL syntax, advanced aggregates, accessing remote data sources, and more.

  • DuckDB in Server Mode: If you aren't interested in MySQL but just want to share a DuckDB instance with your team or among your applications, MyDuck is also a great solution. You can deploy MyDuck to a server, ignore all the MySQL configuration, and connect to it with any PostgreSQL client.

  • Seamless Integration with MySQL Dump & Copy Tools: MyDuck plays perfectly with modern MySQL tools, especially the MySQL Shell, the official advanced MySQL client. You can load data into MyDuck in parallel from a MySQL Shell dump, or leverage the Shell’s copy-instance utility to copy a consistent snapshot of your running MySQL server to MyDuck.

  • Bulk Data Loading: MyDuck supports fast bulk data loading from the client side with the standard MySQL LOAD DATA LOCAL INFILE command or the PostgreSQL COPY FROM STDIN command.

  • Standalone Mode: MyDuck can run in standalone mode, without MySQL replication. In this mode, it is a drop-in replacement for MySQL, but with a DuckDB heart. You can CREATE TABLE, transactionally INSERT, UPDATE, and DELETE data, and run blazingly fast SELECT queries.

📊 Performance

Typical OLAP queries can run up to 1000x faster with MyDuck Server compared to MySQL alone, especially on large datasets. Under the hood, it's just DuckDB doing what it does best: processing analytical queries at lightning speed. You are welcome to run your own benchmarks and prepare to be amazed! Alternatively, you can refer to well-known benchmarks like the ClickBench and H2O.ai db-benchmark to see how DuckDB performs against other databases and data science tools. Also remember that DuckDB has robust support for transactions, JOINs, and larger-than-memory query processing, which are unavailable in many competing systems and tools.

🎯 Roadmap

We have big plans for MyDuck Server! Here are some of the features we’re working on:

  • [ ] Be compatible with MySQL proxy tools like ProxySQL and MariaDB MaxScale.
  • [ ] Replicate data from PostgreSQL.
  • [ ] ...and more! We’re always looking for ways to make MyDuck Server better. If you have a feature request, please let us know by opening an issue.

🏃‍♂️ Getting Started

Prerequisites

  • Docker (recommended) for setting up MyDuck Server quickly.
  • MySQL or PostgreSQL clients for connecting and testing your setup.

Installation

Get a standalone MyDuck Server up and running in minutes using Docker:

docker run -p 13306:3306 -p 15432:5432 apecloud/myduckserver:latest  

This setup exposes:

  • Port 13306 for MySQL wire protocol connections.
  • Port 15432 for PostgreSQL wire protocol connections, allowing direct DuckDB SQL.

Usage

Connecting via MySQL

Connect using any MySQL client to run MySQL-style SQL queries:

mysql -h127.0.0.1 -P13306 -uroot  

Connecting via PostgreSQL

For full analytical power, connect using the PostgreSQL-compatible port and write DuckDB SQL directly:

psql -h 127.0.0.1 -p 15432 -U mysql  

Replicating Data

We have integrated a setup tool in the Docker image that helps replicate data from your primary MySQL server to MyDuck Server. The tool is available via the SETUP_MODE environment variable. In REPLICAmode, the container will start MyDuck Server, dump a snapshot of your primary MySQL server, and start replicating data in real-time.

docker run \  
  --network=host \  
  --privileged \  
  --workdir=/home/admin \  
  --env=SETUP_MODE=REPLICA \  
  --env=MYSQL_HOST=<mysql_host> \  
  --env=MYSQL_PORT=<mysql_port> \  
  --env=MYSQL_USER=<mysql_user> \  
  --env=MYSQL_PASSWORD=<mysql_password> \  
  --detach=true \  
  apecloud/myduckserver:latest  

Connecting to Cloud MySQL

MyDuck Server supports setting up replicas from common cloud-based MySQL offerings. For more information, please refer to the replica setup guide.

今日荐书


彩蛋:国产数据库周边生态

当然一款数据库要流行起来, 除了自己要强大, 还离不开生态. 用好周边生态工具, 管理水平战胜90%老司机!!! 下面简单介绍一下国产数据库周边生态.

1、管控软件

鸣嵩(前阿里云数据库总经理 / 研究员)等大佬们创业创办的云猿生, 核心产品是KubeBlocks. 他们的理念是让管理数据库和搭积木一样简单, 如果你要管理很多套并且种类(OLTP\OLAP\NoSQL\KV\TS\MQ等)很多的数据库产品, 推荐首选.

  • https://github.com/apecloud/kubeblocks

PG中文社区核心委员唐成老师的公司乘数开源的Clup, 专用管理PostgreSQL和PolarDB的集群管理软件, 如果你要管理很多套数据库, 推荐选择. 并且Clup还提供了企业版、自研的连接池、分布式存储、一体机、备份平台等, 是企业用户推荐之选.

  • https://www.csudata.com/

若航开源的pigsty, 集成了300多个PG插件的PG集群和PolarDB集群管理软件, 如果你要管理很多套PG或PolarDB数据库, 且对插件有特别多的需求, 推荐选择.

  • https://pigsty.cc/zh/

2、审计监控诊断优化

翟总(曾经是我背后的男人)到海信聚好看后研发的 DBdoctor, 采用ebpf技术, 在对数据库几乎没有影响的情况下实时监控数据库和服务器的各项指标, 发现和诊断问题根因非常方便.

  • https://www.dbdoctor.cn/

天舟老哥的核心产品Bytebase 是位于您和数据库之间的中间件。它是数据库 DevOps 的 GitLab/GitHub，专为开发人员、DBA 和平台工程师打造。

  • https://bytebase.cc/docs/introduction/what-is-bytebase/

PawSQL, SQL优化和诊断产品.  

D-Smart, Oracle老前辈白老大出品, 专注企业级市场, 将业界顶级DBA经验的产品化作品, 产品功能包括数据库监控、诊断、优化等.

  • https://www.modb.pro/db/567140

3、国产数据库IDE

IDE是开发者的必备工具，例如社区有pgAdmin, 国产IDE则可以看看老程序猿达刚老师的DeskUI:

  • https://www.deskui.com

4、数据同步&迁移&备份恢复

NineData, 老领导出去创业做的产品, 产品涵盖了数据同步、迁移、备份、比对、devops、chatDBA等.

  • https://www.ninedata.cloud/home

DSG, 非常老牌的数据库同步迁移企业级产品, 支持各种数据库的异构和同构迁移, 用他们的话说, 没有dsg搞不定的迁移, 比goldengate还牛.

  • https://www.dsgdata.com/

公开课

如果你对PolarDB学习感兴趣可以阅读这个公开课系列:

除了PolarDB还非常值得关注的几款PG栈国产数据库:

  • HaloDB(基于PG兼容PostgreSQL、Oracle、MySQL. http://www.halodbtech.com/ )、
  • IvorySQL(基于开源PG兼容PG、Oracle. https://www.ivorysql.org/zh-cn/ )、
  • ProtonBase(云原生分布式数仓. https://protonbase.com/ )、
  • 成都文武数据库（https://ww-it.cn）

参考文档点击阅读原文获得

感谢关注我的github (https://github.com/digoal/blog) 及视频号:
Image

KEEP THE THREAD GOING

更多来源文章