Based on Yang Xiangbo's (PostgreSQL ACE) presentation at HOW 2026.

1. The Problem: A Local vs Remote SQL Performance Gap​

A real production case: a SQL query executes in 17 milliseconds on a local PostgreSQL instance, but through Postgres_fdw accessing a remote database, it takes 200 seconds — over 10,000x slower.

Initial investigation naturally suspects network issues. But examining the execution plan via EXPLAIN ANALYZE reveals the true root cause.

On the remote side, the actual SQL executed had the original Join condition (e.g., A.sid = B.id) not correctly pushed down. Instead, the local WHERE filter (e.g., xt = 1) was incorrectly treated as the Join condition and pushed to the remote side. The result: the remote end returned massive intermediate data, while the actual Join was performed locally — performance naturally degraded catastrophically.

2. Root Cause: How Type Conversion Breaks Pushdown Logic​

2.1 Postgres_fdw's Join Pushdown Mechanism​

Postgres_fdw doesn't unconditionally push Join operations to the remote side. The optimizer comprehensively evaluates cost across four Join paths (Nestloop, Hash Join, Merge Join, and Foreign Join), selecting the optimal plan. Even with Foreign Join, pushdown isn't guaranteed — it still requires cost model comparison.

The core validation for pushdown conditions happens in the is_foreign_expr function, requiring:

• Join type support: Only Inner Join, Left Join, and specific types
• Foreign table safety flag: Must be marked as safety
• Expression restrictions: Join conditions across local and foreign tables cannot contain non-pushdown ops or functions
• Final verification: is_foreign_expr must return True

2.2 The Fault Chain in This Case​

The SQL in question had a seemingly harmless pattern: both A.sid and B.id were explicitly cast to text. This type conversion node triggered a cascade:

1. is_foreign_expr detected the type conversion node, classified it in the "unknown or unsafe expression" branch (default branch), returning False
2. The originally valid Join condition (A.sid = B.id) was rejected for pushdown
3. In FDW's optimization logic, to avoid subquery issues, it attempts to merge other valid conditions into the Join clause
4. The only remaining valid condition was the local WHERE xt = 1, which was forcibly "disguised" as a Join condition and pushed to the remote end
5. The remote end executed with xt = 1 as the Join condition, returning massive intermediate rows, while the actual table association was left to local execution

This is the root cause of the performance avalanche.

3. Solution: Using GUC Parameters to "Catch" Performance​

3.1 Why a GUC Solution​

Directly modifying SQL to remove type conversions is clearly the most thorough solution. But in practice, such patterns often involve legacy systems or third-party closed-source software — the SQL is outside the application owner's control. Modifying SQL isn't feasible.

This calls for a solution at the FDW kernel level.

3.2 Intervention Point & Implementation​

The key optimization point: intervene during the optimizer's Foreign Join Path generation phase. Specifically, embed a GUC parameter switch within the path generation logic related to add_foreign_join_paths.

Implementation logic:

1. Add a new GUC parameter (e.g., enable_fdw_join_pushdown), defaulting to on (allow Join pushdown)
2. Before is_foreign_expr validation, check this parameter's state
3. If set to off, directly prevent Foreign Join path generation, forcing Joins to execute locally
4. The parameter can be dynamically modified at session level without database restart

3.3 Optimization Results​

When the GUC parameter disables Join pushdown, the same SQL execution time drops from 200 seconds to 79 milliseconds. The execution plan shows: the remote end only performs base table filtered scans (SELECT ... WHERE xt = 1), pulling data back locally for Join — though abandoning the remote Join optimization opportunity, in this specific scenario, network transmission volume is far smaller than the intermediate result set from erroneous pushdown, dramatically improving overall performance.

Performance improvement: ~3,000x.

4. Lessons Learned​

First, when troubleshooting FDW performance issues, the primary step is to check the SQL actually executed on the remote side. Local execution plans alone are insufficient. Use EXPLAIN (VERBOSE, ANALYZE) or check remote logs to confirm whether pushed-down SQL matches expectations.

Second, type conversions on Join keys are a common FDW pushdown trap. Type conversion nodes are judged as unsafe expressions by is_foreign_expr, causing otherwise valid Join conditions to be rejected for pushdown. When writing cross-database SQL, avoid explicit type conversions on Join keys whenever possible.

Third, GUC parameters are effective rapid remediation tools. When SQL-level changes are impossible, using kernel parameters to control optimizer behavior solves problems without application code changes. For cloud services or DBA operations teams, this "safety net" capability is especially valuable — recovering business first, then planning long-term solutions at leisure.

The GUC parameter added in this case is just a simple switch, but the underlying approach is worth promoting: many FDW-related performance issues can be addressed through similar "intervention point" designs, giving users more runtime control rather than baking all optimization decisions into hard-coded logic.

Based on Liu Zhilong's presentation at HOW 2026. Liu is a PG database expert at Ping An Technology.

1. Cgroup Memory Management Overview​

1.1 Cgroup Basics​

Cgroup (Control Group) is a Linux kernel mechanism for process group resource management — quota limits, isolation, and usage statistics for CPU, physical memory, I/O, and network. In cloud environments, multiple PG instances run on the same physical host, each managed via Cgroup — standard practice across cloud providers. Container technologies also rely on Cgroup for resource constraints.

1.2 The Fundamental Difference: CPU vs Memory Management​

Cgroup's CPU and memory management logic is fundamentally different:

• CPU: Core mechanism is time allocation via timeslice rotation. Even at the CPU limit, tasks just run slower — they still execute.

• Memory: Core mechanism is page counting. Memory must be immediately available; working memory occupied by one task cannot be reused by another (except shared memory). When memory hits the limit, there's no "slow down" option — only reclamation or death.

The critical difference: hitting the CPU cap means slower execution. Hitting the memory cap means reclamation or termination.

1.3 Overselling in Cloud Environments​

On shared cloud hosts, multiple Cgroup instances run concurrently. Sum all Cgroup limits and they may far exceed physical memory — "overselling." But for PG, overselling doesn't necessarily mean actual memory shortage — because Cache is reclaimable.

Actual memory usage falls into three categories: private memory (non-reclaimable), shared memory (PG Shared Buffers, shared by multiple processes, non-reclaimable), and Cache (file cache, reclaimable). When multiple instances have significant idle or reclaimable Cache space, real usage may not exceed limits even if the sum of quotas exceeds physical memory.

But scaling up or adding instances can push OS memory watermark thresholds, triggering aggressive Cache reclamation. Whose Cache gets reclaimed becomes a "luck of the draw" — potentially impacting critical instances and causing performance jitter.

1.4 PG Memory Layers on OS​

PG process memory decomposes into three layers:

• RSS (Resident Set Size): Divided into private and shared memory, generally non-reclaimable
• Page Cache: File cache, reclaimable
• Kernel: Kernel's own memory overhead

At the Cgroup level, each instance has an independent LRU list; reclamation only affects the local Cgroup, achieving resource isolation.

2. Page Counting & Statistical Pitfalls​

2.1 Key Interface Files​

Core files in Cgroup memory management:

• memory.stat: Detailed statistics for all memory categories — the primary observation entry point
• memory.usage_in_bytes: Not recommended for direct use; accurate usage requires manual calculation
• cgroup.procs: Must include all PG processes (postmaster, checkpointer, bgwriter, walwriter, backend, etc.)
• memory.limit_in_bytes: Defines the Cgroup memory cap

2.2 Core memory.stat Metrics​

Using a real example: PG instance with shared_buffers=64GB, shared_memory_type=mmap, ~800 clients. Key metrics in memory.stat:

• cache: Page Cache size
• rss: Anonymous pages and Swap memory — different from OS process RSS!
• mapped_file: File shared memory size — PG shared memory is counted here
• pgpgin / pgpgout: Charged/uncharged page counts for RSS+Cache
• inactive_anon / active_anon: LRU distribution for anonymous pages and swap cache
• inactive_file / active_file: LRU distribution for file cache

2.3 Core Statistical Deviations​

Manual calculation reveals Cgroup v1's statistical confusion:

shared_mem_mapped = inactive_anon + active_anon - rss
cache - shared_mem_mapped = inactive_file + active_file
rss + mapped_file = inactive_anon + active_anon
inactive_file + active_file + rss + mapped_file = rss + cache

• Cgroup v1's RSS does not include file map shared memory
• PG shared memory (whether mmap or sysv) is classified under mapped_file
• With huge pages, shared memory doesn't appear in ANY statistical metric (including rss_huge)

2.4 Process RSS vs Cgroup RSS​

Sort PG processes by RSS: checkpointer and bgwriter typically have the largest RSS (60GB+ range) — because they frequently operate on Shared Buffers and occupy significant shared memory. Postmaster's RSS is much smaller since it only needs to create the shared memory virtual address space for child processes.

Examining /proc/[pid]/smaps reveals: checkpointer, bgwriter, and postmaster have identical shared memory virtual addresses, but different RSS. Forked child processes inherit the same virtual address mapping, but RSS only grows for processes that actually access corresponding physical pages.

2.5 How to Correctly Reproduce Cgroup OOM​

Due to PG's Double Buffer architecture (Shared Buffers + Page Cache), with shared_buffers = 1/4 × cg_mem as typical config, Page Cache can occupy up to 3/4 of Cgroup memory. Normal business operations don't consume much private memory, so even at Cgroup memory saturation, the system can reclaim from Page Cache — the instance may not OOM.

To effectively trigger Cgroup OOM, simply create many sessions consuming private memory (large sorts, Hash Joins, etc.), rather than flooding Page Cache with writes. The latter only triggers Cache reclamation, not actual memory exhaustion.

3. Cgroup OOM Behavior & Impact​

3.1 OOM Trigger Mechanism​

Two scenarios:

OOM Killer On: Kernel selects the highest OOM Score process (typically a Backend user process) and sends SIGKILL. PG's postmaster detects abnormal child termination, and if shared memory is intact, auto-restarts a new process. Brief service interruption but instance-wide recovery possible. Logged in dmesg and PG logs.

OOM Killer Off: No kernel kill, but PG processes may hang due to memory allocation failure (state D, wait event mem_cgroup_oom). Critical processes like walwriter, if unable to obtain memory, can crash the entire instance. Even with OOM Killer off, the instance may still fail — not killed, but dead by resource starvation.

3.2 Key Distinctions​

• Cgroup OOM ≠ System VM OOM. System-level VM OOM is independently judged by vm.overcommit
• With Cgroup v1's memory.oom_control off, OOM Killer doesn't start, but processes may still hang or crash due to memory shortage

4. Cgroup v1 Deficiencies & v2 Improvements​

4.1 v1's Monitoring Blind Spots​

• No Page Table statistics: High process counts can consume tens of GB in Page Tables, invisible in v1
• No Slab statistics: Kernel Slab memory overhead not tracked
• No HugePage statistics: Huge pages are a complete statistical black hole
• Cannot distinguish async/sync reclamation: pgscan_kswapd vs pgscan_direct indistinguishable at Cgroup granularity
• Shared memory statistical chaos: shmem and file_mapped mixed together
• RSS statistical inconsistency: Cgroup RSS ≠ process RSS

4.2 v2 Improvements​

Cgroup v2 (Linux 4.5+) provides significant memory management enhancements:

Management layer: Clearer hierarchy, more intuitive configuration; new memory.min/low/high watermarks for fine-grained reclamation control; better spike handling; removes direct OOM Killer disable, replaced with gentler control; adds memory_hugetlb_accounting — finally tracking huge pages.

Observation layer: Adds Slab, Page Table, pgscan_kswapd/direct, pgsteal_kswapd metrics; new Socket, Vmalloc, Transparent Huge Pages, Zswap, zero-page swap dedicated statistics; shmem and file_mapped indicators now separated; huge page info finally observable.

5. Huge Pages: Management & Challenges​

5.1 Advantages​

Huge pages benefit PG stability and performance:

• Slightly improved TPS
• Reduced TLB (Translation Lookaside Buffer) refresh pressure, lower CPU cache contention
• Significantly reduces Page Table size in main memory — saves memory and accelerates virtual-to-physical address translation during reclamation
• Physical contiguity improves memory access locality
• Non-huge shared memory is theoretically OS-reclaimable; huge pages are not

5.2 Shared Buffers & Huge Pages Configuration​

Setting shared_buffers to 1/4 of Cgroup memory is industry standard, but reality is more complex. Smaller shared_buffers slightly increases Page Cache (larger total cache); larger shared_buffers slightly reduces total cache but improves Shared Buffers hit rate. Both directions have trade-offs.

Based on actual stress testing and production experience, a more reliable recommendation:

• Without huge pages: shared_buffers = min(1/4 × total_memory, 20GB)
• With huge pages: shared_buffers = min(1/4 × total_memory, 60GB)

Read-heavy scenarios can increase, but must synchronously adjust bgwriter parameters (bgwriter_delay, bgwriter_lru_maxpages).

5.3 Management Challenges​

Statistical black hole: In Cgroup v1, huge pages completely escape all metrics. An instance's Cgroup limit set to 20GB may actually occupy 30GB physical memory — monitoring shows everything normal. RDS monitoring memory metrics may be completely unreliable with huge pages.

Resource planning difficulty: Huge pages must be pre-allocated at boot, can't dynamically scale. Problems: mixed PG/MySQL deployments on same host have different huge page strategies; unknown instance count and specs; over-allocation means resources can't meet new instances' needs; under-allocation wastes memory.

6. Memory Fault Models & Troubleshooting​

6.1 Fault Classification​

Four layers of memory issues:

OS layer: Memory fragmentation, high Swap-in. Note: huge pages reduce fragmentation but may slightly worsen Swap-in.

PG kernel layer: Metadata bloat, version bugs.

Cgroup layer: Performance jitter from intra-Cgroup memory reclamation. Huge pages significantly improve this.

Private memory layer: Individual backend processes consuming excessive private memory (large result set sorts, Hash Joins), straining the overall Cgroup.

6.2 Monitoring Essentials​

• System layer: /proc/buddyinfo for fragmentation; /proc/meminfo si/so for swap pressure; /proc/vmstat pgscan_kswapd/pgscan_direct for async/sync reclamation
• Cgroup layer: memory.stat metrics, noting v1/v2 differences
• Process layer: ps for per-process RSS/PSS; process wait events (mem_cgroup_oom)
• Database layer: pg_stat_bgwriter checkpoints and buffer allocation; IPC wait events

For buffer hit rate, Read count matters more than Hit Rate. High Hit Rate with high Read count means massive buffer read operations (probably due to large scan range) — this is ineffective cache competition; hit rate alone is misleading.

6.3 Leading Indicators​

Common precursors: page allocation failure, prolonged high-CPU kswapd, frequent direct reclaim, process state D (uninterruptible sleep), increasing malloc-related wait events. OOM is the end result, always accompanied by process disappearance and log records.

Summary​

Cloud PG Cgroup memory management is far more complex than single-machine scenarios. Core conclusions:

1. Migrate to Cgroup v2 to resolve shared memory stats, huge page stats, and reclamation observation blind spots
2. Huge pages improve stability and performance, but must pair with v2, otherwise create statistical black holes
3. Cgroup OOM on/off both have risks: ON — Backend killed but instance recovers; OFF — critical process may hang, instance unavailable
4. Shared Buffers should not exceed 60GB (with huge pages), beyond which flush pressure and non-huge page management costs escalate sharply
5. Memory monitoring is about mastering real usage — especially understanding the Cgroup RSS vs process RSS difference and shared memory statistical attribution

Based on Jiao Shuntian's presentation at HOW 2026.

1. Background & Project Overview​

1.1 The Challenge of De-O (De-Oracle)​

Oracle has long served as the database foundation for enterprise core systems, especially in energy, telecom, and manufacturing industries. De-O migration faces four key challenges that directly determine project success:

• Syntax incompatibility: Requires massive application code changes.
• Semantic behavior differences: Programs run but produce incorrect results.
• Asset reuse difficulty: High cost of migrating PL/SQL business logic.
• High operational complexity: Inconsistent drivers and tools increase coordination overhead.

1.2 IvorySQL Project Overview​

IvorySQL was initiated in 2021 by Inspur Software Group's HighGo, and donated to the OpenAtom Open Source Foundation in 2025 under the Apache 2.0 license. Its core positioning:

• Positioning: Not a standalone DBMS, but deep evolution based on PostgreSQL kernel, avoiding reinventing the wheel.
• Mission: Connect the world through open source, providing an open alternative to Oracle.
• Goal: Lower migration barriers while preserving PG ecosystem and sustainable upgrade paths.
• Release rhythm: Follows PG kernel releases, approximately 40 days; minor versions fix bugs, major versions deliver significant compatibility features.
• Latest: IvorySQL 5.3 (based on PostgreSQL 18.3). Note: IvorySQL 5.4 (PG 18.4) has been released.

1.3 Core Features & Community Progress​

IvorySQL provides comprehensive Oracle compatibility covering common data types, character/time functions, NLS parameters, and built-in capabilities. Since v1.0 in December 2021, continuous updates now reach v5.4. The project offers full-platform packages for x86, ARM, MIPS, and LoongArch, plus Docker, K8s Operator, and browser-based online trials to lower evaluation barriers.

2. Core Architecture & Design Philosophy​

2.1 Dual-Mode Design​

IvorySQL uses a "dual-mode" architecture, specified at initialization via initdb -m pg/oracle:

• PG Mode: Fully compatible with native PostgreSQL.
• Oracle Mode (default): Provides Oracle-compatible syntax and enhanced features.
• Dynamic switching: Use SET ivorysql.compatible_mode to toggle modes in real-time without sacrificing stability on either side.

2.2 Dual Port & Dual Parser Design​

• Port 5432: Native PostgreSQL compatibility.
• Port 1521 (Oracle mode default): Enterprise Oracle syntax support, ready out of the box.

A patented isolation-level dual syntax parsing core adds an independent Oracle parser module, minimizing interference between PostgreSQL and Oracle syntax.

2.3 PL/iSQL Implementation​

IvorySQL adds a plisql language entry in the pg_language system catalog to handle Oracle-style procedural language. The design reuses and evolves PostgreSQL's plpgsql mechanism rather than rewriting from scratch. Oracle mode initialization auto-registers related plugins and language capabilities through a plugin architecture that minimizes intrusion into the mainline kernel.

2.4 Five-Layer Compatibility Framework​

1. SQL Syntax Compatibility: High-frequency DDL/DML, MERGE, q-quoting, LIKE, etc.
2. Object & Stored Procedure Compatibility: Packages, system views, object access patterns.
3. Execution Semantics Compatibility: NULL handling, empty-string-to-NULL, function details, boundary conditions.
4. Procedural Language Compatibility: Procedures, Functions, Cursors, anonymous blocks.
5. Migration & Operations Tools: Complete tool ecosystem.

3. Key Compatibility Features​

3.1 SQL Syntax & Object Compatibility​

• Basic DDL/DML: Table creation, modification, queries, and writes.
• Package Compatibility: Oracle-style custom packages, package bodies, enhanced management via \dk.
• View syntax: Reduced object-layer modification costs.

3.2 Semantic & Behavioral Compatibility — The Real Challenge​

3.3 Advanced Feature Compatibility​

• Sequences & System Views: CACHE/NOCACHE, SCALE, SESSION, GLOBAL options; NEXTVAL and CURRVAL; ALL_SEQUENCES, DBA_SEQUENCES, USER_SEQUENCES views.
• Invisible Column: Not shown in SELECT * by default; must be explicitly referenced.
• XML Functions: 11 new XML functions covering APPENDCHILDXML, UPDATEXML, EXISTSNODE.
• System Views: ALL_TAB_COLUMNS, ALL_CONSTRAINTS, ALL_INDEXES, DBA_/USER_ system views.

3.4 Future Feature Roadmap​

• Autonomous Transaction
• ROWNUM pseudocolumn
• Outer Join (+) syntax
• Synonym
• Reverse Key Index, CREATE INDEX ONLINE
• Pivot/Unpivot, Trigger syntax, Bitmap Index, IOT

3.5 Testing & Quality Assurance​

Dual-framework testing: Oracle test framework validates compatibility behavior alignment; PG test framework maintains kernel stability through mature verification.

4. Migration Practice & Methodology​

4.1 Migration Toolchain​

1. Assessment: Identify object differences, risks, and scope.
2. Conversion: Auto/semi-auto conversion of SQL, objects, and procedural logic.
3. Integration: JDBC, ODBC, libpq adaptation; PostGIS, FDW, pg_stat_statements verification; Kylin, UOS, Anolis compatibility.
4. Operations: Monitoring, backup, management, and daily delivery.

4.2 Full-Platform Delivery & Cloud-Native​

Ready-to-use installation packages for all platforms. Docker images, K8s Operator for modern delivery. Comprehensive documentation, migration guides, FAQs, and best practices.

4.3 Case Study: Financial Core System Migration​

A financial system involving general ledger and other core Oracle systems underwent deep compatibility with Oracle Packages, collection data types, system views, and partition indexes. After rigorous testing, performance and stability met requirements — proving compatibility capabilities directly translate to core business migration success.

5. Community Governance & Future Roadmap​

5.1 Open Governance​

Donated to the OpenAtom Open Source Foundation in 2025 for transparent governance. Code, testing, documentation, Issues, and PRs are all welcome contributions.

5.2 International Ecosystem Cooperation​

A two-year MOU with France's Data Bene, which values PG compatibility, Oracle compatibility, and cloud-native advantages for Oracle migration projects.

5.3 IvorySQL 6.0 Preview​

• Mar 2026: v5.3 (PG 18.3)
• Jun 2026: v5.4 (PG 18.4)
• Sep 2026: v5.5 (PG 18.5)
• Nov 2026: v6.0 (PG 19.0)
• Dec 2026: v6.1 (PG 19.1)

v6.0 highlights: Autonomous Transaction, ROWNUM, Rebuild Index, DBMS_OUTPUT, component adaptation.

5.4 2026 Product & Ecosystem Focus​

• AI ecosystem integration
• New domestic platform certifications
• Cloud and containerized ecosystem
• Migration guides, tuning guides, industry solutions

5.5 Innovation Sandbox​

Real-world feature incubation: logical replication with auto-identity for no-PK tables, safe DDL after view creation, global unique indexes on partitioned tables.

Today, I'm testing PostgreSQL 18.4-bookworm on a real openEuler machine, covering every step, every pitfall in detail.

Previously, pulling PG images from Docker Hub was painful: hundreds of KB/s at peak hours, disconnections at 90% progress, and completely blocked in data centers without external internet access. Now, the IvorySQL community's registry.highgo.com mirrors official PG images — no need to switch registries. One registry for both IvorySQL and native PG, perfect for migration comparison and compatibility testing.

No fluff, all real-machine copy-paste-ready commands. Version tag pitfalls, disk-full disasters, missing host clients, command paste errors — all dissected. Even beginners can deploy successfully by following along.

This article is based on actual openEuler machine testing. All screenshots are real operation records, not simulated demos.

1. Environment Setup​

[root@openeuler-server ~]# docker --version
Docker version 20.10.24, build 297e128
[root@openeuler-server ~]# docker-compose --version
Docker Compose version v2.2.0.0
[root@openeuler-server ~]#

1.1 Critical Tag Pitfall​

Many people will directly execute this command — and it will 100% fail:

# Wrong — bare version tag, registry path is also incorrect (missing postgres/postgres)
docker pull registry.highgo.com/postgres:18.4

Error output:

[root@openeuler-server ~]# docker pull registry.highgo.com/postgres:18.4
Error response from daemon: unknown: bad request: invalid repository name: postgres
[root@openeuler-server ~]#

Root cause: PG images in this registry must carry a distribution suffix like -bookworm or -trixie. Pure numeric version tags don't work. Additionally, the correct path is postgres/postgres, not postgres.

Correct pull command:

[root@openeuler-server ~]# docker pull registry.highgo.com/postgres/postgres:18.4-bookworm
18.4-bookworm: Pulling from postgres/postgres
dc4bf025fa70: Pull complete
07800c4658aa: Pull complete
d3b29a572ab1: Pull complete
9c32480c36e2: Pull complete
ab0222f6080d: Pull complete
26068876c106: Pull complete
204ea99b8184: Pull complete
b9b7e4c43bc2: Pull complete
d626a1ee2499: Pull complete
df093a451bde: Pull complete
4551c7621254: Pull complete
b7cc028a2062: Pull complete
4ea1c3550cc2: Pull complete
Digest: sha256:d323ae4f611b8c41e2488693785b770d2797c10e859b259e4e7706246debb069
Status: Downloaded newer image for registry.highgo.com/postgres/postgres:18.4-bookworm
registry.highgo.com/postgres/postgres:18.4-bookworm
[root@openeuler-server ~]#

2. Lightning-Fast Pull: From 20 Minutes to 30 Seconds​

2.1 Execute Pull​

[root@openeuler-server ~]# docker pull registry.highgo.com/postgres/postgres:18.4-bookworm
18.4-bookworm: Pulling from postgres/postgres
67cf8384fcca: Pull complete
047252d203fc: Pull complete
285c515ff12e: Pull complete
c04bac481cf9: Pull complete
4d2f8983bdeb: Pull complete
928cbc530e9b: Pull complete
b7d7c0b1553c: Pull complete
ef11daf91965: Pull complete
5dab6177213d: Pull complete
b73f5647695f: Pull complete
9c1186127d05: Pull complete
c73bb36f2399: Pull complete
142af709c9ab: Pull complete
Digest: sha256:1fa4bccc8246ee2ba9592895d804dcf767119e632999ef22d2b3355a93b627c3
Status: Downloaded newer image for registry.highgo.com/postgres/postgres:18.4-bookworm
registry.highgo.com/postgres/postgres:18.4-bookworm

Real result: From hitting Enter to download completion, about 30 seconds. Docker Hub was crawling at hundreds of KB/s. Now tens of MB/s. A night-and-day difference.

2.2 Image Integrity Verification​

[root@openeuler-server ~]# docker images | grep postgres
registry.highgo.com/postgres/postgres  18.4-bookworm  dda003c01642  2 weeks ago  440MB
registry.highgo.com/postgres/postgres  18.3-bookworm  3c67953c3a90  7 weeks ago  440MB
[root@openeuler-server ~]#

3. Container Launch: A Trail of Pitfalls​

Image pulled, time to start. Important warning: PostgreSQL Docker images differ from MySQL/IvorySQL — they don't expose port 1521 by default, and environment variable handling is stricter.

3.1 Pitfall 1: Shell History Expansion !! Causes Chaos​

[root@openeuler-server ~]# docker run --name pg184 -p 5432:5432 -e POSTGRES_PASSWORD=StrongPass456!! -v pgdata:/var/lib/postgresql/data -d registry.highgo.com/postgres/postgres:18.4-bookworm
Unable to find image 'run:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 202.160.130.66:443: connect: connection timed out.
See 'docker run --help'.
[root@openeuler-server ~]#

Root cause: Notice the password StrongPass456!!. In Bash, !! means "repeat the last command". Shell expands !! before executing, resulting in a mangled command.

Fix: Wrap password in single or double quotes.

[root@openeuler-server ~]# docker run --name pg184 -p 5432:5432 -e POSTGRES_PASSWORD='StrongPass456!!' -v pgdata:/var/lib/postgresql/data -d registry.highgo.com/postgres/postgres:18.4-bookworm
5867e8daa44e925e7818598013a1da62ad7dbdd685e6aa0fae68a94d590edbce
docker: Error response from daemon: driver failed programming external connectivity on endpoint pg184: Bind for 0.0.0.0:5432 failed: port is already allocated.
[root@openeuler-server ~]#

3.2 Pitfall 2: Port 5432 Already Occupied by IvorySQL​

Error: Bind for 0.0.0.0:5432 failed: port is already allocated.

Check docker ps -a:

[root@openeuler-server ~]# docker ps -a
CONTAINER ID  IMAGE                                            COMMAND                 STATUS    PORTS                              NAMES
5867e8daa44e  registry.highgo.com/postgres/postgres:18.4-...   "docker-entrypoint.…"   Created                                        pg184
90f5945502a0  registry.highgo.com/ivorysql/ivorysql:5.3-ubi8   "docker-entrypoint.…"   Up 9 days 0.0.0.0:1521->1521/tcp, 0.0.0.0:5432->5432/tcp  ivorysql

The ivorysql container already occupies 5432. One port serves one process at a time.

Fix: Change the port mapping. Map host 5433 to container 5432:

[root@openeuler-server ~]# docker stop pg184 && docker rm pg184
[root@openeuler-server ~]# docker run --name pg184 -p 5433:5432 -e POSTGRES_PASSWORD='StrongPass456!!' -v pgdata:/var/lib/postgresql/data -d registry.highgo.com/postgres/postgres:18.4-bookworm
aa42ab39ec930934befb825efa15e7b646af0cd3b84560c3ff5aba73818af60b
[root@openeuler-server ~]#

3.3 Pitfall 3: PG18+ Data Path Breaking Change​

Port issue resolved, but the container still won't start:

[root@openeuler-server ~]# docker ps | grep pg18
[root@openeuler-server ~]#

Check logs:

[root@openeuler-server ~]# docker logs pg184
Error: in 18+, these Docker images are configured to store database data in a
       format which is compatible with "pg_ctlcluster" (specifically, using
       major-version-specific directory names).

       Counter to that, there appears to be PostgreSQL data in:
         /var/lib/postgresql/data (unused mount/volume)

       The suggested container configuration for 18+ is to place a single mount
       at /var/lib/postgresql which will then place PostgreSQL data in a
       subdirectory...

Root cause: Starting from PostgreSQL 18, the official Docker image no longer stores data at /var/lib/postgresql/data. Instead, it uses version-specific paths like /var/lib/postgresql/18/docker. Your mount point is the old path, causing PG 18 to fail.

Fix: Change mount from /var/lib/postgresql/data to /var/lib/postgresql:

[root@openeuler-server ~]# docker stop pg184 && docker rm pg184
[root@openeuler-server ~]# docker run --name pg184 -p 5433:5432 -e POSTGRES_PASSWORD='StrongPass456!!' -v pgdata:/var/lib/postgresql -d registry.highgo.com/postgres/postgres:18.4-bookworm
0ea9e62cd3d0f19a6a791914f2b946ca417ba7247556a4b8f56b3c851118e2ba
[root@openeuler-server ~]#

3.4 Success!​

[root@openeuler-server ~]# docker ps | grep pg18
0ea9e62cd3d0  registry.highgo.com/postgres/postgres:18.4-bookworm  "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:5433->5432/tcp  pg184
[root@openeuler-server ~]#

Check logs to confirm ready:

[root@openeuler-server ~]# docker logs pg184 | tail -10
2026-06-27 09:56:43.153 UTC [1] LOG:  starting PostgreSQL 18.4 (Debian 18.4-1.pgdg12+1) on x86_64-pc-linux-gnu
2026-06-27 09:56:43.170 UTC [1] LOG:  database system is ready to accept connections

🎉 Seeing ready to accept connections — success!

4. Docker Compose for Production​

Single docker run is fine for testing. For long-term testing and pre-production, use Compose for centralized config, easier scaling, and simpler migration.

4.1 Create Work Directory​

[root@openeuler-server ~]# mkdir -p /data/pg-compose && cd /data/pg-compose

4.2 docker-compose.yml (PG 18.4 Compatible)​

version: '3.8'
services:
  pg18:
    image: registry.highgo.com/postgres/postgres:18.4-bookworm
    container_name: pg18
    restart: always
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: PGtest@2026
      POSTGRES_DB: testdb
    volumes:
      - ./pgdata:/var/lib/postgresql  # PG18+: mount root, not /data subdir
    networks:
      pg_net:
networks:
  pg_net:
    driver: bridge

4.3 Port Conflict Again​

[root@openeuler-server pg-compose]# docker-compose up -d
Error response from daemon: Bind for 0.0.0.0:5432 failed: port is already allocated

Fix: Change 5432:5432 to 5435:5432.

4.4 Revised Compose File​

ports:
  - "5435:5432"  # Host port 5435, avoiding 5432/5433

4.5 Restart​

[root@openeuler-server pg-compose]# docker-compose down && docker-compose up -d
[+] Running 2/2
 ✔ Network pg-compose_pg_net  Created  0.2s
 ✔ Container pg18             Started  0.4s
[root@openeuler-server pg-compose]#

[root@openeuler-server pg-compose]# docker logs pg18 | grep "ready to accept"
2026-06-27 10:31:46.188 UTC [1] LOG:  database system is ready to accept connections

5. PostgreSQL 18.4 CRUD Verification​

Once the container logs show database system is ready to accept connections, enter and verify:

5.1 Enter Terminal & Check Version​

[root@openeuler-server pg-compose]# docker exec -it pg18 psql -U postgres
psql (18.4 (Debian 18.4-1.pgdg12+1))
Type "help" for help.

postgres=# select version();
                                                                 version
------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 18.4 (Debian 18.4-1.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
(1 row)

5.2 Complete CRUD​

-- Create table
CREATE TABLE test_user (
    id BIGSERIAL PRIMARY KEY,
    username VARCHAR(50) NOT NULL,
    age INT,
    create_time TIMESTAMP DEFAULT now()
);

-- Insert data
INSERT INTO test_user (username,age) VALUES ('zhangsan',24),('lisi',28),('wangwu',32);

-- Query all
SELECT * FROM test_user;
--  id | username | age |        create_time
-- ----+----------+-----+----------------------------
--   1 | zhangsan |  24 | 2026-06-27 12:29:53.710586
--   2 | lisi     |  28 | 2026-06-27 12:29:53.710586
--   3 | wangwu   |  32 | 2026-06-27 12:29:53.710586

-- Update
UPDATE test_user SET age=29 WHERE username='lisi';

-- Delete
DELETE FROM test_user WHERE username='wangwu';

-- Final result
SELECT * FROM test_user;
--  id | username | age
-- ----+----------+-----
--   1 | zhangsan |  24
--   2 | lisi     |  29

6. Connection Port Reference​

7. High-Frequency Pitfall Summary​

8. Honest Assessment​

Many see China mirrors as just speed boosters, but the real value for PG migration scenarios is bigger:

✅ Advantages:

1. One registry offers both IvorySQL + native PG images — perfect for Oracle syntax comparison and migration assessment without switching registries
2. PG 18.4 syncs the latest community kernel fixes with minimal lag from Docker Hub
3. Images are completely official and unmodified — compliance-friendly for regulated and government environments

Summary​

This complete hands-on test of the registry.highgo.com PostgreSQL 18.4-bookworm image covers the full pipeline: pull, single-machine deployment, production Compose orchestration, and troubleshooting.

Core advantages: domestic high-speed pulls, official unmodified images, one-stop migration comparison with IvorySQL.

Key reminders: always include -bookworm suffix, always mount host data directories to prevent disk-full, prefer single-line commands to avoid parameter errors.

Author note: All operations in this article were performed on an openEuler environment, centered around the IvorySQL community mirror registry.highgo.com for PostgreSQL 18.4-bookworm compatibility verification. PG images iterate quickly — please refer to the official documentation and latest tag list for production deployment. This is personal experience only, not official community guidance.

Author: ShunWah, "shunwah星辰数智社" blog. Holds OceanBase, MySQL, OpenGauss, KingBase, KaiwuDB, TiDB, GBase certifications. YashanDB YVP, KaiwuDB MVP, MoTianLun MVP. Multiple 1st/2nd/3rd prizes in database tech writing competitions.

Based on Oreo Yang's presentation at HOW 2026. Video replay: https://www.youtube.com/watch?v=e5QJUBXgJ7k

1. Project Background​

1.1 What is IvorySQL​

IvorySQL is an open-source relational database built on PostgreSQL with Oracle compatibility, providing flexible, high-performance data management. Key features include Oracle type/PL/SQL/Package support, an active open-source community, and multiple deployment options (traditional, containerized, cloud, sandbox).

1.2 Project Goals​

Database operations face multiple challenges: scattered metrics, difficult diagnostics, high labor costs, cumbersome documentation. The IvorySQL Agent leverages LLM technology for intelligent conversations and automated analysis, building a unified monitoring, diagnostics, and operations platform. Six core problems: intelligent documentation retrieval, database operations, performance tuning, SQL efficiency, enterprise knowledge management, transitioning from "passive search" to "proactive service."

1.3 What is an Agent​

Agent = LLM + Tools + Reasoning. An agent combines language models with tool capabilities to reason about tasks, decide which tools to use, and iterate toward solutions.

IvorySQL Agent tech stack:

• Model Layer: OpenAI GPT, Anthropic Claude, Zhipu AI, and other backends
• Tool Layer: 21 database tools across 7 expert domains
• Framework Layer: LangGraph workflow orchestration with ReAct loop iteration
• Knowledge Layer: Matrix knowledge base covering multiple versions and modes

2. System Architecture​

2.1 Overall Architecture​

Four-layer design:

2.2 Core Data Flows​

Monitoring collection flow: Collector covers 50+ metrics (connections, cache, locks, transactions, queries, table bloat, etc.). Oracle-compatible mode dynamically registers custom type codecs. Data stored as JSONB in PostgreSQL; APScheduler runs every 30s with Delta incremental computation.

Conversation flow: User input routed to corresponding Agent, context built, tool chain invoked for reasoning, response generated.

2.3 Anomaly Detection: Rule Engine + LLM​

Dual-track mechanism: predefined threshold rules trigger LLM analysis; Claude/OpenAI/Zhipu AI for intelligent interpretation; 512-dim local vectors (configurable 256-2048) with pgvector indexing; historical case retrieval reuses diagnostic experience.

3. Key Technical Implementation​

3.1 Two-Stage Smart Routing​

• Phase 1 (Deterministic): Keyword + regex matching, ~0ms, ~70% coverage
• Phase 2 (LLM Classification): LLM intent recognition on rule mismatch, ~300ms
• Post-Confirm: Cache hit reuse, 0ms

Key design decisions: mutually exclusive keywords preventing cross-domain conflicts, bilingual regex, progressive fallback (keyword → LLM → general), mode-aware tool table pruning based on compat_mode, LLM fault tolerance (exponential backoff ×2, timeout fallback to general), domain isolation to prevent tool pollution.

3.2 Local Vector Retrieval (Zero External Dependencies)​

BAAI/bge-small-zh-v1.5 embedding model, 512-dim, fully offline. pgvector + IVFFlat index. Knowledge base includes IvorySQL docs, PostgreSQL docs, and historical analysis. RAG uses "weighted vector search + BM25 reranking + MMR diversification." Document chunks managed by chapter with YAML frontmatter metadata including version and mode tags.

3.3 Context Management & Smart Compression​

Schema cache: 1-hour TTL, auto-invalidate on DDL. Package cache: IvorySQL Oracle mode package info. Real-time metrics: dynamic session, lock, slow query loading. Smart compression for 15+ turn conversations: DDL detection → selective retention (DDL + last 6 turns) → token check → LLM summary → context injection.

3.4 Tool System & Security​

21 built-in tools across domains. Four-layer SQL validation: prompt pre-filtering (80% invalid SQL blocked) → syntax cache verification → pre-confirmation validation → error feedback retry (max 3 attempts). API Token + SHA256 authentication with hot-reload. Audit logging. Oracle/PG mode tool differentiation.

3.5 Chat Flow: Plan · Generate · Validate · Stream​

LLM generates JSON step plans. SSE streaming pushes token, tool_start, tool_end events in real time. State round-tripping maintains multi-step plan coherence across rounds.

3.6 Deployment​

Docker containerized: PostgreSQL storage + Agent service + Grafana + knowledge import. settings.json runtime config with hot updates. Multi-target monitoring. docker-compose one-click launch.

4. Technical Features​

• Fully offline local operation, no external API dependencies
• Multi-RAG domain expansion (7 Agents currently, extensible)
• Intelligent context caching and compression
• Multi-layer security validation and audit logging
• Modular extensible design supporting custom tools and knowledge bases

5. Application Scenarios​

6. Future Plans​

6.1 Short-Term​

• Refine Agent framework with multi-model backend and streaming
• Dual-mode IvorySQL/PostgreSQL compatibility
• Multi-modal input: file analysis, screenshot recognition, voice
• Knowledge base management: online CRUD for entries

6.2 Long-Term​

Multi-version support, automated low-risk repairs, multi-instance management, granular access control, alerting (email/WeChat/Slack), self-learning from user preferences, more domain Agents (security, HA, performance tuning).

Vision: IvorySQL as AI data infrastructure — native pgvector integration, multi-language SDK (3-line RAG), Agent development kit with LangGraph/LlamaIndex templates, one-click Docker Compose deployment.

7. Summary​

The IvorySQL Agent is a systematic exploration of AI-powered database management. Built on the ReAct framework with LangGraph orchestration, it delivers end-to-end automation from natural language to database operations, covering routing, retrieval, context management, tool invocation, and security auditing. As the project evolves toward multi-modal interaction, automated repairs, and an SDK ecosystem, IvorySQL aims to be not just a great database, but the infrastructure for intelligent data management.

A legacy system doesn't just store tables in Oracle. Business logic lives in stored procedures, functions, triggers, and packages. SQL is full of ROWNUM, SYSDATE, DUAL, NUMBER, VARCHAR2. Applications query ALL_TAB_COLUMNS and DBA_PROCEDURES. Then there are subtle behavioral differences: empty strings treated as NULL, implicit type conversions, case rules, NLS parameters.

This is the hardest part of Oracle-to-PG migration: full PG syntax rewrites mean massive application changes and extended testing cycles; superficial SQL rewrites can't cover stored procedures, packages, and data dictionary dependencies.

IvorySQL was created to solve exactly this problem. It's a deeply Oracle-compatible kernel in the PG ecosystem, tracking the latest community releases, and it's open-source and free.

What Is IvorySQL?​

IvorySQL is an open-source database project built on PostgreSQL. Its goal is to provide deep Oracle compatibility while preserving PG's kernel capabilities and ecosystem compatibility. It's not a rewrite, nor is it a string replacement layer — it works at the PostgreSQL kernel, extension, and system object level.

The compatibility scope covers: Oracle mode entry, independent Oracle parser, Oracle data types, implicit conversions, PL/iSQL, Packages, built-in functions, built-in packages, system views, ROWNUM, ROWID, FORCE VIEW, MERGE semantics, sequence behavior, and session-level differences like NLS, empty strings, and identifier case.

This is what sets it apart from typical compatibility layers. IvorySQL focuses on whether your application can run with its original business logic.

How IvorySQL Achieves Oracle Compatibility: From Source Code​

Dual Parser Architecture​

The compatibility entry point is mode isolation. ora_compatible.h defines database types (DB_PG, DB_ORACLE) and parser types (PG_PARSER, ORA_PARSER). Oracle mode defines a specialized search path:

#define ORA_SEARCH_PATH "sys,\"$user\", public"

sys comes first so Oracle-compatible objects are found first. Mode switching is controlled by two variables in ivy_guc.c:

int database_mode = DB_PG;
int compatible_db = PG_PARSER;

The dual parser approach at parser.c uses two function pointers:

raw_parser_hook_type sql_raw_parser = standard_raw_parser;
raw_parser_hook_type ora_raw_parser = NULL;

raw_parser() no longer calls the PG parser directly — it calls (*sql_raw_parser). In Oracle mode, the pointer switches to ora_raw_parser. This is crucial: rather than stuffing Oracle syntax into PG's grammar (a maintenance nightmare), the two parsers diverge at the raw parse stage, then both feed into PostgreSQL's analysis, rewriting, optimization, and execution framework.

ROWNUM Handling​

ROWNUM isn't a normal column or a window function. Its assignment timing depends on scan, qual, sort, and limit phases. IvorySQL handles it at multiple levels:

• parse_expr.c generates RownumExpr when rownum appears in Oracle mode
• execExprInterp.c implements ExecEvalRownum() from executor state
• The optimizer in planner.c converts simple ROWNUM <= N patterns to LIMIT, but only when semantically equivalent

Type System​

IvorySQL adds Oracle-compatible types in pg_type.dat under the sys namespace: number, binary_float, binary_double, blob, clob, raw, long, rowid, urowid, yminterval, dsinterval. These types include I/O functions, typmod, casts, operators, btree/hash/brin opclasses, comparison functions, and sort support.

Implicit type conversion uses Oracle's precedence rules via compatible_oracle_precedence.c, only activated in Oracle mode. This lets string-to-number conversions work like Oracle without affecting PG mode behavior.

PL/iSQL and Packages​

PL/iSQL is implemented in src/pl/plisql/src. Packages are not stored as plain text — IvorySQL adds pg_package and pg_package_body system catalogs with their own indexes and syscache. PackageCache in CacheMemoryContext uses syscache callbacks for invalidation, linking to plan cache on DDL changes.

Compatible Objects​

Through the ivorysql_ora extension and sys schema, IvorySQL provides: Oracle functions (sysdate, add_months, nvl, decode, etc.), built-in packages (dbms_output, dbms_lock, utl_file), and Oracle-style data dictionary views (SYS.DBA_PROCEDURES, SYS.ALL_TAB_COLUMNS).

Additional features: FORCE VIEW, MERGE hooks, Oracle sequence behavior, ROWID/UROWID support at the executor level, empty-string-to-NULL, identifier case switching, and NLS parameters.

Why Oracle Compatibility Doesn't Hurt Performance​

Any feature has its own execution cost. Oracle types, functions, PL/iSQL, and packages can't be free. But IvorySQL's design ensures Oracle compatibility doesn't burden PG queries:

1. Parser divergence at session start: PG sessions use standard_raw_parser; Oracle sessions use ora_raw_parser. No per-query parser switching.

2. Hook triggers are conditional: Type precedence hooks only activate when no suitable operator is found AND in Oracle mode. Package lookups check parser type first.

3. Native object integration: Oracle types use PG's own function calls, expression evaluation, sorting, indexing, and caching. No external rewriting layer.

4. Package caching: PackageCache avoids repeated catalog reads and parsing. Cached objects are invalidated only on DDL changes.

5. Conservative ROWNUM optimization: Conversions only happen when semantically safe.

6. Schema isolation: Oracle objects in sys don't affect PG mode's name resolution.

7. Separate test targets: oracle-check, oracle-installcheck, oracle-pg-check test both modes independently.

In summary: Oracle compatibility isn't a layer placed in front of all SQL — it's placed on specific paths through modes, parser pointers, hook conditions, schemas, extensions, catalogs, and caches. PG queries run as PG. Oracle queries pay Oracle costs only when Oracle semantics are actually used.

Conclusion: IvorySQL's Contribution to the PG Ecosystem​

IvorySQL adds a kernel-level Oracle migration path to the PostgreSQL ecosystem. By handling parsers, types, packages, PL/iSQL, system views, built-in functions, and session semantics at the database level, it lowers the barrier for enterprises to embrace PostgreSQL.

For businesses, this means migration projects don't need massive rewrites from day one. Legacy stored procedures, packages, implicit conversions, and data dictionary dependencies can be partially carried by database compatibility, letting application teams focus on business logic that truly needs refactoring.

For the PG ecosystem, IvorySQL fills the compatibility depth gap that has always been a concern for enterprises coming from Oracle-heavy environments. It gives PG a realistic landing option in finance, telecom, government, and traditional manufacturing — without compromising PG's native experience. PG applications keep running as PG; Oracle migration applications get semantics closer to their original systems.

This path is ecosystem-friendly: it expands PostgreSQL's reach without turning PG itself into an unrecognizable hybrid.

This article is based on Tao Zheng's (IvorySQL Core Contributor) presentation at HOW 2026. Video replay: https://www.youtube.com/watch?v=n2GZRLCiabg

1. The Evolution of Data Architecture​

1.1 Limitations of Traditional Architecture​

Business data must flow through multiple systems before analysis. Two main architectures dominate:

• Linear Architecture: data enters transactional databases → ETL → data lake → data warehouse → AI query. Long pipeline, many processing steps.
• Variant Architecture: data is written simultaneously to transactional and analytical databases, but AI queries still cross multiple systems.

Both face common challenges: complex processing pipelines creating minutes-to-hours latency; data fragmentation between queries; high multi-system operational costs.

1.2 Three Generations of Evolution​

• Gen 1 (2000s — Data Warehouse): Designed for specific business needs, but costly to restructure when new analysis requirements emerge.
• Gen 2 (2010s — Data Lake): Reduced storage costs by storing raw data in cheap lakes, but increased pipeline complexity.
• Gen 3 (circa 2020 — Lakehouse): Merged warehouse and lake benefits, but fundamentally still a two-system combination.

1.3 Native HTAP: The Next-Generation Architecture​

The latest trend is moving from "lakehouse" to "native HTAP"—storing transactional and analytical data in a single database, eliminating ETL entirely.

A single database, single API for AI, no cross-database queries, lower operational costs, real-time data.

2. Three Pain Points of Traditional Architecture in the AI Era​

• Pain Point 1: Unpredictable query patterns — Agent-generated queries are random; traditional architectures can't efficiently handle both point queries and aggregations simultaneously.
• Pain Point 2: Data freshness sensitivity — ETL latency means data may change between queries, unacceptable in risk control, recommendation, and finance.
• Pain Point 3: Cross-system latency stacking — When a single task requires both point queries and aggregations, cross-system latency compounds.

3. Native HTAP: Industry Trends & IvorySQL's Positioning​

3.1 Industry Signals: Databricks' Two Key Acquisitions​

• May 2025 — Acquired Neon: Serverless PostgreSQL provider, signaling OLAP giants entering OLTP.
• October 2025 — Acquired Mooncake Labs: Core project pg_mooncake embeds DuckDB's columnar engine into PostgreSQL, enabling HTAP without ETL. IvorySQL's solution is built on pg_mooncake.

3.2 Two Gaps Left by pg_mooncake​

• Gap 1: Open-source maintenance void — Updates slowed significantly after acquisition. The community needs a vendor-independent implementation.
• Gap 2: Oracle migration capability gap — pg_mooncake never considered Oracle compatibility, yet many Chinese enterprises still run core data on Oracle.

3.3 IvorySQL's Position​

IvorySQL is an open-source PostgreSQL fork, community-maintained, fully PostgreSQL-compatible, with Oracle syntax, data type, and function support. Its HTAP engine:

• Fills Gap 1: Independently maintained pg_mooncake evolution, community-driven roadmap
• Fills Gap 2: Full HTAP capabilities in Oracle migration scenarios
• Real-time lakehouse access engine: one dataset, two query paths, real-time availability

4. Technical Architecture​

4.1 Row vs. Column Storage​

PostgreSQL uses row storage — ideal for transactions. Column storage organizes data by column — ideal for large-scale aggregation. HTAP combines both, automatically selecting the optimal execution path based on query characteristics.

4.2 Core Architecture​

Smart routing: Embedded via PostgreSQL hooks (non-invasive), the Planner identifies query patterns and routes to row store (OLTP) or column store (OLAP).

Dual storage engines: Row store uses PG Heap files (OLTP); column store uses DuckDB + Parquet format (OLAP).

In-memory sharing & auto-sync: The same data is accessible through both row and column paths. Columnar data persists in Parquet format; synchronization between stores is automatic and real-time, completely transparent to the application layer.

4.3 Why ETL is Eliminated​

Traditional: Database → ETL → Warehouse (minutes-to-hours delay). Native HTAP: one copy of data, two access paths — no ETL, no delay.

4.4 Write Path & Query Routing​

1. Application SQL: standard SQL, no modifications needed
2. PostgreSQL query layer: Parser → Planner → Executor, pg_mooncake embedded via hooks
3. Auto routing: Planner identifies query type, auto-selects optimal path
4. Dual engine execution: row → PG Heap; column → DuckDB + Parquet
5. Auto sync: data remains unified across both stores

5. Application Scenarios​

Scenario 1: Oracle Data Warehouse Migration​

Oracle types, syntax, and function behaviors are fully compatible. HTAP capabilities arrive with migration — no need to rebuild the analytics pipeline.

Scenario 2: AI Agent Mixed Queries​

Point queries go through row store, aggregations through column store — one connection for everything. Real-time consistent data, improved Agent response and decision quality.

Scenario 3: Real-Time Risk Control​

Transactions written while column store queries simultaneously. Risk rules trigger in real time, dropping from minutes to sub-second latency.

6. Open Source Release Plan​

• Release: Planned alongside IvorySQL 6.0
• Components: Three core modules — ivy_mooncake, ivy_duckdb, ivy_moonlink

Note: A preview version 1.0 beta1 is already available.

7. Summary​

The IvorySQL HTAP Real-Time Lakehouse Access Engine is a strategic move by the IvorySQL community in database architecture evolution. Built on pg_mooncake and validated by industry leaders like Databricks, it fills the twin gaps of independent open-source maintenance and Oracle migration capabilities. Through row-column store fusion, smart routing, and automatic synchronization, it achieves real-time unification of transactions and analytics in a single database — delivering practical solutions for AI Agent queries, real-time risk control, and Oracle migration. With the open-source release of version 6.0, the IvorySQL community will provide enterprises with a truly open, independently evolvable, Oracle-compatible HTAP infrastructure.

By Yang Yu, PG ACE, 2025 PG Ecosystem Conference Gold Speaker, Harbin User Group core member. Oracle OCM, MySQL OCP, TiDB PCTP, OBCP V4, KCP certified. Specializes in PostgreSQL architecture design, disaster recovery, and performance optimization.

Introduction: The "Best Bridge" in the Database World​

In today's diverse database landscape, enterprises often face a dilemma: embrace PostgreSQL with its rich open-source ecosystem, or stick with Oracle's powerful but costly solutions? The complexity of migration, high licensing fees, and application compatibility concerns often paralyze decision-making.

IvorySQL was born to bridge this gap. Built on PostgreSQL, it maintains 100% compatibility while deeply integrating Oracle syntax compatibility, providing a smooth, low-cost migration path.

Core Positioning: Born from PostgreSQL, Excelling in Oracle Compatibility​

IvorySQL's philosophy: PostgreSQL as the foundation, Oracle compatibility as the core.

Committed to Open Source & PostgreSQL Ecosystem​

The community commits to 100% compatibility with the latest PostgreSQL, preserving the full extension ecosystem (PostGIS, pgvector, pg_cron, etc.).

Powerful Oracle Compatibility​

The key innovation is the ivorysql.compatible_mode GUC parameter. Toggling between oracle and pg modes at the session level enables "one database, two modes."

Deep Technical Analysis: How "Seamless Compatibility" Works​

1. GUC-Driven Dual-Mode Initialization​

# PostgreSQL-only mode
initdb -D /usr/ivory-5/data -m pg

# Oracle-compatible mode (default)
initdb -D /usr/ivory-5/data -m oracle

SHOW ivorysql.compatible_mode;   -- oracle
SET ivorysql.compatible_mode TO pg;

2. PL/iSQL: Seamless Oracle PL/SQL Compatibility​

CREATE OR REPLACE FUNCTION add_numbers(p_a NUMBER, p_b NUMBER) RETURN NUMBER IS
BEGIN
    RETURN p_a + p_b;
END;
/

Full support for anonymous blocks, packages, exception handling, and other Oracle core programming features.

3. Data Types & Functions​

VARCHAR2, NUMBER, NVL, DECODE, SYSDATE, ADD_MONTHS, ROWID — all Oracle-native features:

SELECT SYSDATE FROM dual;
SELECT NVL(NULL, 'Default Value') FROM dual;
SELECT DECODE(1, 1, 'One', 2, 'Two', 'Other') FROM dual;

Also supports sequences, invisible columns, %TYPE/%ROWTYPE attributes, and identifier_case_switch for Oracle-PG case mapping.

Deployment: From Standalone to Cloud-Native​

Yum Install​

sudo dnf install -y ivorysql5-5.3

Docker​

docker pull ivorysql/ivorysql:5.3-ubi8
docker run --name ivorysql -p 5432:5432 -p 1521:1521 \
  -e IVORYSQL_PASSWORD=mysecretpassword -d ivorysql/ivorysql:5.3-ubi8

K8s Operator​

apiVersion: ivory-operator.ivorysql.org/v1beta1
kind: IvoryCluster
metadata:
  name: hippo
spec:
  image: ivorysql/ivorysql:5.3-ubi8
  instances:
    - name: instance1
      replicas: 2

Operations Ecosystem​

Streaming replication for HA, pgMonitor + Grafana for observability, and built-in diagnostics:

EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM large_table WHERE id = 100;
SELECT pid, state, query FROM pg_stat_activity WHERE state = 'active';

Conclusion: A Future-Ready Data Infrastructure​

IvorySQL successfully solves the fundamental Oracle migration barrier: compatibility. It combines Oracle's power with PostgreSQL's flexibility — not to replace either, but to let history and the future coexist on an efficient, reliable platform.

For the full article (Chinese), please visit the Chinese version.

This article is based on the talk given at HOW 2026 — by Tang Cheng, Senior Database Architect, Founder & CTO of Zqshucheng Technology, and author of "The Way to PostgreSQL Mastery".

Recording: https://www.youtube.com/watch?v=HTJEjcgmBtA

Over the past few years, "full cloud migration" was the dominant direction for database infrastructure. But as core business systems continue to scale, more enterprises are turning their attention back to private cloud architectures — especially in finance, government, and telecom, where databases must balance cloud capabilities with data security, resource control, cost optimization, domestic platform adaptation, high availability, and stability. Database private clouds have evolved from simple "resource virtualization platforms" into full-fledged data infrastructure platforms.

Why Choose PostgreSQL / IvorySQL for Private Cloud​

Business Challenges​

As business scenarios grow more complex and data volumes explode, traditional database architectures reveal several pain points:

1. Insufficient elasticity: Long expansion cycles, low resource utilization, unable to adapt to dynamic peaks and valleys.
2. Growing operational complexity: Manual deployment, scaling, and failover can no longer support large-scale cluster management.
3. Security & compliance pressure: Core business data demands higher data sovereignty, access control, and audit oversight.
4. Rising database costs: Commercial license fees, high-end storage costs, and DBA expenses continue to grow — enterprises are paying more attention to TCO.

Private Cloud Advantages & Database Selection​

Private clouds emphasize unified resource management and platform capabilities. In this context, PostgreSQL and IvorySQL have become key choices. PostgreSQL relies on a mature open-source ecosystem, rich extensibility, and enterprise-grade reliability. IvorySQL builds on PostgreSQL compatibility while adding Oracle compatibility, making it especially suitable for domestic substitution and migration scenarios.

IvorySQL Advantages​

As a PG-based open-source database, 100% PG-compatible and highly Oracle-compatible, built for cloud-native environments:

1. 100% PostgreSQL compatible: Built on the latest PostgreSQL versions with all core features.
2. Highly Oracle compatible: PL/iSQL syntax, data types, functions, stored procedures — dramatically reducing Oracle migration costs.
3. Apache 2.0 licensed: Free to use, modify, and distribute without vendor lock-in.
4. Rich cloud-native ecosystem: Full cloud-native toolchain supporting K8s deployment, elastic scaling, and automated operations.
5. Comprehensive Oracle compatibility: 21 Oracle key features including ROWID, %ROWTYPE, %TYPE, NLS parameters, and case sensitivity.
6. Performance & stability: PG 18.0 async I/O, skip scan, and enterprise-level optimizations.

Private Cloud Architecture Design​

The core of a database private cloud is infrastructure capability.

Infrastructure — Bare Metal Servers​

For high-concurrency, high-IO core workloads, bare metal remains a key deployment option — running databases directly on physical machines avoids virtualization overhead.

Key considerations include NVMe SSDs, local high-performance storage, 10GbE/25GbE networking, and RDMA — especially critical for PostgreSQL streaming replication.

Infrastructure — Containers​

For systems demanding maximum resource utilization, Kubernetes (K8S) provides an elastic runtime for IvorySQL instances.

The Operator pattern is the core component for HA deployment:

• Operator: https://github.com/IvorySQL/ivory-operator/
• Cloud Frontend: https://github.com/IvorySQL/ivory-cloud-web
• Cloud Backend: https://github.com/IvorySQL/ivory-cloud

High Availability​

PostgreSQL HA is built on streaming replication. Production environments typically use primary-standby, synchronous replication, and automatic failover.

Database Performance Optimization​

Database performance tuning goes beyond parameter tweaking. Key areas include OS, database parameters, connection pooling, SQL execution efficiency, and storage I/O.

Storage I/O is often the bottleneck — optimization spans RAID, filesystem, kernel scheduling, and disk partitioning:

• RAID: HDD with cache RAID card (RAID10 preferred); NVMe with software RAID
• Filesystem: XFS for production, optimized for high concurrency and large file I/O
• I/O scheduler: noop for NVMe, deadline for regular disks
• Read-ahead & alignment: partition alignment with physical sectors, appropriate read-ahead tuning

Intelligent Monitoring & Operations​

Real-time Monitoring & Alerting​

The core goal of monitoring in a database private cloud is rapid problem detection and diagnosis.

Building Intelligent Monitoring​

Using open-source tools to upgrade monitoring from reactive alerting to proactive prediction:

• Metrics & Alerting: Prometheus Operator with node_exporter and pg_exporter for host and database metrics
• Log Management: EFK or Loki for centralized log collection
• Tracing Integration: Database monitoring integrated with APM (SkyWalking, Jaeger) for end-to-end tracing
• Anomaly Detection: Machine learning on historical metrics for baseline prediction and anomaly detection

Key Metrics​

• Host-level: CPU, memory, disk I/O, network throughput
• Instance-level: active connections, cache hit ratio, TPS, QPS, lock wait count
• Object-level: table CRUD frequency, index usage, index bloat, WAL generation rate
• Session & SQL: slow queries, long transactions, idle transactions

Getting Database Metrics​

PostgreSQL provides built-in system views for granular performance metrics:

• Session waits: pg_wait_events, pg_stat_activity
• WAL: pg_stat_wal
• Checkpointer: pg_stat_checkpointer
• Async I/O: pg_aios
• Memory & NUMA: pg_stat_memory_usage, pg_shmem_allocations_numa
• Vector operations: pg_stat_vector_ops
• Parallel queries: pg_stat_parallel_queries

Automated Operations​

As database scale grows, automation becomes essential — covering infrastructure as code, configuration management, and CI/CD pipelines.

Summary​

As enterprise core workloads continue to evolve toward cloud-native, database private clouds are no longer just about "putting databases in the cloud" — they are moving toward platform-centric, intelligent, and cloud-native infrastructure.

PostgreSQL, with its stability, extensibility, and mature open-source ecosystem, has become a key technical foundation for database private clouds. IvorySQL builds on this foundation with enhanced Oracle compatibility and cloud-native capabilities, opening more possibilities for domestic database adoption and smooth migration.

Operations are shifting from manual to automated, from reactive alerting to intelligent prediction, and from single-instance management to unified platform management. As AI-powered operations, self-service database platforms, and multi-modal data capabilities continue to evolve, database private clouds will gradually become complete data infrastructure platforms.

This article is by Yan Shao'an, PostgreSQL ACE, IvorySQL Contributor, and IvorySQL Expert Advisory Committee member. He holds PGCM, PGCE, PGCA, HGCP, HGCA certifications and writes the WeChat blog "少安事务所" (Shawn's Office), focused on data & AI technology. This article is based on his talk at HOW 2026, originally published on his WeChat blog.

On April 27, HOW 2026 was held at the Shandong Mansion in Jinan. For a recap of the morning main forum, see:

HOW 2026: PostgreSQL & AI Collide in the City of Springs

As an IvorySQL Expert Advisory Committee member, I gave a talk in the afternoon session where I introduced IvorySQL Skills for the first time — a compilation of the IvorySQL operations and development techniques I've refined in my daily work, now open-sourced for everyone. Let's dive into what IvorySQL Skills is and why you need it.

01. In the Agentic AI Era, "Show Me How" Beats "Show Me Why"​

I put three statements in my slides — they capture the motivation behind this project.

First, AI is great at exploring, not decision-making. Let AI draft an initial version, generate test cases, or explain execution plans — but the final decision for production must always be human-reviewed. The problem is: if you don't know "the right way," how can you judge if the AI's suggestion is good?

Second, AI is an amplifier, not a replacement. The stronger your own skills, the better AI amplifies them. If you have a shallow understanding of SQL optimization, you can't judge AI's suggestions either. So a solid foundational skill library must come first.

Third, picking the right tool matters more than using many. Not every AI tool is worth trying, but a production-verified, plug-and-play "tech cookbook" definitely belongs at your fingertips.

This is exactly the positioning of IvorySQL Skills — a "tech cookbook." It's not a repeat of official documentation but a collection of hands-on scripts and configuration templates that I've repeatedly refined and verified over the past two years while migrating from Oracle 19c to IvorySQL, deploying on domestic platforms (LoongArch + KylinOS), and conducting internal team training.

Project repo: https://github.com/shawn0915/ivorysql-skills

Feel free to give it a Star ⭐

If the access speed is slow, use the domestic mirror:

https://gitcode.com/mydb/ivorysql-skills

02. What's Inside IvorySQL Skills​

The entire toolkit is organized into four major modules with 22 skills total. Each skill is a standalone file containing scenario description, applicable versions, step-by-step instructions, and verification commands.

Core Skills​

Development Skills​

Feature Skills​

Auxiliary Skills​

Example: Oracle Compatible Function Quick Reference (Migration Adaptation)

IvorySQL 5.0, based on PostgreSQL 18.0, added 21 Oracle compatibility features. But developers often get stuck on specific functions during actual migration. This skill doesn't list every compatible function — it focuses on the Top 20 most error-prone ones.

-- Scenario: Migrating from Oracle 19c to IvorySQL 5.3,
--            quickly verify commonly used function compatibility
-- Applicable versions: IvorySQL 5.0+ / PG 18.x

-- 1. Date handling: Oracle ADD_MONTHS
SELECT ADD_MONTHS('2026-04-27'::timestamp, 3);  -- Native IvorySQL support

-- 2. String concatenation: Oracle ||
SELECT 'HOW' || '2026' || 'Jinan';  -- Compatible, no rewrite needed

-- 3. NULL handling: Oracle NVL
SELECT NVL(NULL, 'default_value');  -- Compatible

-- 4. Row number: Oracle ROWNUM
SELECT * FROM (SELECT *, ROWNUM() OVER() AS rn FROM pg_tables) t WHERE rn <= 5;

-- Verification: Check the current IvorySQL compatibility mode
SHOW ivorysql.compatible_mode;  -- Returns 'oracle' or 'pg'

03. What Makes a Good Skill​

During my HOW 2026 talk, I also shared the design standards for IvorySQL Skills. Nothing fancy — just four hard requirements that ensure every skill is "ready to use."

04. How to Install & Use: Three Ways, 30 Seconds​

Beyond the three methods below, there's an even simpler path. Just tell your AI coding agent:

Install this skills package: shawn0915/ivorysql-skills

Method 1: npx one-click install (Recommended)

# Install skills manager globally, then pull IvorySQL skills
npx skills add shawn0915/ivorysql-skills

# After install, list available skills
skills list | grep ivorysql

Method 2: npm install

npm i ivorysql-skills

Method 3: Git clone (for customization & internal team use)

git clone https://github.com/shawn0915/ivorysql-skills.git

Directory structure:

ivorysql-skills/
├── SKILL.md                      # Package-level entry (routing hub)
├── README.md                     # Documentation
│
├── ivorysql/                    # Main skill (product overview)
│   └── references/              # Reference docs
├── ivorysql-app-builder/        # End-to-end app dev
│   └── references/              # Reference docs
├── ivorysql-architecture/        # Architecture design
│   └── references/              # Reference docs
...

In an internal environment: fork the Git repo to your company's GitLab. DBAs and developers each contribute their verified skills, forming a team-level "private skill library." When a new hire joins — or when a new AI agent comes online — install this skill pack first, and it covers most daily operations.

05. Closing Thoughts​

At the end of my talk, I shared this perspective: in the Agentic AI era, Skills are not what AI replaces — they're the anchors that give AI its "factual baseline."

What does that mean? If you tell AI to generate an IvorySQL backup script, it might give you a pg_basebackup command with parameters that work for PG 15 but not PG 18, or that ignore the domestic SSL environment. But if you've already installed IvorySQL Skills for your Agent, the model's output has boundaries — it stays grounded in verified practice.

My biggest takeaway from HOW 2026 — after a full morning of keynotes and an afternoon of hallway conversations — is: China's PG ecosystem is shifting from "chasing open source" to "contributing to open source." IvorySQL Skills is a tiny project. No complex code, just a bunch of Markdown docs. But at its core, it exists so that when you use AI coding agents, they understand IvorySQL better. Let your agent know your stack.

Skills are distilled experience. They're agent playbooks. And they're the practical toolkit that levels up your efficiency.

Postscript — May 2, 2026

This post was drafted on April 28, the day after HOW 2026. But on April 30, I saw that Oracle Skills was open-sourced, and I realized there's still much to improve in IvorySQL Skills. For instance, we need to better articulate the differences between IvorySQL and PostgreSQL. And we need more cases, scripts, workflows, and tools for Oracle-to-IvorySQL migration.

To be continued.

IvorySQL: “advanced, fully featured, open source Oracle compatible PostgreSQL with a firm commitment to always remain 100% compatible and a Drop-in replacement of the latest PostgreSQL.”（GitHub）

Also, it’s our team’s favorite engine to translate both foreign SQL and foreign applications for PostgreSQL! We believe in transitions, not migrations - a gradual shift from the old database engine to the new, specifically to reduce risk and optimize results. Using a project like this makes a huge difference in achieving both these goals.

It’s been a little bit since the latest major version was released, but we’re still excited about what’s new and wanted to share.

What’s new​

Version 5.0 (release notes) came out on November 25, 2025 and there’s already been a minor version released since then (5.1 release notes) on December 18, 2025. These editions of the project were the direct result of massive efforts from the IvorySQL team to provide a lot of quality feature improvements as well as implement PostgreSQL 18 compatibility.

IvorySQL 5 brought a lot of necessary changes with it, including:

• PLiSQL - a compatible subset of Oracle PL/SQL
• Oracle-Compatible Package Support
• Oracle-Style Sequence Support
• v5.0 enhancements such as ROWID, %TYPE, %ROWTYPE, and nested subfunctions

In particular, we really enjoy a lot of the functionality upgrades.

For example, IvorySQL now has better NULL handling, where NULL is now treated like an empty string in compatible mode (matching Oracle behavior) to avoid bugs during migration. So, when you have situations such as SELECT CONCAT ('a' || NULL), IvorySQL will now return 'a' instead of following PostgreSQL’s behavior of returning NULL.

One enhancement we really enjoy is that you can now nest functions and procedures. Functions are able to be embedded inside other functions (like Oracle packages, but simpler, using only private methods). This lets you organize complex logic in one place.

Along the same lines, support was also added for DO [ LANGUAGE lang_name ] code [USING IN | OUT | IN OUT, ...].

Trying it out​

Currently, we’re ingesting applications that range from only a few packages, procedures, and functions (around 10-50 objects) to massive data loads that include around 50,000 objects (10,000+ procedures and hundreds of packages).

While there are still many features that will need to be added over time, it considerably reduces the effort taken to complete the transition from Oracle to PostgreSQL successfully.

If you’re thinking about testing it for yourself, there are a plethora of options to get started (including building it from source, using containers, etc.)… and even a WASM build! This in particular allows IvorySQL to be run directly in the browser without having to fully install it locally; it’s very convenient to look at syntax support and take the first step into the world of PostgreSQL.

You can deploy the IvorySQL-WASM project locally in just a few steps, following the blog post IvorySQL published on the subject.

Or, you can try out the hosted WASM on the IvorySQL website.

What’s next​

Hoping to learn more about how to use IvorySQL yourself? Keep an eye on the events page on their website (latest updates), or check out recordings from the HOW2025 conference in Jinan, China in the meantime.

The next conference in Jinan is already scheduled, from April 26th to the 28th! (By the way, the Call for Proposals is open until February 27th, 2026.)

Our team is happy to employ contributors to the IvorySQL project. Some of our team members (Cédric Villemain, Yasir Hussain Shah) have been recognized in the release notes for 5.0 and 5.1 - and we have more additions planned. For the next release, we have planned some exciting developments, including:

• ENABLE/DISABLE constraint syntax
• UTL_FILE package addition
• Oracle-style CREATE TRIGGER body (without needing to create a function beforehand)
• Support for Oracle’s legacy join operator (+)

We’re looking forward to seeing the result of ongoing community work for the next release as well. Many great features are actively being added; for example, recently a new contributor to the project, Rophy Tsai, added the DBMS_OUTPUT and DBMS_UTILITY packages: something widely used across Oracle-based SQL codebases. It was recently merged to IvorySQL’s main branch, and will likely be included in the next minor or major release.

For those looking to transition to or explore PostgreSQL without extensive modifications to their existing database logic and applications, IvorySQL (specially v5.x) offers a practical solution. A direct migration to PostgreSQL could, in some cases, require significant changes to both database objects and application code. IvorySQL helps bridge this gap, allowing a smoother adoption of PostgreSQL with greater compatibility.

Additional resources​

• PostgreSQL.org announcement for IvorySQL 5
• IvorySQL 5.0 roadmap on GitHub

1. PG Asynchronous I/O (AIO) Framework: The First Step to Breaking the Synchronous Blocking Bottleneck​

PostgreSQL 18 has newly introduced an asynchronous I/O subsystem. The new mechanism allows the parallel execution of multiple asynchronous prefetch operations in specific scenarios. The CPU can continue to advance queries without waiting for data to be returned, reducing waiting overhead to a certain extent. This framework lays the foundation for more in-depth and thorough performance optimization of asynchronous I/O in PostgreSQL's future development, marking the first crucial step forward.

Core Scenarios for Improvement​

【Only asynchronous read is implemented at present, with asynchronous write not yet supported】 In all Seq Scan scenarios, parallel sequential prefetch can be achieved through the ReadStream facility adapted for asynchronous I/O, which improves the performance of Seq Scan with better results than the original advisory prefetch of posix_fadvice. Especially in cloud storage scenarios, a single blocking read I/O takes much longer than local I/O, making the advantages of parallel prefetch powered by asynchronous I/O more prominent. At present, asynchronous I/O supports asynchronous reading for sequential scans, bitmap heap scans and VACUUM operations. Early tests show that the performance of read-intensive queries can be improved by 2 to 3 times.

The ReadStream mechanism with asynchronous I/O can asynchronously prefetch subsequent buffers that may be used after receiving a read request. In contrast, with the synchronous I/O method, the system has to wait for the completion of the I/O operation every time a buffer read request is made, which reduces the system throughput.

Usage Method​

# - I/O -

#backend_flush_after = 0		# measured in pages, 0 disables
effective_io_concurrency = 300		# 1-1000; 0 disables issuing multiple simultaneous IO requests
maintenance_io_concurrency = 300	# 1-1000; same as effective_io_concurrency
io_max_combine_limit = 256kB		# usually 1-128 blocks (depends on OS)
                    # (change requires restart)
io_combine_limit = 256kB		# usually 1-128 blocks (depends on OS)

io_method = io_uring # worker, io_uring, sync
                    # (change requires restart)
io_max_concurrency = 128		# Max number of IOs that one process
                    # can execute simultaneously
                    # -1 sets based on shared_buffers
                    # (change requires restart)
#io_workers = 3				# 1-32;

Users can enable asynchronous I/O by selecting three different io_method options:

• worker: Several background I/O workers receive and process I/O requests from backend processes.
• io_uring: The io_uring subsystem in the Linux system processes PG's I/O requests through operating system kernel threads.
• sync: Synchronous I/O that meets the interface requirements of the asynchronous I/O framework.

To enable asynchronous I/O, users need to set the above GUC parameters according to their own situations. Among them, the maximum number of asynchronous I/O handles that each process can have is io_max_concurrency, which users can set to -1 to allow the database to select an appropriate value automatically. If the self-set value is too large, the database may fail to start due to the excessive memory occupied by asynchronous I/O; if the value is too small, the performance of asynchronous I/O cannot be fully utilized.

After starting the database, users can obtain the real-time execution status of asynchronous I/O in the current system through the pg_aios view:

postgres=# select * from pg_aios;
-[ RECORD 1 ]---+-------------------------------------------
pid             | 85834
io_id           | 14208
io_generation   | 204
state           | SUBMITTED
operation       | readv
off             | 116252672
length          | 8192
target          | smgr
handle_data_len | 1
raw_result      |
result          | UNKNOWN
target_desc     | block 14191 in file "base/5/16427"
f_sync          | f
f_localmem      | f
f_buffered      | t
...

For the meaning of each column, please refer to the official documentation.

Framework Design​

PostgreSQL 18 introduces an asynchronous I/O framework that supports flexible configuration of asynchronous I/O through GUC parameters, including the implementation method (io_method, optional worker, io_uring or sync), concurrency scale (such as *_io_concurrency, io_max_concurrency) and implementation-related parameters (such as io_workers).

The framework abstracts the I/O targets (currently supporting smgr, with WAL support planned for the future) and the behaviors of different stages and data sources (shared buffer/local buffer) through the PgAioHandleCallbacks structure to support subsequent expansion. The relevant memory is allocated in the shared memory at startup and will not be scaled subsequently. Processes access their affiliated asynchronous I/O resources by number, and handles are marked for reuse with generation numbers.

At present, the asynchronous I/O of this version mainly provides asynchronous read support for smgr, and does not yet support asynchronous read and write for WAL. The asynchronous write function of smgr is still under development.

Modifications to the Original Facilities​

1. Extend the smgr interface: Add the smgr_startreadv method to support asynchronous reading.
2. Implement callback structures: smgr needs to implement the callback structures of PgAioTargetInfo and PgAioHandleCallBacks.
3. Adapt existing modules: Modules such as smgr and buffer manager need to fill the asynchronous I/O abstract structure to be compatible with the framework.
4. PG critical section processing: Synchronous I/O can be initiated in the PG critical section; due to the segmented execution and callback of asynchronous I/O, operations that may fail in the callback need to be removed (all operations in the critical section cannot fail, such as using RelPathStr instead of the char* string of palloc) to ensure security.
5. Optimize the upper-layer interface: Transform interfaces such as ReadStream by using asynchronous I/O to achieve true prefetch, which greatly improves the I/O performance of operations such as sequential scan, pg_prewarm and ANALYZE, with better results than the original posix_fadvise solution.

Usage Notes and Future Outlook​

1. io_uring requires a relatively new kernel: The old version of the Linux Kernel does not support io_uring. Although some early version kernels support io_uring, their functions and performance are quite different from those of the new kernel.
2. Architecture limits concurrency granularity: Restricted by the multi-process architecture, there are few computing tasks that can run in parallel during PG asynchronous I/O, making it difficult to achieve finer-grained task-level asynchrony. The current main performance gains are concentrated on parallel I/O operations such as ReadStream sequential prefetch.
3. Possible performance improvements in the future: Linux io_uring supports the Direct I/O (DIO) feature, laying the foundation for enabling DIO in PG. Enabling Direct I/O in the future can eliminate double buffering (the OS layer buffers I/O data, and the PG layer also buffers I/O data) to reduce unnecessary data copying. On high-speed NVMe, the IORING_SETUP_IOPOLL option can also be enabled with DIO to use the polling method to check the completion of I/O, which can further improve performance.
4. More asynchronous I/O backends in the future: In addition to the sync mode, the official version of PostgreSQL 18 only supports two asynchronous I/O backends: worker and io_uring. At present, the design of the asynchronous I/O framework is basically complete, and future versions are expected to support I/O backends such as Windows IORing, IOCP and Posix asynchronous I/O, providing users with more choices.

Summary​

The PostgreSQL 18 asynchronous I/O framework improves the I/O capability of the database system and also enhances the scalability of the PostgreSQL architecture. Users only need to modify the GUC parameters according to their own situations to obtain the benefits brought by asynchronous I/O. At present, the design of the asynchronous I/O framework is basically complete, and it will be very convenient to support other asynchronous I/O backends in the later stage. Postgres users on any platform can try io_method = worker or sync. If you want to use the io_uring backend on an old Linux kernel distribution that is not officially adapted to io_uring, you need to conduct sufficient tests before use.

2. Skip Scan: Speed Up B-Tree Indexes​

In versions of PostgreSQL prior to 18, a multi-column B-tree index can be used for query conditions that include any subset of columns in the index, and it is most efficient when constraints are applied to the leading (leftmost) column. An equality constraint on the leading column, plus any inequality constraint on the first column without an equality constraint, is used to limit the portion of the index to be scanned.

For example, given an ascending index based on the non-null fields (a, b, c) and the query condition WHERE a = 5 AND b >= 42 AND c < 77, the index will start scanning from the first entry with a = 5 and b = 42 and continue until the last entry with a = 5. Index entries with c >= 77 will be skipped, but they still need to be scanned.

In principle, this index can be used for queries with constraints on b and/or c but no constraints on a — however, the entire index must be scanned, so the optimizer prefers a sequential table scan over an index scan in most cases.

Core Scenarios for Improvement​

Starting from PostgreSQL 18: If a B-tree index scan can apply Skip Scan, constraints for each column are applied while traversing the index, which can reduce index reads. The working principle of Skip Scan is to internally generate a dynamic equality constraint that matches each possible value in the index column.

Effect Test​

Comparison Versions: PostgreSQL 17 vs PostgreSQL 18

Table Structure and Index​

CREATE TABLE t1
(
c1 int,
c2 int,
c3 float
)
WITH (fillfactor=80);


CREATE INDEX idx_t1_c1c2 ON t1(c1, c2);

Data Generation​

INSERT INTO t1
SELECT (random()*1000)::int, (random()*10000)::int, random()
FROM generate_series(1,1000000) g;

Import Data via COPY​

COPY t1 FROM '/.../t1.csv' WITH (FORMAT csv);

Query Statement: Using the Second Column of the Composite Index​

EXPLAIN ANALYZE SELECT * FROM t1 WHERE c2=100;

PostgreSQL 17 Execution Plan: Selecting Parallel Sequential Scan​

                                                    QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..12986.33 rows=100 width=16) (actual time=1.125..76.076 rows=90 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on t1  (cost=0.00..11976.33 rows=42 width=16) (actual time=1.414..68.624 rows=30 loops=3)
         Filter: (c2 = 100)
         Rows Removed by Filter: 333303
 Planning Time: 0.792 ms
 Execution Time: 76.165 ms
(8 rows)

Disabling sequential scan to force index scan is not an optimal plan and results in slower execution.

                                                        QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_t1_c1c2 on t1  (cost=0.42..18773.42 rows=100 width=16) (actual time=1.846..100.758 rows=90 loops=1)
   Index Cond: (c2 = 100)
 Planning Time: 0.147 ms
 Execution Time: 100.806 ms
(4 rows)

PostgreSQL 18 Execution Plan: Selecting Index Scan, showing a significant improvement in the efficiency of Skip Scan​

                                                        QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_t1_c1c2 on t1  (cost=0.42..3900.84 rows=100 width=16) (actual time=0.225..11.464 rows=90.00 loops=1)
   Index Cond: (c2 = 100)
   Index Searches: 1002
   Buffers: shared hit=3096
 Planning Time: 0.141 ms
 Execution Time: 11.522 ms
(6 rows)

Disabling index scan and bitmap scan to force sequential scan.

postgres=# EXPLAIN ANALYZE SELECT * FROM t1 WHERE c2=100;
                                                      QUERY PLAN
----------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..12986.33 rows=100 width=16) (actual time=1.486..86.881 rows=90.00 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   Buffers: shared hit=6768
   ->  Parallel Seq Scan on t1  (cost=0.00..11976.33 rows=42 width=16) (actual time=2.758..78.712 rows=30.00 loops=3)
         Filter: (c2 = 100)
         Rows Removed by Filter: 333303
         Buffers: shared hit=6768
 Planning Time: 0.141 ms
 Execution Time: 86.926 ms
(10 rows)

Usage Notes​

Skip Scan currently only supports equality comparison conditions.

Summary​

The Index Skip Scan of PostgreSQL 18 enables multi-column BTREE indexes to be used for queries that only reference the second or subsequent index columns with equality, greatly reducing the number of entries that need to be accessed for index scanning and significantly improving its efficiency.

3. Virtual Generated Columns: A Flexible Balance Between Storage and Computation​

The features related to development experience in PostgreSQL 18 focus on simplifying the development process, improving code flexibility, and enabling developers to utilize the capabilities of PostgreSQL more efficiently.

IvorySQL has long been committed to Oracle feature compatibility, including syntax compatibility for virtual columns: column [datatype][generated always] AS (column_expression)[VIRTUAL]

PostgreSQL 18 has finally brought the virtual column feature this time. A virtual column is a table column that does not store data, and its value is dynamically calculated when queried. Compared with stored columns, virtual columns save column storage space, and the value of a virtual column is obtained by calculating the value of the virtual column expression when querying.

Basic Syntax​

The syntax of virtual columns in PostgreSQL 18 is similar to that of stored columns, with the new keyword VIRTUAL added. When the STORED and VIRTUAL keywords are omitted, the column is a virtual column by default. Its syntax is as follows: GENERATED ALWAYS AS ( generation_expr ) [ STORED | VIRTUAL ]

Virtual Column Use Cases​

The identification of a virtual column is indicated in the column constraints. A column is identified as a virtual column through the virtual column constraint syntax. The following are the creation, query and addition of virtual columns for a table with virtual columns:

-- Create a table containing a virtual column, where price_with_tax is a virtual column
CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    price NUMERIC(10,2) NOT NULL,
    tax_rate NUMERIC(5,2) DEFAULT 0.20,
    price_with_tax NUMERIC(10,2) GENERATED ALWAYS AS (price * (1 + tax_rate)) VIRTUAL
);

-- Insert data
INSERT INTO products (name, price, tax_rate)
VALUES ('Laptop', 1000.00, 0.20);

-- Query data (the virtual column is calculated automatically)
SELECT name, price, tax_rate, price_with_tax FROM products;
  name  |  price  | tax_rate | price_with_tax
--------+---------+----------+----------------
 Laptop | 1000.00 |     0.20 |        1200.00
(1 row)

-- Add a virtual column to the table
ALTER TABLE products ADD COLUMN selling_price NUMERIC(10,2)
GENERATED ALWAYS AS (
    price * (1 - 0.2) * (1 + tax_rate)
) virtual;

Brief Analysis of Implementation Principles​

Creation of Virtual Columns​

The storage method of virtual columns in a created table is similar to that of ordinary columns, and the column information of both is stored in the pg_attribute system table. The attgenerated column stores the generated column information: if the value of this column is 's', it means the column is a stored column. The identifier of the newly added virtual column in PostgreSQL 18 in this field is 'v', and the expression of the virtual column is stored in the pg_attrdef system table.

-- View virtual column information, where attgenerated is v indicating that the column is a virtual column
postgres=# select * from pg_attribute where attname='price_with_tax';
-[ RECORD 1 ]--+---------------
attrelid       | 16388
attname        | price_with_tax
atttypid       | 1700
attlen         | -1
attnum         | 5
atttypmod      | 655366
attndims       | 0
attbyval       | f
attalign       | i
attstorage     | m
attcompression |
attnotnull     | f
atthasdef      | t
atthasmissing  | f
attidentity    |
attgenerated   | v
attisdropped   | f
attislocal     | t
attinhcount    | 0
attcollation   | 0
attstattarget  |
attacl         |
attoptions     |
attfdwoptions  |
attmissingval  |

-- View the storage of virtual column expressions, the following expression is the virtual column expression
postgres=# select pg_get_expr(adbin, adrelid) from pg_attrdef where adnum = 5;
             pg_get_expr
-------------------------------------
 (price * ((1)::numeric + tax_rate))
(1 row)

Insert or Update of Virtual Columns​

Since the data of virtual columns does not occupy storage space, any operation that specifies to update or insert virtual columns will be restricted.

-- Specify to insert a virtual column
INSERT INTO products (name, price, tax_rate, price_with_tax) VALUES ('Laptop', 1000.00, 0.20, 1);
ERROR:  cannot insert a non-DEFAULT value into column "price_with_tax"
DETAIL:  Column "price_with_tax" is a generated column.

-- Specify to update a virtual column
update products set price_with_tax = 1 where name = 'Laptop';
ERROR:  column "price_with_tax" can only be updated to DEFAULT
DETAIL:  Column "price_with_tax" is a generated column.

Query of Virtual Columns​

The implementation of querying virtual columns in PostgreSQL 18 is completed at the stage of generating the execution plan. In the logical rewrite and optimization stage, it is judged whether the range table of the query contains virtual columns. If it does, the expression of the virtual column is obtained from pg_attrdef and replaces the original virtual column name. In this way, querying the value of a virtual column is equivalent to calculating the value of its expression, that is, select price_with_tax is equivalent to select (price * ('1'::numeric + tax_rate)) as price_with_tax. It can be seen that the following virtual column is replaced with its expression:

postgres=# explain verbose SELECT name, price, tax_rate, price_with_tax
FROM products;
                              QUERY PLAN
----------------------------------------------------------------------
 Seq Scan on public.products  (cost=0.00..23.12 rows=750 width=76)
   Output: name, price, tax_rate, (price * ('1'::numeric + tax_rate))
(2 rows)

Applicable Scenarios​

• Virtual columns can be used when storage space is a concern, because virtual columns do not occupy disk space.
• Virtual columns need to be used when the value of a column needs to change with the change of dependent columns, because the value of a virtual column is obtained dynamically.
• Virtual columns can be used when the virtual column expression is simple, because querying virtual columns consumes CPU resources, and complex expressions will consume too many CPU resources.
• Since Oracle has the virtual column feature, it is more convenient to migrate Oracle's virtual columns to PostgreSQL.

Unfinished Parts​

Some functions are not yet supported but may be added as incremental functions in subsequent versions:

• Creating indexes on virtual columns or using virtual columns for indexing.
• No unique constraints on virtual columns.
• Extended statistical information on virtual columns.
• Foreign key constraints on virtual columns.
• Non-null constraints on virtual columns (check constraints are supported).
• ALTER TABLE / DROP EXPRESSION.
• Virtual columns cannot have domain types.
• Logical replication does not support virtual columns.

Summary​

Virtual columns are essentially different from ordinary columns and stored columns. Because the values of virtual columns do not occupy disk space, the way to obtain their values is also different from that of ordinary columns and stored columns: ordinary columns or stored columns need to obtain data from the disk, while virtual columns obtain their values through dynamic calculation.

4. Enhanced UUID Functions: Improved Orderliness and Usability​

The Disorder of Traditional UUIDs: The Main Pain Point of Using It as a Primary Key​

The complete randomness of traditional UUIDs (especially v4) is the pain point of using them as database primary keys:

• UUIDs are generated randomly with uncertain insertion positions, leading to frequent splitting and reorganization of the index tree and a significant reduction in write performance.
• It destroys the physical storage order of clustered indexes (such as InnoDB) and increases disk I/O.
• The efficiency of range queries and sorting is low with poor performance.

Key Breakthrough of UUIDv7: Time-Ordered Architecture Design​

UUIDv7 solves the problem of completely random UUID generation by introducing a timestamp in the high-order part of the UUID, enabling newly generated UUIDs to be naturally sorted by creation time. In this way, B-tree indexes can perform sequential insertion like auto-incrementing integers, while still maintaining the advantages of global uniqueness and distributed generation of UUIDs.

This feature makes UUIDv7 as a primary key have the following prominent advantages:

• Strictly increment according to the order of creation time.
• Reduce index fragmentation.
• Improve cache hit rate.
• Suitable for scenarios with high-concurrency insertion and efficient query.

Structural Design of UUIDv7​

Analysis of Key Design Points:

• High-precision time prefix (48 bits): Unix timestamp accurate to milliseconds to ensure that IDs are strictly incremented by time (NTP clock synchronization is required).
• Trailing random bits (62 bits): Ensure distributed uniqueness and avoid the risk of MAC address leakage in version v1.

How Orderliness Solves Performance Problems?

• B-Tree index optimization: Newly generated UUIDv7 is always larger than the previous value, so it is appended to the end of the index to avoid splitting of intermediate nodes.
• Buffer pool friendly: Sequential writing makes new records concentrate on a small number of data pages. When a page is full, the database only needs to allocate new pages for appending, reducing the elimination of old pages and disk I/O.
• Range query acceleration: Time orderliness enables WHERE id > '2025-06-01' to be converted into timestamp range filtering, greatly reducing the scanning range.

How to Apply UUIDv7 in PostgreSQL 18​

PostgreSQL 18 has introduced a number of new functions to support UUIDv7, facilitating the generation, operation and extraction of UUID information.

1. uuidv7() function: Used to generate a new UUIDv7 value

-- Generate UUIDv7 using the current timestamp
SELECT uuidv7();
-- Output example: 0197f96c-b278-7f64-a32f-dae3cabe1ff0

-- Generate UUIDv7 for 1 hour ago
SELECT uuidv7(INTERVAL '-1 hour');
-- Generate UUIDv7 for 30 minutes later
SELECT uuidv7(INTERVAL '30 minutes');

2. uuidv4() function: As an alias for the existing function gen_random_uuid(), easy to use with uuidv7

-- The two are equivalent
SELECT gen_random_uuid();
SELECT uuidv4();

3. uuid_extract_timestamp() function: This function now supports UUIDv7 (originally only supported UUIDv1)

-- Extract timestamp from UUIDv7
SELECT uuid_extract_timestamp(uuidv7());
-- Example output: 2025-09-18 12:20:49.409+00

4. uuid_extract_version() function: Used to detect the version of UUID:

-- Check UUID version
SELECT uuid_extract_version(uuidv7());  -- Returns 7
SELECT uuid_extract_version(uuidv4());  -- Returns 4

Using UUIDv7 as the primary key in the PostgreSQL database:

-- Create a table with a UUIDv7 primary key
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT uuidv7(),
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(100) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

In this way, the id of each new record will be automatically assigned a UUID sorted by timestamp.

-- Insert data
INSERT INTO users (username, email)
VALUES ('alice', 'alice@example.com');
INSERT INTO users (username, email)
VALUES ('bob', 'bob@example.com');

-- View by UUID time order
SELECT id, username, uuid_extract_timestamp(id) as uuid_timestamp
FROM users
ORDER BY id;

Performance Advantages​

1. The timestamp order of UUIDv7 can significantly reduce page splitting and cache invalidation, effectively improving the efficiency of B-tree indexes.

-- Create a performance test table
CREATE TABLE performance_test (
    id_v4 UUID DEFAULT uuidv4(),
    id_v7 UUID DEFAULT uuidv7(),
    data TEXT DEFAULT 'sample data'
);
-- Use UUIDv7 for indexing
CREATE INDEX idx_v4 ON performance_test (id_v4);
CREATE INDEX idx_v7 ON performance_test (id_v7);

After batch insertion, you can use pg_statio_user_indexes to view the index hit status, and UUIDv7 usually performs better.

2. UUIDv7 has built-in time sorting, which significantly improves sorting performance in most scenarios.

-- Utilize the natural sorting of UUIDv7
SELECT id_v7, data
FROM performance_test
ORDER BY id_v7
LIMIT 10;

Compared with the random order of UUIDv4, the query results of UUIDv7 are returned in the order of creation, which is more intuitive.

UUIDv7 Best Practices​

Scenarios suitable for using UUIDv7:

• Multi-tenant applications: UUIDv7 can be used as the primary key, and a composite index for (tenant_id, id) can be created to maintain both uniqueness and time sorting.
• Distributed systems: Multiple services can independently generate UUIDv7 and still maintain time order on a global scale.

Limitations and Notes​

• Dependence on system clock: Time synchronization mechanisms such as NTP need to be enabled to avoid clock drift.
• Timestamp precision: UUIDv7 is in milliseconds. When multiple UUIDs are generated in the same millisecond, the order may not fully reflect the actual creation order, but uniqueness is still maintained.
• Migration planning: When migrating from UUIDv4 to UUIDv7, it is necessary to check the application logic, indexes and external dependencies.

Summary​

PostgreSQL 18's support for UUIDv7 solves the performance bottleneck of using UUID as a primary key. While maintaining global uniqueness, UUIDv7 has the orderliness similar to auto-incrementing integers, making B-tree insertion more efficient and queries faster.

For modern applications that require distribution, high concurrency and high performance, UUIDv7 provides a practical solution that balances uniqueness and performance.

5. Enhanced EXPLAIN: Intuitive Presentation of Execution Details​

PostgreSQL 18 has made a major upgrade to the EXPLAIN command. By providing richer and more intuitive execution plan information, it enables database developers and DBAs to perform query performance analysis and optimization more easily.

Automatic Buffer Analysis​

EXPLAIN ANALYZE now includes BUFFER statistics by default without the need to manually add the BUFFERS option:

• Shared Hits: Displays the number of data blocks read from the cache, reflecting the memory usage efficiency.
• Shared Reads: Identifies the data blocks that must be read from the disk, helping to identify I/O bottlenecks.
• Shared Dirtied: For data modification operations, displays the number of blocks that have been changed.

Refined Index Monitoring​

New index scan count statistics are added to allow developers to accurately understand the efficiency of index usage:

-- Example output showing index usage
Index Scan using orders_pkey on orders
Index Searches: 1  -- Clearly displays the number of index lookups
Buffers: shared hit=2 read=2

Enhanced Statistical Information​

• Support for fractional row counts to provide more accurate row count estimates.
• Output memory and disk usage details for Material, Window Aggregate and CTE nodes.
• Display detailed parameter information in window functions.
• Show worker cache statistics for Parallel Bitmap Heap Scan.
• Output disabled nodes.
• Output WAL buffer information.

Basic Query Analysis​

-- Create a test table
CREATE TABLE orders (
    order_id SERIAL PRIMARY KEY,
    customer_id INTEGER NOT NULL,
    order_date DATE NOT NULL,
    total_amount DECIMAL(10, 2) NOT NULL
);
CREATE INDEX idx_orders_customer_id ON orders(customer_id);

-- Insert test data
INSERT INTO orders (customer_id, order_date, total_amount)
SELECT
    (n % 10) + 1,
    CURRENT_DATE - (n % 365),
    (50 + (random() * 950))::decimal(10,2)
FROM generate_series(1, 50000) n;

-- View the enhanced execution plan
EXPLAIN ANALYZE
SELECT *,sum(total_amount) OVER (PARTITION BY customer_id)
FROM orders WHERE order_id>49900;

Execution Plan:

QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
 WindowAgg  (cost=13.46..15.04 rows=99 width=50) (actual time=0.630..0.745 rows=100.00 loops=1)
   Window: w1 AS (PARTITION BY customer_id)
   Storage: Memory  Maximum Storage: 17kB
   Buffers: shared hit=5 read=2
   ->  Sort  (cost=13.30..13.55 rows=99 width=18) (actual time=0.307..0.333 rows=100.00 loops=1)
         Sort Key: customer_id
         Sort Method: quicksort  Memory: 28kB
         Buffers: shared hit=5 read=2
         ->  Index Scan using orders_pkey on orders  (cost=0.29..10.02 rows=99 width=18) (actual time=0.089..0.174 rows=100.00 loops=1)
               Index Cond: (order_id > 49900)
               Index Searches: 1
               Buffers: shared hit=2 read=2
 Planning:
   Buffers: shared hit=64 read=22
 Planning Time: 2.080 ms
 Execution Time: 1.343 ms

Insights from Execution Plan Output:

• Buffer usage (cache hits vs disk reads): 64 shared buffers were accessed from the cache and 22 were read from the disk when generating the execution plan; 5 shared buffers were accessed from the cache and 2 were read from the disk during execution.
• Index efficiency statistics: 1 orders_pkey index scan was executed, and 2 shared buffers were accessed from the cache, clearly showing the efficiency of index usage.
• Memory usage details of window functions: 17kB of disk space was used.
• Accurate row statistics.
• Detailed parameters of window functions.

WAL Log Analysis​

EXPLAIN (ANALYZE, WAL)
INSERT INTO orders (customer_id, order_date, total_amount)
SELECT
    (n % 10) + 1,
    CURRENT_DATE - (n % 365),
    (50 + (random() * 950))::decimal(10,2)
FROM generate_series(1, 50000) n;

Execution Plan:

QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------
 Insert on orders  (cost=0.00..2000.00 rows=0 width=0) (actual time=767.116..767.118 rows=0.00 loops=1)
   Buffers: shared hit=299156 read=2 dirtied=500 written=501
   WAL: records=152158 bytes=10427828 buffers full=139
   ->  Subquery Scan on "*SELECT*"  (cost=0.00..2000.00 rows=50000 width=28) (actual time=5.742..336.699 rows=50000.00 loops=1)
         Buffers: shared hit=50013
         WAL: records=1516 bytes=150084 buffers full=2
         ->  Function Scan on generate_series n  (cost=0.00..1750.00 rows=50000 width=24) (actual time=5.460..227.650 rows=50000.00 loops=1)
 Planning Time: 0.114 ms
 Execution Time: 767.179 ms

WAL Statistics:

• Monitor the log generation volume of write load: 1516 logs were generated in the WAL buffer with a total of 150084 bytes of data.
• Diagnose write performance bottlenecks: The buffer was full 2 times.

Parallel Query Optimization​

EXPLAIN (ANALYZE)
SELECT * FROM orders WHERE customer_id IN (1, 2, 3, 4, 5, 6);

Execution Plan:

QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=2752.40..10357.99 rows=327855 width=18) (actual time=22.375..121.296 rows=330000.00 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   Buffers: shared hit=3810
   ->  Parallel Bitmap Heap Scan on orders  (cost=2751.40..10029.13 rows=136606 width=18) (actual time=12.868..88.329 rows=110000.00 loops=3)
         Recheck Cond: (customer_id = ANY ('{1,2,3,4,5,6}'::integer[]))
         Rows Removed by Index Recheck: 53967
         Heap Blocks: exact=170 lossy=566
         Buffers: shared hit=3810
         Worker 0:  Heap Blocks: exact=387 lossy=957
         Worker 1:  Heap Blocks: exact=369 lossy=1055
         ->  Bitmap Index Scan on idx_orders_customer_id  (cost=0.00..2669.44 rows=327855 width=0) (actual time=21.219..21.220 rows=330000.00 loops=1)
               Index Cond: (customer_id = ANY ('{1,2,3,4,5,6}'::integer[]))
               Index Searches: 1
               Buffers: shared hit=266
 Planning:
   Buffers: shared hit=30
 Planning Time: 0.510 ms
 Execution Time: 158.523 ms

Insights into Parallel Execution Efficiency:

• Detailed cache statistics for each worker process: Worker 0 hit 387 exact blocks and 957 lossy blocks, and Worker 1 hit 369 exact blocks and 1055 lossy blocks.
• Exact block and lossy block analysis: The appearance of lossy blocks indicates that work_mem may be too small to allow the bitmap to accurately locate tuples.

Technical Advantages and Value​

Real-Time Performance Diagnosis​

• Lower the threshold: Automated buffer statistics allow beginners to quickly identify I/O problems.
• In-depth insights: Provide more fine-grained performance data for expert users.
• Comprehensive coverage: Obtain multi-dimensional information such as execution plans, cache usage and index efficiency with a single command.

Optimization Guidance​

• Index optimization: Avoid over-indexing or under-indexing through accurate index usage statistics.
• Memory tuning: Guide the adjustment of work_mem parameters according to the frequency of lossy blocks.
• Query rewriting: Optimize the SQL statement structure based on detailed execution cost data.

Future Outlook of the Function​

Although the enhanced EXPLAIN in PostgreSQL 18 has brought significant improvements, there are still some aspects that can be further improved:

1. Output readability: With the increase in information volume, the output becomes more complex and may require better formatting or visualization tool support.
2. Historical comparison: The lack of a built-in mechanism for direct comparison with historical execution plans makes performance regression analysis still rely on external tools.
3. Threshold alert: There is no built-in mechanism to issue warnings for abnormal values (such as an abnormally high number of buffer reads), which requires manual analysis.
4. Execution plan visualization: Text-based output is still difficult to understand intuitively in complex queries and needs to be supplemented by third-party tools.

Summary​

The enhanced EXPLAIN in PostgreSQL 18 represents a major advance in database observability. By automatically collecting key performance indicators and providing more in-depth execution insights, it significantly lowers the threshold for query optimization and provides more powerful analysis capabilities for experienced DBAs at the same time.

6. OAuth 2.0 Authentication Support: Strengthen the Data Protection Barrier​

In terms of security, IvorySQL is committed to adding a variety of national cryptographic authentication functions to ensure data security. PostgreSQL 18 has further strengthened identity authentication by introducing support for OAuth 2.0, an open standard authorization protocol used to authorize an application or service to access a user's resources in another application without providing a username and password.

This feature mainly includes the following core elements:

• OAuth2 validator framework: Provides an extensible framework that enables PostgreSQL to integrate with OAuth 2.0 providers. PostgreSQL itself does not implement specific token verification algorithms (such as JWT verification), but delegates this work to an external shared library (*.so file).
• Client authentication support: libpq (PostgreSQL's C client library) now supports the OAuth 2.0 authentication process.
• Custom verification logic: Allows the implementation of custom token verification and user mapping logic through a callback mechanism.

Configure OAuth Authentication Method​

Server-Side Configuration​

1. Selecting the OAuth authentication method is similar to selecting the national cryptographic authentication method in the Highgo Database. You need to specify the METHOD as oauth in the pg_hba.conf file to enable OAuth authentication.

   At the same time, the OPTIONS must specify the issuer and scope parameters, in addition to several optional parameters: validator, map, delegate_ident_mapping. The following is a minimal configuration example:

   local all test oauth issuer="http://127.0.0.1:9000" scope="openid postgre"
   

2. Specify the external OAuth validator, configure the newly provided oauth_validator_libraries parameter in the postgresql.conf file, and the configuration content is the library file provided by the OAuth validator.

Client-Side Configuration​

The client needs to specify the following connection parameters to achieve connection:

• oauth_issuer: Required parameter, an HTTPS URL that is the issuer identifier of the authorization server.
• oauth_client_id: Required parameter, the OAuth 2.0 client identifier issued by the authorization server.
• oauth_client_secret: Optional parameter, the client password to be used when accessing the OAuth authorization server.
• oauth_scope: Optional parameter, the scope of the access request sent to the authorization server, specified as a space-separated list of OAuth scope identifiers.

Authentication Implementation Principle​

The overall OAuth authentication process is roughly as shown in the figure below:

Client (libpq)​

PostgreSQL implements a non-blocking, state machine-based asynchronous network client. The state machine includes the following states: OAUTH_STEP_INIT, OAUTH_STEP_DISCOVERY, OAUTH_STEP_DEVICE_AUTHORIZATION, OAUTH_STEP_TOKEN_REQUEST, OAUTH_STEP_WAIT_INTERVAL. Its core principles include the following parts:

• DISCOVERY: The client obtains the authorization server metadata from the user request.
• DEVICE_AUTHORIZATION: The client sends a request to the authorization server, which returns a device_code and a verification_uri. The client outputs the message "Visit xxxxxx and enter the code: xxxxxx" to prompt the user to perform operations.
• TOKEN_REQUEST and WAIT_INTERVAL: Poll the authorization server until the user completes the authorization, and the authorization server returns an access_token to the client.
• Set the obtained access_token in the connection object. libpq will send it to the PostgreSQL server as a password, and the OAuth validator on the server side will be responsible for verifying this token.

Server-Side​

The PostgreSQL server processes the OAuth authentication process, also implemented through a state machine, but it is much simpler than the client, with a total of three states: OAUTH_STATE_INIT, OAUTH_STATE_ERROR, OAUTH_STATE_FINISHED. The following are the core steps:

• First, parse the message sent by the client, whose format complies with Section 3.1 of RFC 7628.
• Extract the pure Bearer Token from the client message and verify its format (whether it is a valid Base64 string).
• Pass the extracted token to the validator module for substantive verification.
  • Verification successful: The state is converted to OAUTH_STATE_FINISHED, and PG_SASL_EXCHANGE_SUCCESS is returned. Proceed with the subsequent operations of establishing a connection.
  • Verification failed: Generate a JSON error response compliant with Section 3.2.2 of RFC 7628, informing the client of the required scope and where to obtain the token. The state is converted to OAUTH_STATE_ERROR, and PG_SASL_EXCHANGE_CONTINUE is returned, waiting for the client to send the final KVSEP to end the failed handshake.

External Validator​

An external validator usually needs to handle the following matters:

• Token verification: It can be done through online verification and local verification, determined by the validator itself. In online verification, the validator usually sends the token to the dedicated Introspection Endpoint of the authorization server, and the authorization server returns a JSON response to inform whether the token is valid. Local verification requires the validator to implement a set of verification processes internally to verify the signature and validity period of the token locally. The advantage of local verification is fast response, but the disadvantage is that token revocation cannot be detected in real time.
• Identity mapping: After successful verification, the validator needs to extract the unique user identifier from the token and convert it into an identity identifier understandable by the database, that is, a database user.
• Connection decision: If the token is within the validity period and there is a corresponding database user mapping relationship, a session connection is created with the identity of the user.

Advantages and Disadvantages Analysis​

Advantages:

1. OAuth2 provides a modern and standardized authentication mechanism to improve security. Through OAuth2 authentication, the security risk of password exposure during data transmission in traditional password authentication is avoided.
2. It simplifies database user management, supports unified identity policies and access control, and improves management efficiency.

Disadvantages:

1. Compared with traditional password authentication, the implementation is more complex and requires additional configuration and maintenance work, including the setting and management of the OAuth2 provider, which increases the complexity of operation and maintenance. For example, the national cryptographic authentication function of Highgo Database also only requires very simple configuration to use while ensuring password security.
2. It depends on the availability and reliability of the external OAuth provider. If the OAuth provider has problems, it may affect database access.
3. Each connection may require additional network requests to verify the token, which may increase the time to establish a connection, especially in high-concurrency scenarios.

Summary​

The OAuth2 support introduced by PostgreSQL 18 is an important security enhancement feature that allows organizations to use modern authentication mechanisms to protect database access. By providing a flexible validator framework and callback mechanism, PostgreSQL 18 can adapt to various OAuth2 deployment scenarios and business requirements.

Although OAuth2 authentication increases the complexity of the system, it provides significant security advantages, especially in centralized identity management and single sign-on. It is a worthwhile new feature for organizations that want to adopt modern security best practices. OAuth authentication does not conflict with existing authentication methods, and you can still choose methods such as the national cryptographic authentication provided by Highgo Database when you do not want to perform tedious configuration.

When implementing OAuth2 authentication, organizations should carefully evaluate their needs, existing infrastructure and technical capabilities to ensure successful deployment and operation and maintenance. At the same time, factors such as performance, availability and compatibility should be considered to provide the best user experience and system reliability.

Summary​

PostgreSQL 18 has achieved an all-round upgrade in performance, functions and security with six core features:

• Asynchronous I/O breaks the synchronous blocking bottleneck and improves the throughput of read-intensive scenarios.
• Skip Scan makes multi-column B-tree indexes play an efficient role in non-leading column queries.
• Virtual generated columns find a flexible balance between storage and computation and optimize the development experience.
• UUIDv7 solves the performance pain point caused by the disorder of traditional UUIDs and balances uniqueness and orderliness.
• Enhanced EXPLAIN provides more intuitive and detailed execution insights for query optimization.
• OAuth 2.0 authentication builds a modern protection barrier for data security.

These features not only meet the current database requirements for high performance, high concurrency, easy development and strong security, but also lay a solid foundation for future development in cross-platform adaptation and function expansion. They further consolidate PostgreSQL's leading position in the open source database field and provide more powerful and flexible technical support for various application scenarios.

To lower the usage barrier for community users and enhance their experience, the IvorySQL community has launched the IvorySQL-WASM project, enabling users to quickly access online demos.

IvorySQL-WASM is developed based on the open-source Postgres-WASM framework. It allows users to experience IvorySQL directly in web browsers without local installations or complex deployments. By combining WebAssembly and virtualization technologies, IvorySQL-WASM provides a simple, lightweight, and accessible way for developers, testers, and database enthusiasts to effortlessly explore the powerful features of IvorySQL.

Technical Analysis​

Core technologies of IvorySQL-WASM:

• V86: An x86 emulator that simulates the hardware environment required for virtual operating systems, including CPU, memory, VGA, BIOS, and serial ports.

• BuildRoot: Used to build a minimal Linux system image running on V86, integrated with IvorySQL 4.4 services and a stable runtime environment.

• WebAssembly (WASM): A low-level, assembly-like binary instruction format. Tools like Emscripten compile C/C++ code into WASM modules, enabling the migration of existing C library ecosystems to the web platform. With its compact binary format and near-native performance, WASM serves as the core technology for running database services in the IvorySQL-WASM project.

Quick Deployment​

Developers can follow these steps to deploy the IvorySQL-WASM project locally:

Linux Environment​

1. Install dependencies: Node.js and npm.
2. Clone the project: git clone git@github.com:IvorySQL/ivorysql-wasm.git
3. Navigate to the package/runtime directory and start the HTTP server: npx serve
4. Access via browser: http://server_ip:3000

Windows Environment​

1. Download and install Node.js.
2. Execute npx.cmd serve in the corresponding PowerShell directory to start the HTTP server.

Experience​

By accessing http://192.168.31.186:3000/, users can perform online IvorySQL operations, such as creating a t1 table and listing tables.

When running the minimal Linux virtual machine, the browser first downloads the system image files (~35MB) and state snapshot (15MB). Thus, the initial page load may take several seconds to over 20 seconds, depending on whether the HTTP server is hosted internally or externally. The state snapshot enables rapid loading of preconfigured databases.

Task Manager observations show that IvorySQL’s browser-based operation consumes CPU and memory resources efficiently.

Notes​

1. Databases are stored in memory. Refreshing the page clears the database and restores the system to its initial state.

2. IvorySQL uses dual ports. The default connection port is 1521. To log in via port 5432, exit psql in the shell terminal and execute:

   psql -U postgres -p 5432 -d postgres
   

   

3. To save the current database: Click "Save state to file". The browser will automatically generate and download a v86state.bin file. To reuse the saved database: Click "Restore from file" on the page and upload the v86state.bin file.

4. The V86 system has a minimum memory configuration of 128 MB (current setting).

Developers can modify the source code to meet specific needs.​

1. For BuildRoot, refer to package/buildroot/README.md.
2. For V86, refer to https://github.com/copy/v86/releases/tag/latest.
3. For the web interface, refer to package/runtime/README.md.

Community Edition Coming Soon​

The IvorySQL-WASM Community Edition will be released alongside IvorySQL 4.5. For details, follow the IvorySQL Official Website.

IvorySQL v4 introduces block-level incremental backup and incremental backup merge features aimed at optimizing the database backup and recovery process. By supporting incremental backups through the pg_basebackup tool, storage requirements and backup times are significantly reduced. Additionally, the pg_combinebackup tool can merge multiple incremental backups into a single full backup, enhancing the flexibility and efficiency of data recovery.

2. Feature Introduction​

IvorySQL’s incremental backup works by recording data changes during each checkpoint cycle, ensuring that only the data blocks changed since the last backup are included. This mechanism not only reduces the required storage space but also shortens I/O operation times during the backup process. Furthermore, the pg_combinebackup tool supports merging multiple incremental backups into one full backup, eliminating the need to apply incremental backups one by one during data recovery, thus simplifying the recovery process.

2.1 Enabling Incremental Backup Feature​

To enable the incremental backup functionality, the relevant parameters must first be configured in the database, and the configuration file must be reloaded:

ALTER SYSTEM SET summarize_wal = ON;
SELECT pg_reload_conf();

Then, create a test table and insert initial data for subsequent backup operations:

CREATE TABLE sample_table(t INT);
INSERT INTO sample_table VALUES (1);
SELECT * FROM sample_table;

2.2 Performing Full Backup​

Use the pg_basebackup tool to perform a full backup and generate a backup file containing the backup_manifest for use as the base for future incremental backups:

pg_basebackup -Fp -D /backup/\$(date +%Y-%m-%d*%H%M%S-FULL)

This command saves the full backup data to the specified directory and records the current backup state.

2.3 Performing Incremental Backup​

(1) After modifying the data, perform an incremental backup to record changes since the last full or incremental backup.

Modify the table data and perform the first incremental backup:

UPDATE sample_table SET t = 2;
SELECT * FROM sample_table;

Execute the incremental backup:

pg_basebackup -Fp -D /backup/\$(date +%Y-%m-%d*%H%M%S-INCREMENTAL) -i /backup/2025-02-20_161530-FULL/backup_manifest

(2) Modify the table data again and perform the second incremental backup.

UPDATE sample_table SET t = 3;
SELECT * FROM sample_table;

Then perform the second incremental backup:

pg_basebackup -Fp -D /backup/\$(date +%Y-%m-%d*%H%M%S-INCREMENTAL) -i /backup/2025-02-20_161808-INCREMENTAL/backup_manifest

2.4 Merging Incremental Backups​

Merge the full backup and multiple incremental backups to create a new full backup for subsequent recovery operations:

cd /backup
pg_combinebackup 2025-02-20_161530-FULL 2025-02-20_161808-INCREMENTAL 2025-02-20_162115-INCREMENTAL -o 2025-02-20_66666-FULL

Note: The full backup must be listed first in the parameter list, and the incremental backups should be ordered chronologically to ensure data recovery integrity and consistency.

2.5 Restoring Merged Backup​

Start the database and verify the integrity of the data:

pg_ctl start -D /backup/2025-02-20_66666-FULL -o '-p 5435'
SELECT * FROM sample_table;

2.6 Merging Specific Incremental Backup​

If you need to restore to a specific incremental backup state, you can merge up to a specific intermediate incremental backup:

pg_combinebackup 2025-02-20_161530-FULL 2025-02-20_161808-INCREMENTAL -o 2025-02-20_77777-FULL

Then start the database and verify the data:

pg_ctl start -D /backup/2025-02-20_77777-FULL -o '-p 5436'
SELECT * FROM sample_table;

3. Considerations​

Backup Sequence​

Ensure that the full backup is listed first in the merge command, with incremental backups ordered chronologically to avoid data inconsistency during the merge process.

Incremental Backup Storage Management​

While incremental backups reduce storage space requirements, recovery operations depend on the complete incremental backup chain. Any loss of an incremental backup will result in recovery failure. Therefore, it is recommended to regularly check and merge incremental backups to ensure the integrity and reliability of the backup chain.

Performance Monitoring​

Monitor the database performance when performing incremental backup and merge operations to ensure that the impact on system performance remains within acceptable limits, avoiding backup operations during peak load periods.

4. Summary​

The block-level incremental backup and pg_combinebackup tool provided by IvorySQL significantly enhance the efficiency of database backup and recovery. By backing up only changed data blocks, incremental backups show excellent performance in reducing storage usage and shortening backup times. The incremental backup merge feature further simplifies the recovery process, making operations more convenient.

These advanced features make IvorySQL an ideal choice for large-scale databases and environments with strict data recovery requirements, offering flexible and efficient backup and recovery solutions to ensure data security and availability.

Replication Slot Synchronization Mechanism​

Prerequisites​

1. Physical Replication Slot: There must be a physical replication slot between the primary database and the backup database, serving as the foundation for synchronizing logical replication slots and ensuring effective data transfer between them.

2. Configuration Parameters:

   • Configure the primary_slot_name parameter on the backup database and specify the replication slot when using the pg_basebackup tool. This configuration step ensures that the backup database can correctly receive data from the primary database.
   • Enable the hot_standby_feedback feature on the backup database to ensure it can receive and send back WAL logs. This feature guarantees the activity of the backup database during data reception and maintains the integrity of its feedback mechanism.
   • Specify a valid database name (dbname) in the primary_conninfo parameter. This configuration ensures that the target database for the replication process is clear and correct.

3. Recommended Configuration: Configure the standby_slot_names parameter on the primary database to maintain consistency in replication slot synchronization. This configuration helps maintain a consistent state of replication slots between the primary and backup databases, thereby enhancing the reliability of the replication process.

Implementation Method​

1. Creating Logical Replication Slots: When calling the pg_create_logical_replication_slot function, set failover=true to specify the synchronization of the replication slot to the backup database. This setting is crucial as it determines the behavior of the replication slot during primary and standby switches.

2. Subscription Configuration: When using the CREATE SUBSCRIPTION statement, specify failover=true to ensure the replication slot is synchronized to the backup database. This provides database administrators with flexible configuration options, allowing them to tailor replication strategies based on actual business needs and environmental characteristics.

3. Manual Synchronization: Execute the pg_sync_replication_slots function on the backup database to manually synchronize the replication slots. This method offers an immediate synchronization mechanism, particularly suitable for scenarios requiring rapid reflection of changes from the primary database.

4. Automatic Synchronization: Set the sync_replication_slots = on parameter on the backup database to achieve regular automatic synchronization without the need to restart the database. This automation simplifies database maintenance, allowing administrators to focus more on other critical tasks.

The following diagram illustrates the process of configuring and creating synchronized logical replication slots, detailing the steps to configure the basic parameters for synchronized logical replication slots and how to create one.

Functions and Parameters​

1. pg_create_logical_replication_slot

   • A new failover parameter has been added, with a default value of false. If set to true, it indicates that the replication slot needs to be synchronized to the backup database.
   • Example syntax:

     SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true)
     

   • This function provides database administrators with finer control, making the management of logical replication slots more flexible and efficient.

2. pg_sync_replication_slots

   • This function is used to manually synchronize replication slots to the backup database.
   • Example syntax:

     SELECT pg_sync_replication_slots()
     

   • Using this function allows for the instant synchronization of changes from the primary database to the backup database, ensuring data consistency and integrity.

3. sync_replication_slots

   • Setting this parameter to on on the backup database enables regular automatic synchronization of replication slot changes without restarting the database.
   • This automation significantly reduces the burden on database administrators, making the maintenance of replication slots easier and more efficient.

As shown in the figure above, the status of the replication slots for the primary and backup databases is displayed. The primary database has created the logical replication slot test_slots2 with the attribute failover set to true, enabling the failover functionality of this slot. Whether manually or automatically, the status and information of the logical replication slot test_slots2 can be synchronized to the backup database. When a switch occurs or a failover happens, and the backup is promoted to the new primary, the information of the logical replication slot test_slots2 will be retained, ensuring that the subscription side does not lose any data during the transition.

Operational Steps and Verification​

Manual Synchronization of Replication Slots​

Set up the primary and backup database environments, ensuring correct configurations for both physical replication slots and streaming replication users. This process requires careful planning and execution to ensure the stability and reliability of the replication environment.

1. Create the Replication Slot:

   • Execute:

     SELECT pg_create_logical_replication_slot('test_slot2', 'test_decoding', false, false, true)
     

   • This operation is the starting point for synchronizing the logical replication slot, laying the foundation for subsequent data replication.

2. Manually Synchronize the Replication Slot on the Backup Database:

   • Execute:

     SELECT pg_sync_replication_slots()
     

   • This operation ensures that changes to the replication slot are promptly reflected on the backup database, guaranteeing data synchronization and consistency.

3. Verify Replication Slot Synchronization:

   • Execute a query on the backup database to ensure the replication slot has been synchronized.
   • Execute:

     SELECT * FROM pg_replication_slots
     

   • This verification step is crucial for confirming the successful synchronization of the replication slots, ensuring the correctness and completeness of the replication process.

Automatic Synchronization of Replication Slots​

1. Set sync_replication_slots = on on the Backup Database:

   • Execute:

     ALTER SYSTEM SET sync_replication_slots = on
     

   • Then execute:

     SELECT pg_reload_conf()
     

   • This automation simplifies the synchronization tasks for replication slots, allowing administrators to complete synchronization without manual intervention.

2. Verify Automatic Synchronization:

   • Insert data into the primary database and observe changes in the backup database's replication slots to ensure that changes in the primary database are automatically synchronized to the backup database. This verification step is critical for confirming the proper functioning of the automatic synchronization feature, ensuring the automation and reliability of the replication process.

Simulating the Switch Between the Primary and Backup Databases​

1. Perform the Primary Database Promotion on the Backup Database:

   • Simulate a primary database failure by shutting down the primary database.
   • Execute on the backup database:

     pg_ctl -D data promote
     

   • This operation simulates an emergency switch in the event of a primary database failure, serving as an important step in testing the functionality of replication slot synchronization in real failure scenarios.

2. Continue Logical Replication on the New Primary Database:

   • Bind the VIP to the new primary database to ensure logical replication can continue. This operation ensures that logical replication can proceed seamlessly after the switch of the primary database, maintaining business continuity and data consistency.

The above diagram shows the new primary after the switch, retaining the existing logical replication slot my_subscription and publication information my_publication. After configuring other necessary parameters, the logical replication slot synchronization capability of the new backup is restored.

The subscriber routes to the new primary database using the VIP, continuing to utilize the synchronized logical replication slot to ensure normal data replication.

Conclusion​

The logical replication slot synchronization feature enhances the reliability of IvorySQL in high-availability environments, resolving the issue of logical replication slot loss after switching between the primary and backup databases. This feature ensures that information regarding logical replication slots is not lost during primary and standby switches, guaranteeing the continuity and consistency of the data replication process.

The synchronization process must meet the following conditions:

• Replication slots with failover=true set;
• Configuration of physical replication slots between the primary and backup databases;
• Enabling hot_standby_feedback on the backup database;
• Correct configuration of the database name in the primary_conninfo parameter on the backup database;
• Recommended configuration of the standby_slot_names parameter to ensure consistency;
• The backup database can manually or automatically synchronize replication slots through pg_sync_replication_slots or the sync_replication_slots parameter, ensuring seamless logical replication switching.

This series of measures and features collectively constitutes the power and flexibility of the logical replication slot synchronization functionality in IvorySQL, providing database administrators with robust tools to address various complex and challenging data replication scenarios.

With the diversification of database application scenarios, users have raised higher demands for data management flexibility and privacy. IvorySQL, an open-source database based on PostgreSQL and compatible with Oracle, has always been committed to staying ahead and innovating in terms of functionality. In the recently released version 4.0, IvorySQL introduced a new Oracle-compatible feature, Invisible Column, contributed by community member Imran Zaheer, demonstrating the power of open-source collaboration.

The introduction of Invisible Column provides developers with a new option to dynamically adjust the database structure without affecting existing applications. This further enhances IvorySQL's ability in data flexibility management, offering users greater convenience in areas such as database upgrades and compatibility optimization.

This article will provide a detailed introduction to the functionality, use cases, and usage of this feature.

What is Invisible Column?​

In modern database development, column visibility management plays a significant role in affecting the flexibility and migration efficiency of applications. Oracle 12c introduced a powerful feature: Invisible Column. This feature allows columns to be hidden, enhancing data security and implementing business logic. It provides developers with flexibility and control, especially in scenarios such as application migration or version upgrades.

In Oracle, an Invisible Column refers to columns that are invisible to most SQL queries and tools. By setting a column as an invisible column:

• It will not appear in the results of a regular SELECT * FROM query.
• It will not be displayed in description operations in SQL*Plus or OCI.
• It will not be included in record definitions based on the %ROWTYPE attribute.

However, invisible columns still exist in the table and can be accessed or referenced by explicitly specifying the column name. Additionally, there are restrictions when using invisible columns, as they cannot be used in external tables, cluster tables, or temporary tables.

Applications of Invisible Column​

1. Application Migration​

Invisible columns are very useful in application migration processes. When adding new columns to an existing table, invisible columns can prevent the new columns from affecting the functionality of old applications. The old applications will not be aware of the new columns, while the new applications can explicitly reference them. This makes the online migration of applications smoother and simpler.

2. Sensitive Data Protection​

Certain sensitive data can be stored in invisible columns to prevent it from being accessed by most default query tools, thereby reducing the risk of accidental exposure.

3. Data Model Optimization​

During data model adjustments or debugging, some columns can be temporarily set as invisible to ensure they do not impact regular queries, thus avoiding confusion in query results.

Using Invisible Columns in IvorySQL​

Invisible Column is a newly added compatibility feature in IvorySQL 4.0. Please make sure your version is 4.0 before using it.

1. Creating Invisible Columns​

You can define a column as an invisible column directly when creating the table:

CREATE TABLE employees (
emp_id NUMBER,
emp_name VARCHAR2(50),
emp_salary NUMBER INVISIBLE
);

In this example, emp_salary is an invisible column, which is not visible in the default query:

SELECT * FROM employees;
emp_id | emp_name
--------+----------
(0 rows)

2. Inserting Data into Invisible Columns​

When inserting data into the table, you can explicitly specify the column name to insert data into an invisible column:

INSERT INTO employees(emp_id, emp_name, emp_salary) VALUES(1, 'Jack', 20000);
INSERT 0 1
INSERT INTO employees(emp_id, emp_name, emp_salary) VALUES(2, 'Lucy', 30000);
INSERT 0 1;

Inserts without specifying column names cannot include invisible columns:

INSERT INTO employees VALUES(3, 'Peter');
INSERT 0 1

3. Displaying/Modifying Existing Columns as Invisible​

You can use the VISIBLE keyword to change an invisible column back to a regular column:

ALTER TABLE employees MODIFY emp_salary VISIBLE;
ALTER TABLE

To set an existing column as an invisible column, you can use INVISIBLE:

ALTER TABLE employees MODIFY emp_salary INVISIBLE;
ALTER TABLE

Note that not all columns can be set as invisible.

4. psql \d Meta-Command​

When using the \d meta-command in psql, the invisible column information will not be displayed:

\d employees
                  Table "public.employees"
   Column   |     Type     | Collation | Nullable | Default 
------------+--------------+-----------+----------+---------
 emp_id     | number       |           |          | 
 emp_name   | varchar2(50) |           |          | 
 emp_salary | number       |           |          | 

To view more detailed table information, including the invisible columns, you can use the \d+ meta-command:

\d+ employees
                                                   Table "public.employees"
   Column   |     Type     | Collation | Nullable | Default | Invisible | Storage  | Compression | Stats target | Description 
------------+--------------+-----------+----------+---------+-----------+----------+-------------+--------------+-------------
 emp_id     | number       |           |          |         |           | main     |             |              | 
 emp_name   | varchar2(50) |           |          |         |           | extended |             |              | 
 emp_salary | number       |           |          |         | invisible | main     |             |              | 
Access method: heap

5. Accessing Invisible Columns​

When using a SELECT * query to retrieve table data, invisible column data will not be displayed:

SELECT * FROM employees;
 emp_id | emp_name 
--------+----------
 1      | Jack
 2      | Lucy
 3      | Peter
(3 rows)

Although invisible columns are not visible in the default query, developers can still access them by explicitly specifying the column names:

SELECT emp_name, emp_salary FROM employees;
 emp_name | emp_salary 
----------+------------
 Jack     | 20000
 Lucy     | 30000
 Peter    | 
(3 rows)

Conclusion​

The invisible column feature is a cleverly designed functionality that provides greater flexibility and security for database development and management. By effectively utilizing invisible columns, developers can easily handle complex application migration scenarios while maintaining system stability and scalability.

If you have a project that uses IvorySQL database, consider integrating this feature into your solution to enhance overall efficiency and reliability.

To help users better understand and utilize IvorySQL 4.0, this article provides a concise overview of the design philosophy behind this feature.

What is an Oracle Package?​

A package is a schema object that contains logically related PL/SQL types, variables, constants, subprograms, cursors, and exceptions. Packages are compiled and stored in the database, and multiple applications can share the contents of a package.

Every package has a package specification that declares public objects accessible outside the package.

If the public objects include cursors or subprograms, the package must also have a package body. The package body must define the code for the public cursors and subprograms. The body can also declare and define private objects, which cannot be referenced outside the package but can be used within the package. Lastly, the package body can include an initialization section, used for initializing variables, performing one-time setup steps, and handling exceptions. Modifying the package body does not require changes to the package specification or to database objects that reference the public objects of the package, so the package body can be considered a black box.

Implementation of Packages in IvorySQL​

In terms of content, the package body is similar to a nested subprogram, and the package specification merely defines the external interface of the package body. Therefore, from an implementation perspective, the process of implementing a package can be similar to that of nested subprograms.

The main tasks we handle include the following: creating, updating, instantiating, deleting packages, and referencing package objects in external processes.

• Creating a package: Modify the psql syntax so that psql can send the entire package creation statement to the server. The server will then add the package creation syntax. The syntax structure is similar to that of ordinary functions, so it is processed similarly, without requiring expansion at the SQL side. After syntax parsing, it follows the DDL process and calls the package creation function to store the package contents in the system tables.
• Updating a package: Support update syntax on the SQL side. After syntax parsing, the package update function is called to update the system table contents. The plisql_package_Handler is invoked to parse the syntax and invalidate the package specification or body tuples, preventing package and body compilation at runtime.
• Deleting a package: Support the delete syntax on the SQL side. After syntax parsing, the package delete function is called to delete the package content from the system tables.
• Instantiating a package: The first time the package is referenced, if its contents are not in memory (specifically in a hash table, similar to the portal hash table storage), the instantiation function is called to reinstantiate the package. This process involves calling the PL/iSQL compile function to recompile both the package specification and body, storing the compiled result in the current process's memory. The instantiation of a package should load the entire package and body into memory.
• Referencing package objects: During parsing, an interface is provided to search for variables, types, and subprograms in the package specification. It first looks for the package type within the current schema, then searches the system tables. When looking for subprograms, it prioritizes searching nested functions, packages, and system tables.
• Package invalidation and state: If a package only contains constants and types, it is stateless. Otherwise, it is stateful. The state of the package is set when accessing its variables and functions. When the package is rebuilt, its local instance is invalidated and recompiled. For other processes' instances, if the package is stateful, accessing variables or types will result in a "state lost" error on the first access, after which normal access resumes.

Package Design in IvorySQL​

New System Tables and Caches​

To store the contents of the package body and specification, two new system tables have been added:

The corresponding system caches are as follows:

Package Instantiation​

Package instantiation, similar to function compilation, converts data defined as strings into structured data. A package consists of both a specification and a body, and its compilation requires some special handling. New functions are introduced to compile both the specification and the body and store the results in a hash table.

Additionally, to handle local cache invalidation during package and body deletion, an invalidation function is registered during cache creation, to clear the package's cache state when it is invalidated.

/* register invalid cache */
CacheRegisterSyscacheCallback(PKGBODYOID, InvalidatePackageCacheCallback, (Datum) 0);
CacheRegisterSyscacheCallback(PKGOID, InvalidatePackageCacheCallback, (Datum) 0);

InvalidatePackageCacheCallback traverses the hash table using the hash value to update the corresponding package's cache state.

The package cache state is represented by a char, with three bits currently in use. 0x01 indicates that the package specification has been updated, 0x02 indicates whether the package has a body, and 0x04 indicates that the package body has been updated.

Referencing Objects in a Package​

Interfaces are provided to search for functions, types, and variables within a package during the parsing phase. Below are some of the functions available:

When non-package functions in PL/iSQL use a package's type, they simply reference the type's address. When using variables, a local mapping is required, with special handling during assignments. Typically, this involves switching to the package's memory context and then calling the package's assignment function.

Function Parameters or Return Values Referencing Package Types​

This part requires modifications to the pg_proc system table structure to add fields that record the names of parameter and return value types. Two new columns are added to the system table to solve this issue.

Similar to proargnames, a new protypenames field records the type names of parameter types, and rettypename records the return value type name. These fields are only populated when the corresponding type references a package; otherwise, they remain empty.

Since protypenames is a text array, it is populated when a function's parameter is of a package type. The type name for package items is serialized into a TypeName structure, and other non-package parameter types are represented as empty strings.

Standard Package​

Support for the standard package under the sys schema is available, allowing users to access objects in the package specification without specifying the package name. Users can create their own standard packages.

DISCARD PACKAGE Syntax​

This feature was introduced for compatibility with PostgreSQL's DISCARD functionality. Currently, PostgreSQL supports DISCARD SEQUENCE, DISCARD TEMP, DISCARD PLAN, and other commands to discard session-level objects such as sequences, temporary tables, and plans. The DISCARD ALL command also deletes temporary storage like portals, temporary tables, plans, and sequences.

IvorySQL supports the DISCARD PACKAGE syntax, which, when called within DISCARD ALL, deletes the session's package cache.

Logical Backup and Restore Support for Packages​

The pg_dump tool now supports backing up and restoring data that includes package functionality.

Conclusion​

The design insights shared above outline the implementation of Oracle package compatibility.

By modularizing related functionality through packages, the database organizes procedures, functions, variables, and other programming elements into self-contained units, facilitating management and maintenance. Since implementation details are hidden in the package body, code security and maintainability are improved. Code in the package body is loaded into memory upon first invocation, allowing subsequent calls to access it directly, reducing parsing and loading time.

This article will detail the process of upgrading from IvorySQL 3.x to IvorySQL 4.0.

Introduction to the pg_upgrade Tool​

Recently, IvorySQL 4.0 was released, offering full support for PostgreSQL 17 and enhanced compatibility with Oracle.

This article will detail the process of upgrading from IvorySQL 3.x to IvorySQL 4.0.

Introduction to the pg_upgrade Tool​

The pg_upgrade tool is a built-in PostgreSQL utility for cross-version upgrades, allowing for in-place upgrades of the database without the need for export and import operations. Since IvorySQL is derived from PostgreSQL, it can also use the pg_upgrade tool for major version upgrades.

pg_upgrade provides a compatibility check before the upgrade (-c or --check option) to identify issues such as plugin and data type incompatibilities. If the --link option is specified, the new version can directly use the original database files without copying, allowing the upgrade to be completed in just a few minutes.

Common parameters include:

• -b bindir, --old-bindir=bindir: Path to the old IvorySQL executable directory
• -B bindir, --new-bindir=bindir: Path to the new IvorySQL executable directory
• -d configdir, --old-datadir=configdir: Path to the old data directory
• -D configdir, --new-datadir=configdir: Path to the new data directory
• -c, --check: Perform only a compatibility check without making any changes to data
• -k, --link: Upgrade using hard links

Next, we will explain how to use pg_upgrade to upgrade IvorySQL to the latest 4.0 version on the CentOS 8 platform.

Upgrade Preparation​

First, stop the old version of the IvorySQL 3.4 database:

Then, install the new version of the IvorySQL 4.0 database:

Initialize the new IvorySQL 4.0 data directory:

Check version compatibility:

Finally, if Clusters are compatible appears, it indicates that there are no compatibility issues between the two versions, and the upgrade can proceed.

Official Upgrade​

When you see Upgrade Complete, it indicates that the upgrade has been successfully completed.

Update Statistics​

pg_upgrade will create new system tables and reuse old data for the upgrade, but the statistics are not migrated during the upgrade process. Therefore, before enabling the new version, you should first re-gather statistics to avoid incorrect query plans due to missing statistics.

First, start the new version of the database:

Manually run the vacuum command:

Post-Upgrade Cleanup​

After confirming there are no issues, delete the old database:

With this, we have completed the entire upgrade process.

If you encounter any issues during the upgrade process, feel free to submit an issue on GitHub, and we will assist you promptly. Additionally, if you have any ideas or expectations for future IvorySQL features, you can also submit an issue to provide feedback.

Classification of Storage Files​

The storage manager used by IvorySQL is essentially the same as PostgreSQL. Its directory and file structure are shown in the following diagram:

From a storage perspective, file management is divided into three main parts:

• Configuration-related files, such as pg_control, postgresql.conf, pg_hba.conf, PG_VERSION, etc.;
• WAL log files, such as 000000010000000000000001, 000000010000000000000002, etc.;
• Data record files for tables and indexes, and corresponding remaining space and visibility mapping files.

Configuration-related files in PostgreSQL are handled with standard file read/write functions, for example, the pg_control file, which records the database state and operations. The background module reads and writes using ReadControlFile(), WriteControlFile(), and UpdateControlFile(). Front-end tools use get_controlfile() and update_controlfile() for reading and writing. Whether front-end or back-end, these read/write operations ultimately use standard file functions like open(), read(), and write(). These direct operations on configuration files are not within the scope of SMGR management.

WAL log files are read and written similarly to configuration files using standard file functions such as read(), write(), etc. However, file names need to be dynamically calculated based on the LSN.

Table and index data records are stored in the subdirectories of the corresponding database under the base directory. Changes to cluster-level catalog (system table) data records are reflected in the global directory. The management of system tables is similar to user tables, using the OID naming convention.

Process of SQL Statements Accessing Storage Media​

An SQL statement typically accesses data records on disk through the following steps:

• First, the statement goes through 5 stages of query processing: parse, rewrite, analyze, plan, and execute.
• Then, it enters the Table/Index Access Method layer.
• The Access Method layer typically uses the Buffer Manager service to operate on data entries, deciding whether to mark data blocks as "dirty" based on the operation.
• The Buffer Manager layer calls the Storage Manager service, using the buffer tag to invoke smgr_read() or smgr_write() to read or write data to the storage media.

SMGR API Interface​

PostgreSQL originally had many implementations of the SMGR API interfaces, but currently, only the implementation for Magnetic Disk (md.c) remains. In fact, the disk manager can support any type of device as long as the operating system provides a standard file system interface for the device. However, the storage manager (smgr.c) layer is kept as an intermediate layer in case other types of storage managers are introduced. Removing this intermediate layer would not save significant overhead, as operations on storage media are much more expensive than a single C function call. f_smgr is a structure of function pointers that contains the functions required by PostgreSQL for storage. smgr.c defines wrapper functions for all function interfaces, and these wrappers eventually call the actual registered implementation functions, i.e., mdXXXX functions.

static const f_smgr smgrsw[] = {
/* magnetic disk */
{
.smgr_init = mdinit,
.smgr_shutdown = NULL,
.smgr_open = mdopen,
.smgr_close = mdclose,
.smgr_create = mdcreate,
.smgr_exists = mdexists,
.smgr_unlink = mdunlink,
.smgr_extend = mdextend,
.smgr_prefetch = mdprefetch,
.smgr_read = mdread,
.smgr_write = mdwrite,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
}
};

SMgrRelation is an important structure, and almost all SMGR-related functions require this structure.

Key SMGR Interface Functions​

• Smgrread(): Locates an 8K data block based on 5 parameters: tablespace, database, relation, forknum, and blocknum, and reads it into the specified memory.

  smgrread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char * buffer)

• Smgrwrite(): Locates an 8K data block based on 5 parameters and then overwrites it with the values from the specified memory. The skipFsync parameter determines whether to ensure data is successfully flushed to disk before returning.

  smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char * buffer, bool skipFsync)

• smgrextend: Extends the current file with a new 8K block and writes the data from the specified memory, similar to smgrwrite.

  smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char * buffer, bool skipFsync)

• smgrnblocks: Returns the total number of 8K data blocks for the specified table, which is important for the planner stage of query processing.

  smgrnblocks(SMgrRelation reln, ForkNumber forknum)

• Shared object existence.

  smgropen(RelFileNode rnode, BackendId backend)

• smgrclose: Responsible for deleting the specified SMgrRelation object from the hash table.

  smgrclose(SMgrRelation reln)

• smgrtruncate: Deletes a specified number of 8K data blocks from the file (data, remaining space, visibility) from the end of the file, potentially removing multiple data blocks from the three fork files to reduce the file size.

  smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks, BlockNumber *nblocks)

• smgrprefetch: Uses POSIX_FADV_WILLNEED to request the operating system to pre-fetch disk data blocks into the cache, helping to avoid disk I/O bottlenecks.

  smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum)

smgrwrite and smgrextend are crucial functions for writing to disk and key to system I/O performance. They are called in the following scenarios:

• smgrwrite:

  • When a Buffer is modified, smgrwrite is called to write the data to the corresponding file on disk.
  • When a transaction commits, it is necessary to ensure that all dirty data is written back to disk, requiring smgrwrite.
  • During the VACUUM operation, when the visibility information of a table is updated, the corresponding buffer may become dirty, requiring smgrwrite to write it to disk.

• smgrextend:

  • When a file (table or index) needs to be extended, smgrextend is called to expand the file to the required size.
  • When creating a new table or index, its initial size is determined, and smgrextend is called.
  • When performing the CLUSTER operation, which reorganizes a table, smgrextend may also be called.

smgrwrite and smgrextend ensure data persistence, with smgrwrite being used to write modified data to disk, while smgrextend is used to expand the file size.