How to Deploy CockroachDB on Red Hat OpenShift Virtualization

Rising VMware licensing costs and the uncertainty following the Broadcom acquisition are prompting many enterprises to explore alternatives for their virtualization platforms. Simultaneously, more organizations than ever want to adopt cloud-native architectures and operational models, even as they grapple with strict on-premises and hybrid requirements.

Red Hat OpenShift Virtualization stands out as a compelling alternative to VMware. It unifies traditional virtual machines (VMs) with container-based workloads under the same Kubernetes-driven platform—delivering cost efficiency, consistent operations, and open standards. Meanwhile, CockroachDB offers a distributed SQL database that is highly scalable, resilient, and truly cloud-native. Together, they provide a forward-looking solution to modernize infrastructure while still protecting mission-critical data and workflows.

This blog presents an in-depth technical blueprint for deploying CockroachDB on OpenShift Virtualization, referencing real-world validation tests, and context for our recently announced partnership with Red Hat. While most relevant to organizations in highly regulated industries and those seeking immediate VMware alternatives, the suggestions apply broadly to any enterprise requiring a cloud-native, future-proof virtualization strategy.

Why Consider OpenShift Virtualization as a VMware Alternative?

Since Broadcom’s acquisition of VMware, VMware has introduced new licensing models and cost structures that can be less than ideal, especially for environments still anchored on-premises. In addition, many enterprises running mission-critical workloads on VMware worry about renewed vendor lock-in, especially as they navigate modernization to containers and microservices. As an alternative to VMware, Red Hat’s OpenShift Virtualization offers a number of key advantages:

1. Unified Platform: Rather than managing virtualization separately from container orchestration, OpenShift Virtualization merges kernel-based virtual machines (KVMs) and Kubernetes containers into a single control plane, simplifying infrastructure and operational overhead.

2. Cost and Operational Efficiency: By consolidating onto a single platform, enterprises often reduce infrastructure sprawl, license expenses, and administrative complexity.

3. Hybrid Cloud-Readiness: Built on Red Hat OpenShift, clusters can span data centers, private clouds, and public clouds—facilitating a measured, step-wise migration away from purely on-premises environments.

4. Compliance and Control: For highly regulated industries (finance, healthcare, government), the ability to keep workloads on-premises under a standardized  platform—while selectively extending to the public cloud—delivers both agility and compliance.

5. Future-Proof Architecture: With virtualization and containers managed together, organizations can steadily refactor or containerize VMs over time without abrupt migrations, mitigating risk during modernization.

Why CockroachDB for VMware-to-OpenShift Migrations?

CockroachDB is a cloud-native, distributed SQL database well-suited for hybrid and multi-cloud deployments. Its architecture is inherently designed to:

• Scale Horizontally: New nodes can be added to the cluster as your needs grow—no complex re-sharding or forklift upgrades.

• Resilience & Fault Tolerance: Built-in replication using the Raft consensus protocol ensures automatic failover with minimal operator intervention.

• Strong Consistency: Retains ACID transactional guarantees across distributed nodes, vital for mission-critical systems.

• Familiar SQL Interface: Postgres-compatible queries minimize developer retraining and reduce adoption friction.

• Cloud-Native Capabilities: Automated rebalancing, rolling upgrades, and integration with orchestration tools like Kubernetes.

Benefits in an OpenShift Virtualization Context

• Shared-Nothing Architecture: Each CockroachDB node runs independently with its own CPU, memory, and storage, aligning seamlessly with the VM-based approach in OpenShift Virtualization.

• Operational Consistency: Whether on-premises or in the cloud, CockroachDB’s operational model stays consistent—fitting well into a hybrid environment managed by OpenShift.

• Reduced Complexity in Migrations: CockroachDB’s built-in backup and restore features ease data transfer from VMware-based environments to OpenShift Virtualization VMs.

Validating CockroachDB on OpenShift Virtualization

While testing CockroachDB, I validated performance, availability, and deployment simplicity on OpenShift Virtualization. My own evaluations covered:

• Seamless Installation: CockroachDB installs cleanly on Linux VMs provisioned by OpenShift Virtualization.

• High Availability: Node failure tests confirmed automatic failover and rebalancing without downtime for read/write workloads.

• Performance at Scale: Even under heavier TPC-C workloads (with 1,750 warehouses in testing), CockroachDB handled throughput without performance bottlenecks.

• Easy Observability: Standard metrics endpoints integrate smoothly with Prometheus/Grafana or other enterprise monitoring tools.

These tests gave me confidence that CockroachDB is ready for deployments in OpenShift Virtualization environments, even for mission-critical apps.

CockroachDB on OpenShift Virtualization: Overall Architecture

OpenShift Virtualization extends the Red Hat OpenShift platform, allowing you to run both containers and virtual machines under a unified Kubernetes-based management plane. In the illustration below, we highlight not only the CockroachDB VMs but also the worker nodes, the KVM hypervisor layer, and essential components of the OpenShift ecosystem that orchestrate and manage these workloads.

Key Components

Red Hat OpenShift Platform

• Provides the Kubernetes-based control plane for container orchestration, networking, and security.

• Includes OpenShift Virtualization, enabling you to run KVM-based virtual machines alongside containers.

Worker Nodes (RHEL CoreOS)

• Each worker node runs a KVM hypervisor and the virtualization components.

• Labeled for scheduling to ensure each CockroachDB VM resides on a different physical node.

Dedicated CockroachDB VMs

• Each VM (e.g., crdb-node-1, crdb-node-2, crdb-node-3) hosts a CockroachDB node.

• Deployed on RHEL/CentOS or another supported OS. Leverages local or persistent storage volumes to maintain the shared-nothing architecture.

Networking Services

• Headless Service (crdb-internal): Enables node discovery via stable DNS entries within the cluster.

• LoadBalancer Service (crdb-loadbalancer): Facilitates external SQL traffic (port 26257) and UI/metrics (port 8080) to all CockroachDB nodes.

Storage Layer

• Local SSDs or a supported Persistent Volume (PV) solution.

• Must provide POSIX compliance and sufficient throughput.

OpenShift API / Management Plane

• Coordinates all scheduling, networking, and lifecycle actions across the cluster.

• Exposes a single interface for both container-based workloads and VMs.

By encapsulating the virtualization layer within OpenShift, enterprises unify both containerized and traditional VM workloads under consistent tooling. CockroachDB’s distributed nature fits neatly into this framework, ensuring strong fault tolerance and near-linear scalability even as you transition away from traditional VMware-based environments.

Architectural Summary CockroachDB on OpenShift Virtualization

Shared-Nothing Architecture in Action

CockroachDB’s shared-nothing model is fully realized in this deployment model:

• Each VM is pinned to its own worker node (using labels and scheduling constraints).

• CPU, memory, and SSD/local storage remain dedicated to each VM, eliminating noisy-neighbor issues commonly found in resource-shared deployments.

• Network communications rely on standard Kubernetes services for node discovery and client ingress.

Core Components of CockroachDB

CockroachDB consists of a number of key components that allow us to provide the always-on availability, operational resilience, and performance at scale that our customers have come to know and depend on. A few are particularly important to call out when leveraging CockroachDB with OpenShift Virtualization

1. SQL Engine: Provides a powerful SQL interface compatible with PostgreSQL.

2. Transactional Key-Value Store: Uses the Raft protocol for replication and consensus-driven writes.

3. Data Distribution & Replication: Dynamically balances data ranges across nodes for optimal performance and fault tolerance.

Deployment Considerations

1. Network Configuration

• Ports

  • Port 26257 for SQL and inter-node traffic.

  • Port 8080 for the DB Console and metrics.

• Firewall: Ensure these ports are open on each VM to allow node-to-node communications.

• Service Types

  • Headless (no ClusterIP) for internal node discovery.

  • LoadBalancer for external access by apps or DBAs.

2. Storage Requirements

• Local SSD or Fast Persistent Storage: CockroachDB depends heavily on low-latency local storage for transaction logs and data files.

• POSIX Compliance: Storage must be POSIX-compliant to ensure ACID guarantees under crash recovery, strongly recommend using SSDs.

3. High Availability & Resilience

• Node Affinity: Label worker nodes to ensure each CockroachDB VM is scheduled separately, avoiding single-point-of-failure scenarios.

• Automatic Failover: If a VM or node fails, CockroachDB automatically re-replicates data to remaining nodes.

4. Load Balancing

• Round-Robin LB: The Kubernetes LoadBalancer typically distributes incoming SQL queries across all healthy nodes.

• Internal Connectivity: The headless service ensures the CockroachDB processes use stable DNS names for Raft replication.

Summary of the Technical Deployment Steps

Below is a quick recap of the recommended approach:

1. Node Preparation

   • Label three OpenShift worker nodes (e.g., node-role.kubernetes.io/crdb-node-1=true) so that each CockroachDB VM can be pinned to a distinct node.

   • Create a dedicated namespace cockroachdb for resources.

2. Networking Setup

   • Headless Service (crdb-internal): No clusterIP, used for internal DNS resolution among the CockroachDB nodes.

   • LoadBalancer Service (crdb-loadbalancer): Exposes external connections to port 26257 (SQL) and 8080 (UI).

3. VM Provisioning

   • Create three VMs via the OpenShift Virtualization console (e.g., crdb-node-1, crdb-node-2, crdb-node-3) using the CentOS 9 image.

   • Allocate resources per CockroachDB production best practices (refer to the production checklist).

   • Update firewall rules (if necessary) to open ports 26257 and 8080.

   • Adjust SELinux settings if needed to ensure the CockroachDB processes can run and bind to required ports.

4. CockroachDB Installation

   • Download CockroachDB binaries on each VM.

   • Place the cockroach binary in /usr/local/bin and ensure the PATH is updated.

5. Cluster Initialization

   • Start each node with --insecure or TLS flags as desired, specifying --join to the internal FQDNs for the other nodes.

   • On one node, run cockroach init to bootstrap the cluster.

   • Validate using cockroach sql --execute="SHOW DATABASES;" to confirm a healthy cluster.

6. External Connectivity

   • Retrieve the LoadBalancer external IP (oc get svc crdb-loadbalancer -n cockroachdb).

   • Connect from outside using cockroach sql --host=<LoadBalancer-IP>:26257.

7. Verification and Testing

   • Run basic DDL, DML, and transaction operations to confirm cluster stability.

   • Confirm high availability by stopping one node’s CockroachDB process; the cluster should remain writable.

Reliability and Performance Observations

1. Test Results at a Glance

• No Installation Barriers: The CockroachDB nodes came online without errors.

• Connectivity: Internal and external clients could connect via the configured services.

• Basic SQL: DDL, DML, and transactional statements worked as expected with ACID guarantees.

• High Availability: Node shutdown or partition tests showed zero downtime for read/write workloads; automatic rejoin upon node recovery.

• Performance: With TPC-C–style workloads at moderate scale (1,750 warehouses), the cluster maintained stable throughput and latency even as I scaled the cluster by adding nodes and also introducing node failures/network partitions.

2. Monitoring

CockroachDB provides a DB Console (port 8080) for real-time cluster stats and integrates seamlessly with Prometheus for metrics collection. During testing, no issues arose in scraping or visualizing performance data.

3. Rolling Upgrades

Upgrading CockroachDB nodes individually can be done without cluster downtime. This is critical for production environments that demand continuous availability.

Migration from VMware to OpenShift Virtualization

1. Key Considerations

• Risk Management: Minimal disruption is paramount for mission-critical apps. CockroachDB’s automated data replication and zero-downtime approach to adding/removing nodes reduce risk during migrations.

• Phased Approach: Migrate certain workloads or lines-of-business first, then gradually decommission VMware clusters.

• Backup & Restore: CockroachDB’s built-in backup tools simplify data transfers from existing databases.

• Kubernetes familiarity: Teams familiar with VMware need training in Kubernetes-based virtualization concepts. Early pilots help build internal knowledge.

2. Practical Steps

1. Set Up OpenShift Virtualization: Create an OpenShift environment with worker nodes sized appropriately for the new VM-based cluster.

2. Network & Storage Alignment: Ensure that your target environment can provide the same or better performance characteristics than VMware (especially regarding storage IOPS and network throughput).

3. Deploy CockroachDB: Follow the steps outlined above or the official deployment guide.

4. Data Migration: Use CockroachDB’s IMPORT, BACKUP/RESTORE, or third-party ETL tools to seed data from legacy systems.

5. Validation & Cutover: Conduct acceptance tests against the new CockroachDB cluster. Transition application connections to the newly deployed environment once validated.

Key Takeaways

1. Cost-Effective and Future-Proof: Moving away from VMware to OpenShift Virtualization provides significant cost and operational advantages.

2. Cloud-Native Database: CockroachDB complements this modernization by offering a highly resilient, horizontally scalable SQL platform that fits both on-premises and across multiple clouds.

3. Validated Architecture: My testing was able to give me confidence and confirm that CockroachDB runs reliably on OpenShift Virtualization with minimal operational friction and feels just as similar to deploying CockroachDB on cloud VMs.

4. Streamlined Migration: Enterprises can migrate mission-critical workloads at their own pace, leveraging proven backup/restore flows and phased cutovers.

As new complexities and uncertainties arise, organizations seeking flexibility, performance, and cost savings will find a robust alternative in Red Hat OpenShift Virtualization, powered by CockroachDB’s distributed SQL capabilities. This approach not only addresses immediate licensing and lock-in concerns, but also paves the way for hybrid and multi-cloud strategies that are increasingly essential in modern IT landscapes.