Proxmox Virtual Environment Cluster Manager-Part07
Results-oriented IT professional in various fields of IT, including Systems, Networking, File Systems, Security, Programming Languages, Virtualization, Storage, and Backups.
Proxmox Cluster is a feature of Proxmox Virtual Environment (Proxmox VE) that enables the creation of a unified and highly available virtualization infrastructure by clustering multiple Proxmox VE nodes (servers) together. The cluster management within Proxmox VE is facilitated by the Corosync cluster engine, which provides the underlying communication and coordination framework for the cluster nodes. Let’s delve deeper into each component and their architecture:
Proxmox Cluster:
A Proxmox Cluster is formed by connecting two or more Proxmox VE nodes over a network. These nodes collaborate to provide a unified platform for virtualization management. Key features of a Proxmox Cluster include:
Cluster Formation: To create a Proxmox Cluster, you typically start with two or more physical servers running Proxmox VE. These servers are configured to communicate and work together as a single entity. They are connected through a shared network and can access shared storage, allowing virtual machines to migrate seamlessly between nodes.
Centralized Management: With a Proxmox Cluster, administrators can manage all nodes and virtual machines from a centralized web-based interface called the Proxmox VE Manager. This interface provides a unified view of the entire cluster, allowing administrators to monitor system health, configure virtual machines, manage storage, and perform other administrative tasks seamlessly.
pmxcfs: Proxmox Cluster File System, is a distributed configuration file system within Proxmox Virtual Environment (Proxmox VE) clusters. It allows all nodes in the cluster to access and modify cluster-wide configuration files, ensuring consistency and synchronization across the entire cluster in real-time. pmxcfs enables centralized management, high availability, fault tolerance, and seamless communication between cluster nodes, enhancing the reliability and efficiency of Proxmox VE clusters.
Live Migration: Proxmox VE supports live migration of VMs and containers between cluster nodes without service interruption. This enables workload balancing, maintenance tasks, and hardware upgrades to be performed seamlessly, minimizing downtime and disruption to users.
High Availability (HA): One of the key functionalities of a Proxmox Cluster is high availability. Virtual machines running on the cluster can be configured with high availability settings, which means if a physical server (node) fails, the virtual machines will automatically be migrated to other healthy nodes within the cluster, minimizing downtime.
Note: We will cover a separate section about High Availability later.Disaster Recovery: Clustering in Proxmox VE facilitates disaster recovery strategies by providing features such as replication and backup. Administrators can replicate VMs and containers between cluster nodes or backup data to remote storage, ensuring data protection and recovery readiness in case of disasters.
Failover and Redundancy: The cluster management functionalities of Proxmox VE include built-in failover mechanisms and redundancy features to ensure the reliability and availability of virtualized workloads. In addition to high availability for virtual machines, Proxmox Clusters also support redundant networking and storage configurations to mitigate single points of failure.
Proxmox Virtual Environment Cluster Manager
Requirements:
All nodes must be able to connect to each other via UDP ports 5405–5412 for corosync to work.
Date and time must be synchronized.
An SSH tunnel on TCP port 22 between nodes is required.
If you are interested in High Availability, you need to have at least three nodes for reliable quorum. All nodes should have the same version.
We recommend a dedicated NIC for the cluster traffic, especially if you use shared storage.
The root password of a cluster node is required for adding nodes.
Online migration of virtual machines is only supported when nodes have CPUs from the same vendor. It might work otherwise, but this is never guaranteed.
Corosync Cluster Engine:
Corosync is an open-source cluster communication and coordination engine that forms the backbone of Proxmox Cluster. It provides the following functionalities:
Messaging Layer: Corosync establishes communication channels between cluster nodes, facilitating the exchange of heartbeats, status information, and messages related to cluster membership and configuration.
Membership and Consensus: Corosync maintains membership information about the cluster nodes and ensures that a consensus is reached among the nodes regarding the cluster’s state and configuration.
Group Management: Corosync organizes cluster nodes into logical groups and manages their membership dynamically, adapting to changes such as node failures or additions.
Quorum:
Quorum is a concept in distributed systems that refers to the minimum number of nodes required for the cluster to function effectively. In Proxmox Clusters, quorum is determined based on the number of active nodes participating in the cluster. Quorum is essential for ensuring the integrity and reliability of cluster operations.
Quorum Calculation: The Corosync cluster engine calculates the quorum based on the number of active nodes and their contributions to the cluster’s overall capacity. This ensures that critical decisions, such as resource allocation and failover actions, are made with consideration for the cluster’s overall health and availability.
Quorum Voting: During cluster operations, nodes participate in quorum voting to determine whether there are enough active nodes to constitute a quorum. If the number of active nodes falls below the quorum threshold, certain cluster operations may be suspended to prevent split-brain scenarios and maintain data consistency.
pmxcfs:
The Proxmox Cluster file system (“pmxcfs”) is a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync. We use this to store all Proxmox VE related configuration files.
Although the file system stores all data inside a persistent database on disk, a copy of the data resides in RAM. This imposes restrictions on the maximum size, which is currently 128 MiB. This is still enough to store the configuration of several thousand virtual machines.
This system provides the following advantages:
Seamless replication of all configuration to all nodes in real time.
Provides strong consistency checks to avoid duplicate VM IDs.
Read-only when a node loses quorum.
Automatic updates of the corosync cluster configuration to all nodes.
Includes a distributed locking mechanism.
Proxmox Cluster File System (pmxcfs)
Architecture:
The architecture of a Proxmox Cluster typically consists of the following components:
Proxmox VE Nodes: Physical servers running Proxmox VE, connected over a network to form the cluster.
Corosync Cluster Engine: Deployed on each node, Corosync manages communication and coordination among the cluster nodes.
Quorum Calculation and Voting: Corosync calculates quorum based on the number of active nodes and facilitates quorum voting during cluster operations.
Shared Storage: Often utilized in Proxmox Clusters to enable live migration and high availability of virtual machines.
+------------------------------------------------------+
| Proxmox Cluster Environment |
+-------------------------+---------------------------+
| Proxmox VE | Shared Storage |
| Nodes (1..n) | (NFS, iSCSI, |
| | GlusterFS, Ceph) |
+------------+------------+---------------------------+
| |
| |
+------------------+---------------------+-------------------+
| | | |
| | | |
+----v-----+ +------v------+ +------v------+ +------v------+
| Node 1 | | Node 2 | | Node 3 | | Node n |
| (Proxmox | | (Proxmox | | (Proxmox | | (Proxmox |
| VE Host) | | VE Host) | | VE Host) | | VE Host) |
+----------+ +-------------+ +-------------+ +-------------+
| Corosync | | Corosync | | Corosync | | Corosync |
| Cluster | | Cluster | | Cluster | | Cluster |
| Engine | | Engine | | Engine | | Engine |
+----+-----+ +------+------++ +------+------++ +------+------+
| | | |
+-------------------+-------------------+-------------------+
| |
+---------------+-------------------+---------------+
| | | |
+----v-----+ +-----v-----+ +-----v-----+ +-----v-----+
| Storage | | Storage | | Storage | | Storage |
| Device | | Device | | Device | | Device |
| (NFS, | | (iSCSI, | | (Gluster,| | (Ceph, |
| Gluster,| | Gluster, | | Ceph) | | etc.) |
| Ceph, | | Ceph, | | | | |
| etc.) | | etc.) | | | | |
+----------+ +----------+ +-----------+ +-----------+
Advantages:
Grouping nodes into a cluster has the following advantages:
Centralized, web-based management
Multi-master clusters: each node can do all management tasks
Use of pmxcfs, a database-driven file system, for storing configuration files, replicated in real-time on all nodes using corosync
Easy migration of virtual machines and containers between physical hosts
Fast deployment
Cluster-wide services like firewall and HA
References
