MPP Sizing Recommendation
The configuration and resources allocated to your CData Virtuality Embedded MPP cluster play a critical role in its performance. Choosing the right setup depends on the nature of your workloads, and performance can vary significantly based on query complexity and data volume. We strongly recommend conducting your own sizing tests using real-world data and queries to determine the best configuration. Here are some key considerations:
When setting up N workers, it’s best to allocate N+2 nodes to ensure that each worker runs on its own node while also providing additional capacity for the coordinator and metadata management.
Scaling up worker nodes generally leads to faster query execution, particularly for operations that can be processed in parallel, such as reading Parquet files.
Increasing CPU cores can also improve query speed, particularly in scenarios with multiple concurrent queries.
Ensure that the total CPU core count does not exceed the maximum permitted by your CData Virtuality license.
Memory availability is crucial—insufficient memory can lead to disk spills or performance degradation under high concurrency.
Suggested Cluster Configurations
Our recommendations are based on internal performance tests using analytical workloads with large datasets. While your exact needs may vary, the following guidelines can serve as a useful starting point:
Start with at least 8 worker nodes, each equipped with 128GB RAM and 16-32 CPU cores. In cloud environments like AWS, instance types such as m6a.8xlarge or r6a.4xlarge are good starting points.
If performance issues arise (e.g., query slowdowns or crashes), consider doubling your cluster size. For workloads involving hundreds of millions of rows, even single queries with no concurrency will generally run much faster on a larger cluster. When scaling up:
Without concurrency, query performance may improve modestly (5-15% gains).
With moderate concurrency (e.g., 5 concurrent queries), execution times may drop by 30-40% when doubling the number of workers, though benefits taper off as scaling increases.
With higher concurrency (10-15 sustained concurrent queries), a larger cluster (16+ workers) becomes necessary to maintain acceptable query speeds.
For heavy concurrent workloads, start with 32 workers (128GB RAM, 32 cores per node) and adjust based on demand.
If adding more worker nodes does not yield further improvements, consider using nodes with higher memory and CPU capacity. Some query operations, such as aggregations consolidating distributed results, may not parallelize efficiently and could require more powerful nodes.
In AWS, opting for memory-optimized instances like rx8large over general-purpose instances (mx8large) may be beneficial. However, memory-optimized nodes tend to be more expensive, and the actual performance gain depends on workload specifics.
Autoscaling for Cost Efficiency
Once optimal performance is achieved, configuring autoscaling ensures that resources are allocated efficiently:
Pod-level autoscaling can be configured as described in the Presto cluster on Kubernetes documentation.
Cluster node autoscaling should follow the guidelines of your cloud provider. For AWS users, refer to the EKS Autoscaling Guide for detailed instructions.
By carefully tuning your CData Virtuality Embedded MPP cluster, you can maximize query performance while keeping infrastructure costs under control.