How OptScale allowed the company with an $80M cloud bill to run ML experiments with optimal performance and reduce infrastructure costs by 37%

Running hundreds of ML experiments daily, ML teams faced the following challenges:

• Lack of automated and efficient instruments for ML/AI model training tracking and profiling/instrumentation

ML/AI model training is a complex process that depends on a defined hyperparameter set, hardware, or cloud resource usage. Monitoring and comparing key metrics and indicators against established benchmarks or thresholds enable gaining profound
insights and enhancing the ML/AI profiling process.

• Limited transparency throughout the ML lifecycle

Without sufficient transparency into the ML process, it became challenging for the company to determine bottlenecks in ML model training and select the optimal configuration of cloud resources. The lack of visibility hinders the ability to maximize
ML/AI training resource utilization and outcome of experiments and accurately plan and forecast resource requirements, leading to overprovisioning or underprovisioning cloud resources.

• Identifying optimization scenarios for improving performance and cloud bill reduction

ML models often require complex and significant cloud infrastructure for training and inference. Inefficient ML model and experiment management mechanisms led to increased resource costs and longer processing times due to bottlenecks in specific resources like GPU, IO, CPU, or RAM. Without proper monitoring, the company faced challenges in identifying bottlenecks, performance issues, or areas for improvement.