As a data scientist, you may occasionally train a machine learning model to be part of a production system. Once you have completed the offline validation of the model, the next challenge often lies in effectively deploying and managing the new model in the production environment. Machine learning model deployment, also known as model rollout, refers to the process of integrating a trained ML model into an existing production environment to make predictions with new data. It is a part of a broader methodology of deploying software versions and it's important to keep in mind that deploying an ML model is simply deploying a new software version.
In this blog post, we will explore five key strategies for rolling out (deploying) machine learning models, analyzing the merits and drawbacks of each. A comparative table is presented below for a brief overview:
Rollout Strategy | When to Use | Pros | Cons |
Shadow Evaluation | Assessing production load and model performance in high-volume prediction services | Efficient model evaluation without disrupting live trafiic. Monitored shadow model for performance and stability checks. | High costs due to resource-intensive shadow models. Increased architectural complexity without user response data. |
A/B Evaluation | Optimizing websites, refining product recommendations, and enhancing ad campaign effectiveness. | Simplicity and efficiency in model evaluation. Targeted deployment for optimized user experiences. | Limited applicability in complex models or scenarios. Resource intensive, demanding time and effort |
Multi-Armed Bandit | Dynamic allocation for e-commerce product recommendations and personalized content in media streaming. | Adaptive testing for continuous refinement. Efficient resource allocation and swift decision-making. | High implementation costs and complex setup. Limited applicability beyond rapid decision-making scenarios. |
Canary Deployment | Evaluating real-time user behaviour and obtaining fast feedback on new model performance. | Thorough assessment with real-world data. Consistent transition and easy rollback in case of issues. | Slower deployment compared to other methods. Constant monitoring required for effective user redirection. |
Rolling Updates | Continuous integration/deployment pipelines and consistent updates in e-commerce platforms. | Minimizes downtime and allows easy rollback. Incremental updates for controlled deployment. | Complexity in setup and management, especially in large-sacle systems. Longer deployment time due to gradual instance updates. |
The concept of model deployment
Several frameworks like Seldon, KServe, and SageMaker facilitate different ML model deployment strategies and implementations, yet the core concept remains consistent across platforms. Typically, the model is hosted as part of a service (which may be containerized) on a virtual machine. The service receives traffic through an API Gateway and may interact with a database. The essence of model deployment lies in updating the service to incorporate the new model, effectively in this diagram it would mean transitioning from Model V1 to its successor.
Model deployment strategy types
Deployment strategies can be divided into two types:
Static Strategies: These are the ones that decide how traffic and requests are managed. This includes approaches like: Shadow Evaluation, A/B Evaluation Canary, Rolling Updates and Blue-Green.
Dynamic Strategies: These are the ones that are a bit more hands-off. The system takes the wheel and manages the traffic or requests automatically. Multi-armed bandit is an example of this strategy. It’s like having a smart autopilot for your deployment.
Which deployment method is your default for LLM-AI-Agent applications?
A / B
Shadow
Canary
Bandit
You can vote for more than one answer.
Shadow Evaluation
Shadow evaluation acts as a safety net. Here's how it functions: When you place a new model into shadow mode, it becomes a candidate model. Whenever a new prediction request arrives, both the current production model and the candidate model in shadow mode process the request. This approach allows for the evaluation of the shadow model using real-world data without interrupting the services of the current model.
Pros
Model evaluation is efficient since both are running at the same time without causing impact on the traffic.
The shadow model can be monitored which allows to check on its performance and stability. This adds an extra layer of safety.
Cons
It can lead to high costs due to the resources needed to back up the shadow model.
Increases architectural and design complexity.
It does not provide any data on users responses, hence, not suitable for reinforcement or online learning models.
Use cases
Used for assessing production load, traffic, and model performance
Mainly applied in high-volume prediction services
A/B Evaluation
The A/B testing framework systematically distributes incoming traffic between two models, A and B (and can be expanded to include additional models if necessary), to evaluate their performance in a controlled setting. This technique is prevalent in e-commerce and social media networks. A/B testing is instrumental in selecting the more effective model by analyzing Key Performance Indicators (KPIs) such as response time, user engagement (e.g., click-through rates, conversions), and other quantifiable metrics.
Model variations may exhibit slight differences in features and target particular user segments, allowing for tailored deployment strategies. For instance, a new model may be introduced exclusively to certain regions, beta testers, or specific device types. This targeted approach mitigates potential negative impacts and can concentrate improvements on aspects of the business that would benefit the most.
Pros
A/B testing is a simple and efficient method for model evaluation.
It provides fast results, helping in the quick identification and elimination of low-performing models.
This system allows for targeted deployment, optimizing user experiences based on specific demographics or regions.
Cons
Increased Complexity: A/B testing may become less reliable when applied to complex models or scenarios.
Resource Intensive: Implementing A/B testing requires dedicated time, effort, and resources for proper execution.
Limited Applicability: A/B testing is most effective in cases involving simple hypothesis testing and may not be suitable for all types of model evaluation.
Uses Cases
Version Optimization: A/B testing can be applied to evaluate different results shown to users to determine which version leads to better KPIs.
Product Recommendations: E-commerce platforms can employ A/B testing to refine their recommendation systems, tailoring product suggestions to specific user demographics or geographic regions for a more personalized shopping experience.
Ad Campaigns: Digital advertisers can test different ad creatives, headlines, and targeting parameters to find the most compelling and effective combination for driving conversions and click-through rates.
Multi-armed bandit
The Multi-Armed Bandit (MAB) strategy is an evolved form of A/B testing. Its goal is to achieve balance between exploration and exploitation in order to maximize rewards. MAB leverages advanced machine learning algorithms to analyze incoming data and optimize Key Performance Indicators (KPIs). This method chooses the user traffic based on the KPI performance of multiple models, deploying the one that shows the best outcome on a global scale. Basically, MAB serves as a complex iteration of A/B testing. Using reinforcement learning principles, explores new model versions and using established ones to reach performance targets. This dynamic allocation of traffic between different versions is led by the performance KPIs or other relevant indicators.
Pros
Adaptive Testing: MAB facilitates dynamic exploration and utilization, allowing for continuous adjustment and refinement of strategies in real-time.
Efficient Resource Allocation: Unlike A/B testing, MAB ensures resources are used in a prudent way, minimizing waste and maximizing the effect of testing efforts.
Swift and Efficient Testing: MAB provides a faster and more streamlined testing process, enabling faster decision-making and implementation of optimized strategies.
Cons
Costly Implementation: MAB can demand significant computing power, leading to higher expenses, especially in cases where extensive resources are required.
Complex Setup and Management: Implementing MAB requires a deeper understanding of both machine learning and reinforcement learning principles, making it potentially more challenging to set up and manage compared to traditional A/B testing.
Limited Applicability: MAB is most beneficial in situations where the focus is primarily on optimizing conversion rates and decisions need to be made quickly. In scenarios where other performance metrics or longer decision-making timelines are a priority, alternative testing strategies may be more suitable.
Use Cases
E-commerce Product Recommendations: MAB can be used to dynamically allocate user traffic to different recommendation algorithms. By exploring and exploiting user interactions, the platform can quickly identify and deploy the most effective recommendation strategy, leading to higher conversion rates and improved user satisfaction.
Content Personalization for Media Streaming: MAB can be used to test different recommendation algorithms. By monitoring user interactions in real-time, the service can adapt and deploy the most effective recommendation strategy, resulting in longer user retention and increased subscription rates.
Canary
Canary Deployment involves updating the existing system, giving some users access to the new version while others stick with the old one. This is done to test the new version's performance, usually exposing a small fraction of users (5%–30%). It's performed in both staging and production environments to evaluate updated model performance. Unlike previous methods, this approach slowly introduces the new model to real users, enabling early bug detection before a global rollout.
Canary Deployment and A/B Testing differ in their approach and objectives. Canary Deployment ensures a stable user group interacts with the updated model, whereas A/B Testing randomly distributes users across different versions. The primary goal of Canary Deployment is to confirm the model's functionality, while A/B Testing focuses on assessing user experience. Also, Canary Deployment involves a limited fraction, never surpassing 50%, of user requests. This constitutes a small portion of the overall user base.
Pros
Enables thorough assessment of the new model with real-world data.
Ensures seamless transition with zero service disruption during the update.
Allows for quick and easy rollback to the previous version in case of any issues.
Cons
Deployment rollouts may be slower compared to other methods.
Requires constant monitoring, especially when testing with a limited user base, to redirect users effectively in case of failure.
May not be suitable for scenarios where gathering enough data for statistical significance is a lengthy process, as it can take time to achieve meaningful results.
Use Cases
Evaluating Real-time User Behavior: Canary deployment is highly effective when you want to assess how a new model performs in real-time with actual user interactions. This is crucial for applications where user behavior is dynamic and may change rapidly.
Time-sensitive Updates: When there's a need for fast feedback on the performance of a new model, Canary deployment is the preferred choice. It provides results in a relatively short timeline when compared to methods like A/B testing, which might require an extended period to collect enough data for meaningful insights.
Rolling Updates
A rolling deployment is a method for gradually updating a model without any downtime. It involves replacing older versions with newer ones one by one in an active instance, without the need for staging or private development. This approach is useful for quickly updating an entire model lineup and allows for easy rollback to previous versions. It is commonly used in testing or staging environments to evaluate new model versions. However, it is typically just one component of a production system.
Pros
Minimizes Downtime: Rolling deployments ensure that there is no operational downtime during the update process, allowing the model to continue running.
Easy Rollback: It’s able to revert back to a previous version if any issues arise with the new update, providing a safety net for model deployments.
Incremental Update: This strategy allows for a gradual and controlled update process, reducing the risk of widespread disruptions and making it easier to monitor and troubleshoot any potential issues.
Cons
Complexity: Rolling deployments can be more complex to set up and manage compared to other deployment strategies, particularly in large-scale or complex systems.
Longer Deployment Time: Since rolling deployments update instances one by one, it may take longer to complete the update process compared to other deployment methods.
Use Cases
Continuous Integration/Continuous Deployment (CI/CD) Pipelines: Rolling deployments are commonly used in CI/CD pipelines to update applications or models in production environments. This ensures that new features or improvements are gradually rolled out without causing downtime or disruptions for end-users.
E-commerce Platforms: In e-commerce, rolling deployments are useful for updating the platform without interrupting the shopping experience. For example, a rolling deployment can be employed to introduce new features or fix bugs in an online store, allowing customers to continue browsing and making purchases while the update takes place in the background.
Summary and practical approach
In this blog post, I review five theoretical approaches for deploying ML models to production. In practice, these will be implemented as part of an ML platform such as Seldon, Qwak, or Vertex AI. As an AI consultant, I recommend examining how models can be rolled out when selecting your ML framework/platform. These considerations can have a significant impact on the stability of the production system as well as on the complexity of the solutions that you will be able to implement.