The first thing that you need to realize is that when you are walking through the AB testing procedure there are essentially seven steps that you need to consider the first thing to do is basically understanding the problem the problem statement this is where you try to make sense of the case problem that you need to solve by asking clarifying questions to the interviewer and also figuring out what is this success metric and what is a user journey

The second thing is that you want to Define your hypothesis testing and what this basically means is that you set up what your null hypothesis and alternative hypothesis is and and you want to set up some parameter values for your experiment such as the significance level and statistical power

The third step is designing the experiment itself and so this is where you talk about what is the randomization unit and which user type you're going to actually Target for this experiment and various other things that you definitely need to consider when you're designing the experiment

The next step is to run the experiment itself and this is where you need to think about the instrumentation that is required to actually collect the data and analyze the result

Once you've collected data the next thing that you need to do even before you actually interpret the result and decide to launch is basically do some sanity check or validity checks because if your experiment design was flawed or if there's some bias that was implemented into the data collection itself then you have flawed result

Once you've done the sending check the next step is to basically interpret the result in terms of what is the lip that you saw the P Val in compress interal and lastly now that you have the statistical result along with the business context this is where you make a decision in terms of whether you're going to launch the change or not

Suppose that an online clothing store called fashion web store wants to test a new ranking algorithm to provide products more relevant to customers how would you design an experiment so in order to tackle this problem that requires AB testing we want to first of all understand the nature of this product

the next Factor you want to consider is the cost of launching if you see that the cost of building this out basically rolling out to all of the users and the cost of maintaining this change is highly costly then maybe this isn't something that you want to actually launch

now you want to kind of go back to the interpretation of the result that you got from the experiment and along with the business context and the statistical result ultimately you want to decide whether you're going to launch it or not

so in this first case what we see is that the lift is placed at a positive value but it's still less than practical significance and you also see that the lower bound and the upper bound of the confidence interval are less than the Practical significance

So based on all of these considerations in terms of the business context along with various statistical outcomes ultimately what the decision you want to make is that you want to launch this new algorithm as a way to provide a more relevant product recommendations to the users thereby improving Revenue overall

when you think about whether you want to launch it or not and in this step there are three factors that you want to consider the first factor is the metric tradeoff so you might have a case where the success metric might have improved but the guard R make metrics or the secondary metrics might have declined and so you have to think about what are the pros and cons of launching it considering that the guard one metrics might have declined

the last Factor you want to consider is a risk of committing false positive or your typon error rate for instance if you falsely conclude that there is an effect when there isn't and you've made a change then it might have a negative consequence to the user