I'll never step on your toes...
how to guard against merge racing in our CI pipeline
Forward: In the next several posts, I am going to share some CI / CD pipeline practices that I felt useful. I feel this topic is beneficial to engineers even though it’s a topic falling into DevOps engineer territory — because knowing why our pipeline always works (if we had a great pipeline on our side) and why our pipeline breaks (if we did not invest enough in it) can establish a great Dev and DevOps team communication starting point. If this series of topics interests you, please let me know by leaving comments below.
Simply setting up CI for each code commit may not guarantee a green master branch
CI tools or platforms have made it easy to run test suite on every pull request (PR in the github word) and merge request (MR in the gitlab word). But it may not be sufficient to guarantee we will have a green branch. For example, if two team members submit MRs, the test suite could pass on the CI process in each MR respectively; and they do not necessarily create any merge conflict to each other; but after both merges land to the master, the master may get into broken status (i.e, if we run test at the current HEAD, the test fails).
Gitlab merge train: prevent stepping on each other’s toes
Gitlab have a built-in feature called merge train. Here is an example from a Gitlab blog post
What merge train does is
queueing up the merge in its incoming order, say M1, M2, M3, M4
stack them together in the order and optimistically run the tests;
Run CI on “HEAD + MR1”
Run CI on “HEAD + MR1 + MR2”
Run CI on “HEAD + MR1 + MR2 + MR3”
Run CI on “HEAD + MR1 + MR2 + MR3 + MR4”
if a test with stacked MRs fails, drop the merging of that MR; at the same time, immediately cancel the CI run containing that MR; then replace the canceled CI jobs with newly created ones: newly stacked MRs without the already dropped MR. In the diagram above,
since CI on “HEAD + MR1 + MR2 + MR3” failed, MR3 is dropped off the attempted merging;
then the merge train system immediately cancels the CI jobs with MR3, i.e, the CI job of “HEAD + MR1 + MR2 + MR3 + MR4”;
then replace the canceled CI job with the newly stacked MRs — a CI job of “HEAD + MR1 + MR2 + MR4”;
So, with merge train system, our branch head will always be green, no bad MR will land on the target branch — we won’t step on other’s toes in a project.
Summary:
Merge time racing is a real issue, and it can cost team productivity.
If your team uses Gitlab as your repository system, turn on merge train — with that, your team members won’t step on each other’s toe.
PS: If you wanted to guard against cross-repo stepping toe’s problem, I’d like to suggest you watch a video presentation (20:00 — 26:00) by James Blair to get a sense how he architected the solution of solving the problem.