Random Thoughts

A long time ago, when I first started working with backend code, I didn't care much about file versioning. Back then, I would crudely modify files directly on the server.

As I gradually wrote more code and the functionality became more complex, I suddenly realized that the traditional, crude approach led to long periods of service downtime. When you publish or modify a file, unexpected issues can arise, causing the system to fail unexpectedly. At that point, you might find that the bug or problem is something you have never encountered before. Then begins the intense troubleshooting and fixes, taking a few minutes if you're lucky or possibly stretching to days. In extreme cases, you might have to revert to a previous version.

After a period of agony, I finally began seeking ways to optimize this release process. Thus:

Preparation

Before tackling a new release process, I undertook some tasks related to the publication:

• Enable CDN services
• Version control for front-end resources (I used my own developed tool mStaticizem, staticize.zhuwenlong.com, which adds version numbers to all static resource files and replaces all the referenced files, maximizing the value of the CDN)

Release Process & Testing Process

In general, there will be development servers, testing servers, and some companies may even have pre-release environments. The purpose is singular: to reliably publish code. However, the existence of these servers significantly affects the release timeline. Often, file transfers are done manually. I've heard about a news story:

The CTO of a well-known internet company abroad resigned due to the lack of effective means to control release failures and reduce overtime on release days. His successor proposed a "website train release model" for automating the release process: the less human intervention, the higher the automation level, thus reducing the chance of introducing faults, and increasing the likelihood of everyone off work on time.

As seen, the release process is quite a serious and important matter. If handled well, it can yield numerous benefits.

Actual Process

After discussions with many frontline colleagues, I adopted the following release process:

1. Development

This follows the approach most people use: there is a dedicated server for development, where all new feature branch developments and self-tests are conducted before proceeding to release.

2. Handling Front-End Resources

Due to the specific nature of front-end resources, processing is required before release, which includes several automated steps:

• Automated merging and compression of CSS and JS files
• Automated lossless compression of image files, with the previous two steps achieved using the widely-used automation tool Grunt
• Version handling of front-end resources. You might often see CSS and JS addresses on BAT company pages like xxx.b3adg.css and xxx.dgt6d.js. Their static resources all have version numbers added, including all references. Achieving this manually is nearly impossible, so I used my self-developed front-end integration tool mStaticize, which not only replaces file names but even replaces all references in CSS and JS files.

3. Packaging for Pre-Release

After completing the previous two steps, I create a timestamped compressed package containing all the involved files (e.g., zhuwenlong.com_20140301) and automatically upload it to the production server.

4. Testing

After the production server receives the files, it will automatically extract the relevant compressed package to the specified location and run the service on a port incrementally. For example, if the previous version was running on 127.0.0.1:8999, the new package’s service will be 127.0.0.1:9000. Note that these ports are not exposed externally; they are only accessible on the intranet. You might notice that multiple versions of the service can run simultaneously, which will be explained in the next step. At this point, our new package has been uploaded to the server and is running on a new port. Next is testing, which doesn’t require a lot of manpower; a simple automated run of the new feature’s unit tests followed by some manual checks if you don’t feel confident should suffice.

5. One-Second Official Release and Rollback

The previous step mentioned multiple simultaneously running services, which comes into play here. Once testing is complete, you simply need to modify the nginx domain port mapping, for instance, changing from 127.0.0.1:8999 to 127.0.0.1:9000, effectively releasing a new version, which truly takes only 1 second (some might wonder how to handle the browser cache for front-end files during this switching; our earlier Step 2 used mStaticize to already replace static resources with the new version numbers, allowing seamless switching between versions). What about the old services online? Here, you can choose how long to keep them, such as one month or an indefinite period. The purpose of keeping them is to allow you to rollback to an older service in just one second when you discover a serious bug in the new version.

This entire release process concludes easily.