For just over a couple of years, I’ve been using GitLab Runners to deploy and perform tests on some projects I develop. It all started as a test with simple projects, like Music DL and socializer. The idea of runners is to use them to complete the CI/CD Cycle (that is, continuous integration and delivery). In general terms, this makes it possible that whenever a specific condition is met within the source code repository (for example, creating a tag for each new version, sending a commit to a repository), a Runner takes charge of executing a series of commands or scripts to run the project’s test units, or can even compile, package and distribute versions ready for users directly. This is what we’ve been using in TWBlue for Snapshot updates for a couple of years, which in the end was an important factor when deciding to shorten the development cycle of the application versions. Personally, I also like that a runner executes all the tasks, as that helps us keep our personal or work computers out of TWBlue distribution. Since these are Python projects, sometimes the modules that help us generate distributable versions (such as cx_freeze) incorrectly include packages that aren’t used in the application. This can’t always be controlled appropriately and sometimes, a TWBlue version generated in the past included more or fewer extra and unnecessary modules depending on whose computer the distributable versions had been built on. With a runner creating each version of our software, we ensure we have a machine that only has the necessary tools installed for what will be built, and we also provide everyone who wants to review it, a list of reproducible steps that need to be executed to create an identical copy of our versions. Finally, the fact that all this happens completely automatically makes us have to worry less about preparing everything and executing each step manually on our machines: A developer only has to send a commit or create the appropriate tag, and the runner will test, build and upload the executables to our ftp site in a matter of minutes.

When I started with the idea of MCV Software, I decided to try to go a step further and start creating some infrastructure for the needs of both public and private projects. One of the things I had in mind is the transition of Socializer from CX Freeze to Nuitka, which would considerably reduce the application’s response time and a bit the weight of the distributable files, in addition to avoiding including unnecessary dlls that CX Freeze sometimes tends to include. Unfortunately, I wasn’t able to reduce the compilation times of the complete project within the shared Windows runners available on Gitlab.com (the maximum limit per step is one hour and there’s no way to change it), so I decided to deploy my own Windows-based runner to build some of the projects that tend to have new versions regularly.

Strategy

There are many ways to create runners to execute jobs in GitLab. My perfect idea would be to deploy runners automatically. These runners would be created during the time they were needed. They would include a minimal Windows installation and Chocolatey installed, and the developer, through the .gitlab-ci.yml file, would be in charge of installing everything the project requires, building whatever needs to be built, uploading the artifacts and finally once the project finishes the continuous integration and delivery cycle, destroying the virtual machine instances created to thus return to an initial state.

Unfortunately, for now it’s not possible for me to perform this configuration without spending a considerable amount of money. Maintaining an auto-scalable runner tends to be a somewhat costly process, and it’s carried out within Azure, Google Cloud Platform or Amazon Web Services clouds. All these services are complex and at this moment I’ve opted for a much simpler solution. I acquired a VPS with Windows Server 2019 at a quite reasonable price to which I’ve installed the essential packages to be able to work:

• Git for Windows.
• PowerShell.
• Python 3.7.9 (I installed both versions, for 32 and 64 bits, as there are some projects that are published with support for both. The paths are c:\python37-32 and C:\python37).
• GitLab Runner for Windows.
• Chocolatey, and thanks to this package manager, it was also simple to install the following applications:
  • nsis

In addition, logically, it already includes the latest operating system updates, to keep everything always at its latest versions.

This deployment strategy has one advantage and at the same time one disadvantage. The advantage is that the Runner is always ready and created, waiting to execute orders and deliver or test the project requested of it. There’s no need to wait for any resource to be created. However, the possible disadvantage is that the system doesn’t start completely clean. Since it’s the same operating system used for all projects, reusing the runner could cause some problem. Although, after evaluating things thoroughly, and performing some internal tests, I’ve considered that using a virtual environment in Python packages is sufficient isolation to not mix packages between different projects. In each project built on the MCV Software infrastructure, a Python virtual environment is created at the beginning of the distributable version construction tasks, and the same environment is used to install and generate everything required by the project. Finally, upon finishing the project, the virtual environment is destroyed, deleting all packages and leaving the system Python as clean as when it was first installed.

Another thing I’d personally like to explore are Runners in Docker. Not for Windows, as support with this type of system in Docker in GitLab is not very developed, but I think it would be a good idea to deploy a second runner that runs on a Linux system and can use Docker containers. This would make much of the testing work easier, uploading artifacts to ftp servers and such, where only the Python environment is necessary and Windows can be left somewhat aside. But all this will be perhaps for another time.