Table of Contents

Let’s make the composition a little more stable.

Enforcing Container Dependency
#

Previously, we managed to get the database and game server containers launched together with Docker Compose, and it seemed to work. But without any coordination, the game server could easily start before the database server was ready, causing it to fail. Luckily this is a very common issue, and we have ways to avoid it.

Healthy Database
#

First, let’s add a health check to the MSSQL container. Waiting for it to start listening on port 1433 should work:

Dockerfile

GitLab

27
28


HEALTHCHECK --start-period=1m --interval=30s --timeout=10s --retries=10 \
  CMD netstat -an | grep LISTEN | grep 1433

Of course we need netstat for that, so we’ll install that first:

Dockerfile

GitLab

3
4
5
6


# make netstat available
RUN apt update
RUN apt install -y net-tools
RUN apt upgrade -y

…and throw in an upgrade that gets rid of the high level vulnerabilities that were being detected in the container:

Using docker image sha256:e3c8f81f90d576ee0f1322ed05436ddd6b45743f3f5905b10fbc42512beb0670 for
 registry.gitlab.com/security-products/container-scanning:6 with digest
 registry.gitlab.com/security-products/container-scanning@sha256:83ce4e1e1872540ebc27159b36a6d4d3d37970d23ee3913e027e9964fb35b82c ...
$ gtcs scan
[INFO] [2023-06-05 16:56:34 +0000] [container-scanning]  >  Scanning container from registry
 registry.gitlab.com/301days/mssql-dragspinexp: for vulnerabilities with severity level HIGH or
 higher, with gcs 6.0.1 and Trivy Version: 0.36.1, advisories updated at 2023-06-05T12:09:02+00:00
[INFO] [2023-06-05 16:56:55 +0000] [container-scanning]  >  Scanning container from registry
 registry.gitlab.com/301days/mssql-dragspinexp: for vulnerabilities with severity level HIGH or
 higher, with gcs 6.0.1 and Trivy Version: 0.36.1, advisories updated at 2023-06-05T12:09:02+00:00
[INFO] [2023-06-05 16:57:12 +0000] [container-scanning]  >  Scanning container from registry
 registry.gitlab.com/301days/mssql-dragspinexp: for vulnerabilities with severity level HIGH or
 higher, with gcs 6.0.1 and Trivy Version: 0.36.1, advisories updated at 2023-06-05T12:09:02+00:00
Uploading artifacts for successful job

Patient Game Server
#

Now that the MSSQL container has a health check, we just tweak the docker-compose.yml to make the game server wait for it to be healthy with depends_on.

docker-compose.yml

GitLab

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


services:
  sql:
    image: registry.gitlab.com/301days/mssql-dragspinexp:latest
    environment:
      ACCEPT_EULA: Y
      SQLSERVR_SA_PASSWORD: ${SQLSERVR_SA_PASSWORD}
  gameserver:
    image: registry.gitlab.com/libipljoe/drag-spin-exp:latest
    ports:
      - 3000:3000
    depends_on:
      sql:
        condition: service_healthy
    environment:
      SQLSERVR_SA_PASSWORD: ${SQLSERVR_SA_PASSWORD}
      SQLSERVR_DB_NAME: ${SQLSERVR_DB_NAME}

And now running docker compose up shows the modified sequence:

$ docker compose up --quiet-pull --detach
sql Pulling 
gameserver Pulling 
gameserver Pulled 
sql Pulled 
Network drag-spin-exp_default  Creating
Network drag-spin-exp_default  Created
Container drag-spin-exp-sql-1  Creating
Container drag-spin-exp-sql-1  Created
Container drag-spin-exp-gameserver-1  Creating
Container drag-spin-exp-gameserver-1  Created
Container drag-spin-exp-sql-1  Starting
Container drag-spin-exp-sql-1  Started
Container drag-spin-exp-sql-1  Waiting
Container drag-spin-exp-sql-1  Healthy
Container drag-spin-exp-gameserver-1  Starting
Container drag-spin-exp-gameserver-1  Started

And a Very Patient Integration Test
#

All this work makes it more reasonable to add some integration testing into the CI configuration. Before kicking off any more complete testing, we should at least know we can log in and pet a dog in a minimal world.

First we need a way to make sure the game server is ready for business. I tried just waiting for it to be listening, like the database server, but that didn’t always work. So instead we can use a simple script to keep checking the logs for the regular status output, e.g.:

06/01/2023 23:58:12: NPCs: [1] | Players: [0] | CPU: [??%] | Rnd: [6]

wait_for_rounds.sh

GitLab

1
2
3
4
5
6
7


#!/bin/bash
while ! docker logs drag-spin-exp-gameserver-1 --tail=5 | grep "Rnd: ";
do
    docker logs drag-spin-exp-gameserver-1 --tail=5
    sleep 30
    echo "Waiting for rounds to show up in logs..."
done

Then we can use VHS running in its own container to script logging in and petting a dog, and check the output.

.gitlab-ci.yml

GitLab

61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76


  script:
    - docker compose up --quiet-pull --detach
    - docker ps -a
    - docker network inspect drag-spin-exp_default
    - sh wait_for_rounds.sh
    - docker logs drag-spin-exp-gameserver-1
    - docker run --rm --network drag-spin-exp_default -v $PWD:/vhs ghcr.io/charmbracelet/vhs dog.tape
    - docker logs drag-spin-exp-gameserver-1
    - docker compose down
    - grep "You pet the dog." dog.ascii
  retry: 2
  artifacts:
    when: always
    paths:
      - dog.ascii
      - dog.gif

Most of those docker calls are to put plenty of useful info into the output for troubleshooting. The fun one is using docker run to:

pull down the VHS image from the repo
bring up the container on the same network as the database and game servers
map the current directory into the VHS container
run VHS with the dog.tape file of instructions

dog.tape

GitLab

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


Output dog.gif
Output dog.ascii
Set Theme "MaterialDarker"
Set Width 810
Set Height 500
Set FontSize 12
Set Padding 5
Set Framerate 15
Type "apt-get install telnet"
Enter
Sleep 3s
Type "telnet drag-spin-exp-gameserver-1 3000"
Enter
Sleep 2s
Type "test123"
Enter
Type "test123"
Enter
Sleep 500ms
Type "1"
Sleep 500ms
Enter
Sleep 4s
Type "se se"
Enter
Sleep 3.5s
Type "pet dog"
Enter
Sleep 3.5s
Type "quit"
Enter
Sleep 1s
Type "/exit"
Enter
Sleep 5s

When run, it will create both a dog.ascii and a dog.gif (most of the Sets are for the gif) with the results of the inputs and timing. We have to install telnet first, since it’s not already in the VHS image. The gif is useful for troubleshooting, but in the CI flow we just check the ascii file for the expected result of the pet dog command.

$ grep "You pet the dog." dog.ascii
You pet the dog.
You pet the dog.
You pet the dog.
Uploading artifacts for successful job 00:03
Cleaning up project directory and file based variables 00:00
Job succeeded

The whole job takes about 5 minutes on GitLab’s free runners, using the minimal database. Of course I also tried doing the same using the production database, but found that there was much more work to be done. Tomorrow.

One More Thing
#

I also spent some time and effort trying to get rid of some of the vulnerabilities in the game server container image.

Dockerfile

GitLab

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


FROM mono:6

RUN apt-get update
RUN apt-get install -y apt-utils
RUN echo "deb http://deb.debian.org/debian buster-backports main" >> /etc/apt/sources.list
RUN echo "Package: *" >> /etc/apt/preferences.d/preferences
RUN echo "Pin: release n=buster-backports" >> /etc/apt/preferences.d/preferences
RUN echo "Pin-Priority: 1001" >> /etc/apt/preferences.d/preferences
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get upgrade -y e2fsprogs

ADD . /source
WORKDIR /source/DragonsSpine

RUN xbuild /p:Configuration=Debug DragonsSpine.sln

# Grant permissions for our scripts to be executable
RUN chmod +x entrypoint.sh

RUN apt-get remove -y curl perl
RUN apt-get autoremove -y

Adding the buster-backports package repo allowed some upgrades, and removing curl and perl obviously resolves the vulnerabilities in those; but the analysis of the container is still pretty grim.

$ gtcs scan
[INFO] [2023-06-07 01:19:18 +0000] [container-scanning]  >  Scanning container from registry registry.gitlab.com/libipljoe/drag-spin-exp: for vulnerabilities with severity level HIGH or higher, with gcs 6.0.1 and Trivy Version: 0.36.1, advisories updated at 2023-06-06T12:10:02+00:00
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
|   STATUS   | CVE SEVERITY | PACKAGE NAME |    PACKAGE VERSION     |                            CVE DESCRIPTION                             |
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
| Unapproved |    High      |  e2fsprogs   |    1.46.2-1~bpo10+2    | An out-of-bounds read/write vulnerability was found in e2fsprogs 1.46. |
|            |              |              |                        | 5. This issue leads to a segmentation fault and possibly arbitrary cod |
|            |              |              |                        |            e execution via a specially crafted filesystem.             |
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
| Unapproved |    High      |  gcc-8-base  |        8.3.0-6         | stack_protect_prologue in cfgexpand.c and stack_protect_epilogue in fu |
...
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
| Unapproved |    High      |  gcc-8-base  |        8.3.0-6         | The POWER9 backend in GNU Compiler Collection (GCC) before version 10  |
...
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
| Unapproved |    High      |   libc-bin   |    2.28-10+deb10u2     | An out-of-bounds write vulnerability was found in glibc before 2.31 wh |
...
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
| Unapproved |    High      |    libc6     |    2.28-10+deb10u2     | An out-of-bounds write vulnerability was found in glibc before 2.31 wh |
...
+------------+--------------+--------------+------------------------+------------------------------------------------------------------------+
| Unapproved |    High      | libcom-err2  |    1.46.2-1~bpo10+2    | An out-of-bounds read/write vulnerability was found in e2fsprogs 1.46. |
...

Before using this image in production, we’d want to peel another layer back and rebuild the mono container itself with a more recent and secure OS image. The benefit of an open source infrastructure is that we can do that pretty easily, at least in theory.

drag-spin-exp New Day 5 code

mssql-dragspinexp New Day 5 code

Enforcing Container Dependency #

Healthy Database #

Patient Game Server #

And a Very Patient Integration Test #

One More Thing #

Enforcing Container Dependency
#

Healthy Database
#

Patient Game Server
#

And a Very Patient Integration Test
#

One More Thing
#