How to spin up GitLab on docker + reusable host ports

13 min readDec 24, 2020

Background

So it began 3 days ago, i was trying to enable container registry on my old GitLab installation with no luck. Something didn’t work properly even though I kept my GitLab up-to-date, it was probably something in configurations.. anyway, back then docker wasn’t a thing for me but now that I got my self a new server I didn’t want to make the same mistake I did 3 years ago, so I just installed docker engine on my new server and tried to do everything with containers, so I searched “GitLab” in Dockerhub and found out that GitLab has an official docker image, everything worked smoothly with few lines of command (as instructed here):

# put this in your ~/.bash_profile
export GITLAB_HOME=/srv/gitlabsudo docker run --detach \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 --publish 22:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

Note: if you are following along, put GITLAB_HOME definition in your ~/.bash_profile for it to survive system reboots.

All that happiness ended when I got this error message:

Bind for 0.0.0.0:22 failed: port is already allocated.

The Problem

Well, you might not be surprised to see port 22 being used on your server already, after all if sshd wasn’t installed on the server we couldn't even run those docker commands anyway. As it happens 22 is the default port for SSH communications, if you are not familiar with sshd:

sshd is the OpenSSH server process. It listens to incoming connections using the SSH protocol and acts as the server for the protocol. It handles user authentication, encryption, terminal connections, file transfers, and tunneling. — source

you might even get similar error messages for the port 80 or even 443, those are ports for HTTP and HTTPS that you might use for serving a website or for your nginx setup.

Lets bind them into custom ports of docker host:

sudo docker run --detach \
  --hostname gitlab.example.com \
  --publish 4443:443 --publish 8880:80 --publish 2222:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

Now visit http://your_ip_address:8880 and you will see:

Voila! it’s working! Are you happy now? probably not 😕, if you are satisfied by now, good for you! but for me.. Although it’s working, that’s not what I want, repository clone urls are not properly set and I don’t like to see those custom ports in them, and I don’t like to access my instance with non-standard HTTP or HTTPS port.

Here's docker-compose.yml file contents with solutions to fix clone urls if you are OK with custom ports:

version: "2.0"services:
    gitlab:
      image: 'gitlab/gitlab-ce:latest'
      restart: always
      hostname: 'gitlab.example.com'
      container_name: gitlab
      environment:
        GITLAB_OMNIBUS_CONFIG: |
          external_url 'http://gitlab.example.com:8880'
          registry_external_url 'http://registry.example.com'
          letsencrypt['enable'] = true
          letsencrypt['contact_emails'] = ['support@example.com']
          gitlab_rails['gitlab_shell_ssh_port'] = 2222
          # Add any other gitlab.rb configuration here, each on its own line
      ports:
        - '8880:8880'
        - '4443:443'
        - '2222:22'
        - '5050:5050'
      volumes:
        - '$GITLAB_HOME/config:/etc/gitlab'
        - '$GITLAB_HOME/logs:/var/log/gitlab'
        - '$GITLAB_HOME/data:/var/opt/gitlab'

What we did here is setting a few environment variables to make sure our custom ports are in sync with GitLab container, one thing to note is external_url variable which is sensitive to protocol, if you are providing http:// url you should also bind it to port 8880 (or any other port you binded from host into the container for this purpose) and it will be shown in clone section of repository; it’s the same story with gitlab_shell_ssh_port=2222 part, but with one important difference, since GitLab uses system sshd service (not a separated one) gitlab_shell_ssh_port just changes what you see at SSH cloning url, so that’s the reason we are binding port 2222 of the host to port 22 of the container, setting gitlab_shell_ssh_port won’t change container sshd port. and the last port 5050 will be used by container registry, which hopefully is not taken by your docker host.

Now that was all for using custom ports with GitLab, if it’s not enough for you, read on!

Goals

What I want to achieve is to use port 80 and 443 for as many containers as I need (for my web applications) and somehow share port 22 of the docker host with my GitLab container.

Solution

Here I will do my best to provide you a solution to each of those goals separately so you can take only what you need. It’s worth mentioning that I will use docker compose to manage my containers through this article.

Setting up Nginx Reverse Proxy (Reusing port 80 and 443)

The first thing I want to setup is nginx reverse proxy, if you don’t know what that is:

An Nginx HTTPS reverse proxy is an intermediary proxy service which takes a client request, passes it on to one or more servers, and subsequently delivers the server’s response back to the client — source

Not comfortable with that definition yet? you don’t know why we want to use it? well, to put it simply: we will use it as an HTTP/HTTPS server which will bind on port 80/443 of the docker host and forward requests to the container which serves that specific domain; this way you can forward request of example1.com to the container that is hosting that domain, and also example2.com to another container which contains it, fortunately there is a good docker image for that, setting it up is easy and straight-forward, however there are few things that you need to take care of, namely HTTPS certificate generation. But we will do it one step at a time and I will try to describe what we’re doing.

version: '3'services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    restart: always
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - dhparam:/etc/nginx/dhparam
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:roletsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    restart: always
    container_name: nginx-proxy-le
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.example.com'
    container_name: gitlab
    environment:
      VIRTUAL_HOST: gitlab.example.com,registry.example.com
      LETSENCRYPT_HOST: gitlab.example.com,registry.example.com
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.com'
        registry_external_url 'https://registry.example.com'
        gitlab_rails['gitlab_shell_ssh_port'] = 2222
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        registry_nginx['listen_port'] = 80
        registry_nginx['listen_https'] = false
        nginx['proxy_set_headers'] = {  "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on" }
        # Add any other gitlab.rb configuration here, each on its own lineports:
      - '5050:5050'
      - '2222:22'    volumes:
      - '$GITLAB_HOME/config:/etc/gitlab'
      - '$GITLAB_HOME/logs:/var/log/gitlab'
      - '$GITLAB_HOME/data:/var/opt/gitlab'volumes:
  conf:
  vhost:
  html:
  dhparam:
  certs:
  acme:

There are two new services in our latest docker-compose: jwilder/nginx-proxy which we already talked about and jrcs/letsencrypt-nginx-proxy-companion, the latter is responsible for generating free certificates automatically using lets encrypt, actually you generally won’t need to change anything about these two, however if you are interested to know how they work and why are the volume bindings, I invite you to read their documentation.

Here are the interesting parts to understand in GitLab service:

Added VIRTUAL_HOST environment variable, this is how nginx-proxy will know which container should it send traffics to, we have two domains which needs to be redirected into the GitLab container, one is for GitLab itself and another for package and container registry.
Added LETSENCRYPT_HOST environment variable, this one will be used by letsencrypt-nginx-proxy-companion to generate and provide SSL certificates, again we have provided the same two domain names for it to operate on.
I have moved from HTTP to HTTPS protocol in external_url and registry_external_url and removed ports because we will be using the standard ones.
I have removed all of the port bindings except 5050 and 2222 which is because nginx-proxy will take care of forwarding requests for ports 80 and 443, keep in mind that letsencrypt-nginx-proxy-companion will take care of SSL certificates.
Added nginx['listen_port'], nginx['listen_https'] (same settings for registry_nginx too) and nginx['proxy_set_headers'] configuration.

Alright, most of the changes above are self explanatory, but about the last one I would like to elaborate a bit.

If you read documentation of those two docker images which are responsible for reverse proxying HTTP and HTTPS requests, there are some things that you need to pay attention to. firstly nginx will redirect HTTP traffic to HTTPS protocol if certificates are provided, secondly for some weird reason authors of nginx-proxy have decided to send those traffics into port 80 of the container regardless of the protocol, this behavior can be changed by providing VIRTUAL_PROTO and VIRTUAL_PORT environment variables, but it’s not what we want anyway.

GitLab has lots of tweaks and tricks in the configuration file, for instance if you provide HTTPS protocol in external_url, it will try to verify SSL certificate which is already taken care of by letsencrypt-nginx-proxy-companion , we need GitLab to show HTTPS version of the url in cloning section of repositories but listen on port 80 instead of 443 of the container, hence those configurations for both nginx (which will used for GitLab itself) and registry_nginx.

You can now run your docker-compose.yml and access your GitLab without any custom ports:

docker-compose up -d

Note: depending on your server hardware capacities, it will take a few minutes for GitLab to boot all of its services, be patient and don’t panic if you cannot reach your instance immediately after spinning it up.

It’s all done except one thing: SSH port is still 2222 instead of 22, if it bothers you as much as it does me, read on…

Forwarding SSH for git user (Reusing port 22)

This part requires a bit of changing in the docker host, if you are uncomfortable or you don’t care that the repository SSH cloning url has a custom port in it, stop now, otherwise continue on. (changes are reversible don’t worry 😇)

You need to know a bit about how GitLab is using SSH; for the same reason that we don’t like to have a separated port for cloning repositories using SSH, GitLab uses host sshd service, but the question remains: how?

If you have ever thought about it, it’s interesting that all users connecting to your GitLab instance by SSH use the same user (git) but they can only access repositories that they own, it means GitLab detects the user which is connecting; you can check this out yourself, simply add your public SSH key in your GitLab account, then run the following command:

ssh git@example.com -p 2222

You will see something like this:

Welcome to GitLab, @Pezhvak!

Note: I’ve added port 2222 because I assume you are following along, originally you won’t need to specify port (-p) argument since it uses 22 by default, we’re fixing the same issue here.

As you can see, once a user is logged in, a script will get executed which is responsible for doing the stuff that GitLab is doing.

The answer to how it does it is in authorized_keys of the git user, in our setup it’s mounted at /srv/gitlab/data/.ssh/authorized_keys , open it up with your editor of choice or run the following command to see what I’m talking about:

head -n 2 /srv/gitlab/data/.ssh/authorized_keys

You should see something like this:

# Managed by gitlab-rails
command=”/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-13",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa …

Pay attention to the part which starts with command= until ssh-rsa.. (which is a public ssh-key in your system), this command will be executed once the user successfully logs in. gitlab-shell is a bash file located in the container but not in the docker host, we will take care of it right now, don’t worry.

There you go, that’s how GitLab uses a single ssh user (git) for all of the members in GitLab.

Alright, now that you know how it works, let’s get to work. We need to create a fake gitlab-shell bash file to proxy all requests into the container: (save it in/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell no need to say you have to create all the necessary folders on your hosts to be able to create the file: mkdir -p /opt/gitlab/embedded/service/gitlab-shell/bin/)

#!/bin/bash
# Proxy SSH requests to docker container
sudo docker exec -i -u git gitlab sh -c "SSH_CONNECTION='$SSH_CONNECTION' SSH_ORIGINAL_COMMAND='$SSH_ORIGINAL_COMMAND' $0 $1"

Note: don’t forget to make it executable:

chmod +x /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell

But that’s not all, GitLab creates a user and a group named git during installation, docker container has that in place already but we don’t have it in docker host, lets create it.

# creating a group named git
groupadd -g 1234 git# create a user named git and add git group to it
useradd -m -u 1234 -g git -s /bin/sh -d /srv/gitlab/data git

Note: depending your OS, you might need to allow git to execute sudo for docker commands without a password, this is essential for our gitlab-shell script to work:

sudo visudo# Add at the end of the file
git  ALL=(ALL)  NOPASSWD: /usr/bin/docker

User home is set to parent of .ssh directory (/srv/gitlab/data) that contains authorized_keys (that’s how you can login without a password); As you can see we are hardcoding group and user id to 1234, feel free to change it to whatever you want (make sure it’s not already taken) but if you changed it to something else, you have to reflect that in the next step, which is updating our docker-compose.yml file:

version: '2'services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    restart: always
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - dhparam:/etc/nginx/dhparam
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:roletsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    restart: always
    container_name: nginx-proxy-le
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:rogitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.example.com'
    container_name: gitlab
    environment:
      VIRTUAL_HOST: gitlab.example.com,registry.example.com
      LETSENCRYPT_HOST: gitlab.example.com,registry.example.com
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.com'
        registry_external_url 'https://registry.example.com'
        gitlab_rails['gitlab_shell_ssh_port'] = 2222
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        registry_nginx['listen_port'] = 80
        registry_nginx['listen_https'] = false
        manage_accounts['enable']=true
        user['username'] = 'git'
        user['group'] = 'git'
        user['uid'] = 1234
        user['gid'] = 1234
        nginx['proxy_set_headers'] = {  "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on" }
        # Add any other gitlab.rb configuration here, each on its own lineports:
      - '5050:5050'volumes:
      - '$GITLAB_HOME/config:/etc/gitlab'
      - '$GITLAB_HOME/logs:/var/log/gitlab'
      - '$GITLAB_HOME/data:/var/opt/gitlab'volumes:
  conf:
  vhost:
  html:
  dhparam:
  certs:
  acme:

Note: putting GitLab service into a different docker-compose.yml file might result in nginx upstream failure, I'm sure there is a way to fix that, but that’s not what this article is about

These are the changes you should pay attention to:

The gitlab_rails['gitlab_shell_ssh_port'] = 2222 environment variable is gone, because we don’t need it anymore
Port binding 2222:22 is gone too, for the same reason.
Added manage_accounts['enable'], user['username'], user['group'], user['uid'] and user['gid'] environment variables

manage_accounts['enable'] is false by default intentionally, setting it to true lets you change uid and gid (even SSH username which we kept as git).

Why do we do all this? Well, linux identifies file ownership by id not username, by default uid and gid of the user git inside the container is hardcoded as 998, but it’s mostly taken by other users in docker host (maybe not, but it’s highly possible) to make sure we have same uid and gid inside GitLab container and docker host we gave them both 1234 for user id and group id.

For these changes to take effect, re-run docker compose up:

docker-compose up -d

Well, we’re done, unless you’re running on any linux distro with SElinux enabed 🤦🏻‍♂️, don’t worry just run the following command and you’re done:

# you need to run this again if you disable and re-enforce SElinux
sudo chcon -t ssh_home_t /srv/gitlab/data/.ssh/authorized_keys /srv/gitlab/data/.ssh

why? because originally users home are located under /home directory, since we’ve changed that… well, that’s what SElinux is for, more pain; 🙄 This last one took me half a day to figure out.

How to clean all this up?

It seems like we did a lot of changes on the docker host, but honestly we didn’t. All you have to do is:

# remove gitlab-shell proxy
rm /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell
# remove git user from system
userdel git
# remove git group from system
groupdel git

Isn’t there a simpler solution for all this?

The most obvious answer is “of course there is”, you can get a dedicated server just for your GitLab instance, but that’s not what most of us are looking for is it?

Another solution is to request an additional IP address for your server and bind those ports on that specific IP (which is not used by other services), but again, if you are running your GitLab on a VPS like me, lots of the providers don’t give more than 1 IP address per VPS and that’s the reason I wrote this article.

Troubleshooting

If anything goes wrong, you will mostly face permission errors due to changing uid and gid of the git user, simply run docker compose up without -d flag to see logs:

docker-compose up

keep your eye on SElinux 👹 and ACL for /srv/gitlab, what else can go wrong?

# get rid of ACL
setfacl -R -b /srv/gitlab# update gitlab permissions
docker exec gitlab update-permissions

An error occurred while fetching folder content

# by the time of writing this article, these files are left behind in update-permissions, do it manually
chown git:git /srv/gitlab/logs/gitaly/gitaly_*

GitLab Runner Issues

If you are running gitlab-runner container instead of installing it on your server, you might see one of these error messages in your ci/cd pipelines (specially in dind)

error during connect: Post http://docker:2375/v1.24/auth: dial tcp: lookup docker on 4.2.2.4:53: server misbehaving

Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

Relax you only need to change a line in your config.toml of your gitlab-runner. You can find it in the mounted volume to your container (depending on your installation it can be anywhere, mine is: /var/lib/docker/volumes/gitlab-runner-config/_data/config.toml)

# find the line below and change privileged from "false" to "true"
privileged = true

Note: If you have used a command to spin up your runner, make sure to include privileged flag:

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v gitlab-runner-config:/etc/gitlab-runner \
    --privileged \
    gitlab/gitlab-runner:latest

ERROR: Uploading artifacts as "archive" to coordinator... too large archive id=6115 responseStatus=413 Request Entity Too Large status=413 token=RkMqDF8Q

If you’re planning to use gitlabs CI/CD, you might want to increase client_max_body_size for your nginx setup so it will be able to upload large artifacts, do this by creating a file named larger_body_size.conf in /var/lib/docker/volumes/web_conf/_data or any other path that you bind from docker host into the nginx-proxy container path /etc/nginx/conf.d/ with the following contents:

client_max_body_size 1024M;

Suggestions

There are lots of configurations that you can do to this setup, however since the goal is to have a working registry, it’s a good idea to increase some limitation that might occur in your CI/CD pipelines, one of them is nginx client_max_body_size , simply create a file with the name of your container registry domain (the same as registry_external_url) and put this in it:

# /var/lib/docker/volumes/sites_vhost/_data/registry.example.com
client_max_body_size 500M;

Conclusion

I know it was a lengthy post for such a simple goal, but if you made it to the end congratulations 🎉🎉🎉

It took me almost 3 days to figure all the parts out, the most important take-away is the final docker-compose.yml and how GitLab is using a single SSH user for all members so you can proxy it into the container.

special thanks to Ardavan Ansari for proofreading this article.