2019-10-12 21:52:04 +05:30
|
|
|
---
|
|
|
|
type: reference
|
|
|
|
---
|
|
|
|
|
|
|
|
# Kubernetes, GitLab and You
|
|
|
|
|
|
|
|
This is a list of useful information regarding Kubernetes that the GitLab Support
|
|
|
|
Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone
|
|
|
|
can make use of the Support team's collected knowledge
|
|
|
|
|
|
|
|
CAUTION: **Caution:**
|
|
|
|
These commands **can alter or break** your Kubernetes components so use these at your own risk.
|
|
|
|
|
|
|
|
If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how
|
|
|
|
to use these commands, it is best to [contact Support](https://about.gitlab.com/support/)
|
|
|
|
and they will assist you with any issues you are having.
|
|
|
|
|
2019-12-21 20:55:43 +05:30
|
|
|
## Generic Kubernetes commands
|
2019-10-12 21:52:04 +05:30
|
|
|
|
|
|
|
- How to authorize to your GCP project (can be especially useful if you have projects
|
|
|
|
under different GCP accounts):
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
gcloud auth login
|
|
|
|
```
|
|
|
|
|
|
|
|
- How to access Kubernetes dashboard:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
# for minikube:
|
|
|
|
minikube dashboard —url
|
2019-12-04 20:38:33 +05:30
|
|
|
# for non-local installations if access via Kubectl is configured:
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl proxy
|
|
|
|
```
|
|
|
|
|
2019-12-21 20:55:43 +05:30
|
|
|
- How to SSH to a Kubernetes node and enter the container as root
|
2019-10-12 21:52:04 +05:30
|
|
|
<https://github.com/kubernetes/kubernetes/issues/30656>:
|
|
|
|
|
|
|
|
- For GCP, you may find the node name and run `gcloud compute ssh node-name`.
|
|
|
|
- List containers using `docker ps`.
|
|
|
|
- Enter container using `docker exec --user root -ti container-id bash`.
|
|
|
|
|
|
|
|
- How to copy a file from local machine to a pod:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl cp file-name pod-name:./destination-path
|
|
|
|
```
|
|
|
|
|
|
|
|
- What to do with pods in `CrashLoopBackoff` status:
|
|
|
|
|
|
|
|
- Check logs via Kubernetes dashboard.
|
2019-12-04 20:38:33 +05:30
|
|
|
- Check logs via Kubectl:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl logs <unicorn pod> -c dependencies
|
|
|
|
```
|
|
|
|
|
|
|
|
- How to tail all Kubernetes cluster events in real time:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl get events -w --all-namespaces
|
|
|
|
```
|
|
|
|
|
|
|
|
- How to get logs of the previously terminated pod instance:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl logs <pod-name> --previous
|
|
|
|
```
|
|
|
|
|
|
|
|
NOTE: **Note:**
|
|
|
|
No logs are kept in the containers/pods themselves, everything is written to stdout.
|
|
|
|
This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/)
|
|
|
|
for details.
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
- How to get cronjobs configured on a cluster
|
|
|
|
|
2020-04-08 14:13:33 +05:30
|
|
|
```shell
|
2020-03-13 15:44:24 +05:30
|
|
|
kubectl get cronjobs
|
|
|
|
```
|
2020-04-08 14:13:33 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
When one configures [cron-based backups](https://docs.gitlab.com/charts/backup-restore/backup.html#cron-based-backup),
|
|
|
|
you will be able to see the new schedule here. Some details about the schedules can be found
|
|
|
|
in [Running Automated Tasks with a CronJob](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#creating-a-cron-job)
|
2020-04-08 14:13:33 +05:30
|
|
|
|
2019-12-21 20:55:43 +05:30
|
|
|
## GitLab-specific Kubernetes information
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2019-12-21 20:55:43 +05:30
|
|
|
- Minimal config that can be used to test a Kubernetes Helm chart can be found
|
2019-12-04 20:38:33 +05:30
|
|
|
[here](https://gitlab.com/gitlab-org/charts/gitlab/issues/620).
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2019-12-21 20:55:43 +05:30
|
|
|
- Tailing logs of a separate pod. An example for a Unicorn pod:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Tail and follow all pods that share a label (in this case, `unicorn`):
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-12-04 20:38:33 +05:30
|
|
|
# all containers in the unicorn pods
|
|
|
|
kubectl logs -f -l app=unicorn --all-containers=true --max-log-requests=50
|
|
|
|
|
|
|
|
# only the unicorn containers in all unicorn pods
|
|
|
|
kubectl logs -f -l app=unicorn -c unicorn --max-log-requests=50
|
|
|
|
```
|
|
|
|
|
|
|
|
- One can stream logs from all containers at once, similar to the Omnibus
|
|
|
|
command `gitlab-ctl tail`:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-12-04 20:38:33 +05:30
|
|
|
kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100
|
|
|
|
```
|
2019-10-12 21:52:04 +05:30
|
|
|
|
|
|
|
- Check all events in the `gitlab` namespace (the namespace name can be different if you
|
2019-12-21 20:55:43 +05:30
|
|
|
specified a different one when deploying the Helm chart):
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl get events -w --namespace=gitlab
|
|
|
|
```
|
|
|
|
|
|
|
|
- Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner
|
|
|
|
pod. You may enter it and run commands inside or run them from the outside:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
# find the pod
|
|
|
|
kubectl get pods | grep task-runner
|
|
|
|
|
|
|
|
# enter it
|
|
|
|
kubectl exec -it <task-runner-pod-name> bash
|
|
|
|
|
|
|
|
# open rails console
|
|
|
|
# rails console can be also called from other GitLab pods
|
|
|
|
/srv/gitlab/bin/rails console
|
|
|
|
|
|
|
|
# source-style commands should also work
|
|
|
|
/srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production
|
|
|
|
|
|
|
|
# run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart
|
|
|
|
/usr/local/bin/gitlab-rake gitlab:check
|
|
|
|
|
|
|
|
# open console without entering pod
|
|
|
|
kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console
|
|
|
|
|
|
|
|
# check the status of DB migrations
|
|
|
|
kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status
|
|
|
|
```
|
|
|
|
|
|
|
|
You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`.
|
|
|
|
|
|
|
|
- Troubleshooting **Operations > Kubernetes** integration:
|
|
|
|
|
|
|
|
- Check the output of `kubectl get events -w --all-namespaces`.
|
|
|
|
- Check the logs of pods within `gitlab-managed-apps` namespace.
|
2019-12-21 20:55:43 +05:30
|
|
|
- On the side of GitLab check Sidekiq log and Kubernetes log. When GitLab is installed
|
|
|
|
via Helm Chart, `kubernetes.log` can be found inside the Sidekiq pod.
|
2019-10-12 21:52:04 +05:30
|
|
|
|
|
|
|
- How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
# find the name of the secret containing the password
|
|
|
|
kubectl get secrets | grep initial-root
|
|
|
|
# decode it
|
|
|
|
kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- How to connect to a GitLab Postgres database:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p
|
|
|
|
```
|
2020-04-08 14:13:33 +05:30
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- How to get info about Helm installation status:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
helm status name-of-installation
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- How to update GitLab installed using Helm Chart:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
helm repo upgrade
|
|
|
|
|
|
|
|
# get current values and redirect them to yaml file (analogue of gitlab.rb values)
|
|
|
|
helm get values <release name> > gitlab.yaml
|
|
|
|
|
|
|
|
# run upgrade itself
|
|
|
|
helm upgrade <release name> <chart path> -f gitlab.yaml
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
After <https://gitlab.com/gitlab-org/charts/gitlab/issues/780> is fixed, it should
|
|
|
|
be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart)
|
2019-10-12 21:52:04 +05:30
|
|
|
for upgrades.
|
|
|
|
|
|
|
|
- How to apply changes to GitLab config:
|
|
|
|
|
|
|
|
- Modify the `gitlab.yaml` file.
|
|
|
|
- Run the following command to apply changes:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
helm upgrade <release name> <chart path> -f gitlab.yaml
|
|
|
|
```
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
- How to get the manifest for a release. It can be useful because it contains the info about
|
|
|
|
all Kubernetes resources and dependent charts:
|
|
|
|
|
2020-04-08 14:13:33 +05:30
|
|
|
```shell
|
|
|
|
helm get manifest <release name>
|
|
|
|
```
|
2020-03-13 15:44:24 +05:30
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
## Installation of minimal GitLab config via Minukube on macOS
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2019-12-21 20:55:43 +05:30
|
|
|
This section is based on [Developing for Kubernetes with Minikube](https://docs.gitlab.com/charts/development/minikube/index.html)
|
|
|
|
and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer
|
2019-10-12 21:52:04 +05:30
|
|
|
to those documents for details.
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Install Kubectl via Homebrew:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
brew install kubernetes-cli
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Install Minikube via Homebrew:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
brew cask install minikube
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Start Minikube and configure it. If Minikube cannot start, try running `minikube delete && minikube start`
|
2019-10-12 21:52:04 +05:30
|
|
|
and repeat the steps:
|
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work
|
|
|
|
minikube addons enable ingress
|
|
|
|
minikube addons enable kube-dns
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Install Helm via Homebrew and initialize it:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
brew install kubernetes-helm
|
|
|
|
helm init --service-account tiller
|
|
|
|
```
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Copy the file <https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml>
|
2019-10-12 21:52:04 +05:30
|
|
|
to your workstation.
|
|
|
|
|
|
|
|
- Find the IP address in the output of `minikube ip` and update the yaml file with
|
|
|
|
this IP address.
|
|
|
|
|
2019-12-04 20:38:33 +05:30
|
|
|
- Install the GitLab Helm Chart:
|
2019-10-12 21:52:04 +05:30
|
|
|
|
2020-03-13 15:44:24 +05:30
|
|
|
```shell
|
2019-10-12 21:52:04 +05:30
|
|
|
helm repo add gitlab https://charts.gitlab.io
|
|
|
|
helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab
|
|
|
|
```
|
|
|
|
|
|
|
|
If you want to modify some GitLab settings, you can use the above-mentioned config
|
|
|
|
as a base and create your own yaml file.
|
|
|
|
|
|
|
|
- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`.
|
|
|
|
The installation could take up to 20-30 minutes depending on the amount of resources
|
|
|
|
on your workstation.
|
|
|
|
|
|
|
|
- When all the pods show either a `Running` or `Completed` status, get the GitLab password as
|
2019-12-04 20:38:33 +05:30
|
|
|
described in [Initial login](https://docs.gitlab.com/charts/installation/deployment.html#initial-login),
|
2019-10-12 21:52:04 +05:30
|
|
|
and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain`
|
|
|
|
where `domain` is the value provided in the yaml file.
|
|
|
|
|
|
|
|
<!-- ## Troubleshooting
|
|
|
|
|
|
|
|
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
|
|
|
|
one might have when setting this up, or when something is changed, or on upgrading, it's
|
|
|
|
important to describe those, too. Think of things that may go wrong and include them here.
|
|
|
|
This is important to minimize requests for support, and to avoid doc comments with
|
|
|
|
questions that you know someone might ask.
|
|
|
|
|
|
|
|
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
|
|
|
|
If you have none to add when creating a doc, leave this section in place
|
|
|
|
but commented out to help encourage others to add to it in the future. -->
|