“502 bad gateway” Gitlab error in nginx server

Before describing the diagnosis and troubleshooting methods that I used to solve this problem, allow me to give you a little rundown on what this is all about.

GitHub: It is a wonderful tool that make managing and administering lots of Git repositories and their associated permissions on a centralized server quite easy. What if you want to do something like this on your own servers where your developers can store their projects and network? GitLab gives you complete control over your repositories and allows you to decide whether they are public, private or free. It’s a github clone to be more specific running on your own hardware.

To learn more about the minimum requirements to run your own gitlab service click here.

Hunting down the errors with GitLab:

One of our gitlab servers started throwing up “”502 bad gateway” error one fine day.  We have a Ubuntu server with nginx+gitlab+ISPconfig (ISPconfig is a free hosting control panel) to server as a gitlab server.

Troubleshooting:

The usual routine is to check the error logs pertaining to the service.

We found this from gitlab_erorr.log

2012/09/21 23:42:29 [error] 19689#0: *5 connect() to unix:/home/gitlab/gitlab/tmp/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: ****, server: gitlab..***, request: “GET / HTTP/1.1″, upstream:

Bad Gateway means nginx wasn’t able to connect to the upstream socket.

Three possible reasons:-

1. Misconfigured path to the socket
2. Unicorn is not running.
3. Unicorn crashes intermittently due to resource shortage

There was no resource shortage. Server was in ship-shape.

We tried to restart gitlab

root@server:/etc/nginx/sites-available# /etc/init.d/gitlab restart

Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran.
Removing stale Sidekiq web server pid. This is most likely caused by the Sidekiq crashing the last time it ran.
Starting the GitLab Unicorn web server…
master failed to start, check stderr log for details
Starting the GitLab Sidekiq event dispatcher…
The GitLab Unicorn webserver is not running.
The GitLab Sidekiq job dispatcher with pid 6296 is running.

The solution still eluded us. After a couple of minutes googling using “The GitLab Unicorn webserver is not running” as search keyword. We stumbled on the possibility of a port number mismatch/misconfiguration. We decided to check that.

Root Cause

We noticed that ISPconfig and Unicorn were running on same port 8080. Now that is the root cause of this problem. One of our associates had installed ISPconfig couple of hours before and inadvertently used the port number 8080 reserved for unicorn.

We changed ISP config port to 8081 from here:-

/etc/nginx/sites-available/ispconfig.vhost

/home/git/gitlab/config/unicorn.rb

Restarted ngnix and gitlab

Now both services are no longer sharing one port.

root@server:/home/git/gitlab/config# netstat -tulpn | grep :808
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 6692/unicorn.rb -E
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 6627/nginx

Source: InstaCarma