We’re huge fans of Gitlab at Anlek and about 6 months ago we decided to retire an old server hosting a custom built of gitlab for a newer, easier to update, docker driven one. We upgrade to the new gitlab server and It was really smooth to setup when using the docker image from sameersbn. The gitlab server has been working great and sameersbn did a great job allowing you to customize it and has been keeping it up to date! Despite a smooth start, we’ve noticed that git commands which involved the gitlab server (pull/push/fetch/etc) were failing at a rate of 1 in 2 commands!

It wasn’t a big issue, more of a nuisance but yesterday it became a show stopper! We’ve been using docker to deploy most of the new applications we develop, but a client reported a bug in an old application. After a quick fixed, we tried to push the fix to the server (using Capistrano 2) but it resulted in a failed deploy! This was due to Capistrano running multiple git commands to the remote server during deploy and since 50% of them fail, the deploy could not complete successfully.

After a bunch of attempts to figure out what was wrong, including updating the gitlab docker image, updating dev machine’s SSH client and the client server’s SSH client/server, nothing was working. So after many hours of trying different work arounds and not being able to solve the issue. The solution came to us when we were on a coffee break. We had setup a load balancer to balance, as well as, apply SSL to our gitlab server and by default it load balanced all ports, including port 22 (SSH) as well! After a quick remap of the ports on Rancher and we had success!

Not only did this solve the issue of the deploy, but now we no longer have a 1 in 2 chances of git push failing during development!

We’re not saying this is the only thing that causes the ssh_exchange_identification: Connection closed by remote host error. There is a great article of things that could be wrong; check out https://edoceo.com/notabene/ssh-exchange-identification. It also seems that at one point gitlab.com (hosted gitlab server) had a similar issue.

Good luck with your battle, leave a comment if you feel we missed anything or you’d like us to clarify anything.