Building a Concourse CI VM on Ubuntu

Recently, I’ve found myself needing a Concourse CI system. I struggled with the documentation on concourse.ci, couldn’t find any comprehensive build guides.  Knew for certain I wasn’t going to use VirtualBox.  So, having worked it out; thought I’d share what I went through to get to a working system.

Starting Position
Discovered that the CentOS version I was using previously did not have a compatible Linux kernel version.  CentOS 7.2 uses kernel 3.10, Concourse requires 3.19+.  So, I’m starting with a freshly-deployed Ubuntu Server 16.04 LTS this time.

Prep Ubuntu
Not a lot we have to do, but still pretty important:

1. Make sure port for concourse is open
   

   sudo ufw allow 8080
sudo ufw status

   sudo ufw disable

   I disabled the firewall on ubuntu because it was preventing the concourse worker and concourse web from communicating.

2. Update and make sure wget is installed
   

   apt-get update
apt-get install wget

Postgresql
Concourse expects to use a postgresql database, I don’t have one standing by, so let’s install it.

1. Pretty straightforward on Ubuntu too:
   

   apt-get install postgresql postgresql-contrib

   Enter y to install the bits.  On Ubuntu, we don’t have to take extra steps to configure the service.

2. Ok, now we have to create an account and a database for concourse. First, lets create the linux account. I’m calling mine “concourse” because I’m creative like that.
   

   adduser concourse
passwd concourse

   

3. Next, we create the account (aka “role” or “user”) in postgres via the createuser command. In order to do this, we have to switch to the postgres account, do that with sudo:
   

   sudo -i -u postgres

   Now, while in as postgres we can use the createuser command

   createuser –interactive

   You’ll enter the name of the account, and answer a couple of special permissions questions.

4. While still logged in as postgres, run this command to create a new database for concourse. I’m naming my database “concourse” – my creativity is legendary. Actually, I think it makes life easier if the role and database are named the same
   

   createdb concourse

5. Test by switching users to the concourse account and making sure it can run psql against the concourse databaseWhile in psql, use this command to set the password for the account in postgress
   

   ALTER ROLE concourse WITH PASSWORD 'changeme';

6. Type \q to exit psql

Concourse
Ok, we have a running postgresql service and and account to be used for concourse. Let’s go.

1. Create a folder for concourse. I used /concourse, but you can use /var/lib/whatever/concourse if you feel like it.
2. Download the binary from concourse.ci/downloads.html into your /concourse folder using wget or transfer via scp.
3. Create a symbolic link named “concourse” to the file you downloaded and make it executable
   

   ln -s ./concourse_linux_amd64 ./concourse
chmod +x ./concourse_linux_amd64

4. Create keys for concourse
   

   cd /concourse

   mkdir -p keys/web keys/worker

   ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N ”
   ssh-keygen -t rsa -f ./keys/web/session_signing_key -N ”
   ssh-keygen -t rsa -f ./keys/worker/worker_key -N ”
   cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
   cp ./keys/web/tsa_host_key.pub ./keys/worker

5. Create start-up script for Concourse. Save this as /concourse/start.sh:
   

   /concourse/concourse web \
   –basic-auth-username myuser \
   –basic-auth-password mypass \
   –session-signing-key /concourse/keys/web/session_signing_key \
   –tsa-host-key /concourse/keys/web/tsa_host_key \
   –tsa-authorized-keys /concourse/keys/web/authorized_worker_keys \
   –external-url http://192.168.103.81:8080 \
   –postgres-data-source postgres://concourse:changeme@127.0.0.1/concourse?sslmode=disable

   /concourse/concourse worker \
   –work-dir /opt/concourse/worker \
   –tsa-host 127.0.0.1 \
   –tsa-public-key /concourse/keys/worker/tsa_host_key.pub \
   –tsa-worker-private-key /concourse/keys/worker/worker_key

   The items in red should definitely be changed for your environment. “external_url” uses the IP address of the VM its running on. and the username and password values in the postgres-data-source should reflect what you set up earlier. Save the file and be sure to set it as executable (chmod +x ./start.sh)

6. Run the script “./start.sh”. You should see several lines go by concerning worker-collectors and builder-reapers.
   • If you instead see a message about authentication, you’ll want to make sure that 1) the credentials in the script are correct, 2) the account has not had it’s password set in linux or in postgres
   • If you instead see a message about the connection not accepting SSL, be sure that the connection string in the script includes “?sslmode=disable” after the database name
7. Test by pointing a browser at the value you assigned to the external_url. You should see “no pipelines configured”.  You can login using the basic-auth username and password you specified in the startup script.
   

   Success!

8. Back in your SSH session, you can kill it with <CRTL>+C

Finishing Up
Now we just have to make sure that concourse starts when the system reboots. I am certain that there are better/safer/more reliable ways to do this, but here’s what I did:
Use nano or your favorite text editor to add “/concourse/start.sh” to /etc/rc.local ABOVE the line that reads “exit 0”
Now, reboot your VM and retest the connectivity to the concourse page.

Thanks

EMC ECS Community Edition project for how to start the script on boot.

Mitchell Anicas’ very helpful post on setting up postgres on Ubuntu.

Concourse.ci for some wholly inadequate documentation

Alfredo Sánchez for bringing the issue with Concourse and CentOS to my attention

Building a Concourse CI VM on CentOS

Recently, I’ve found myself needing a Concourse CI system. I struggled with the documentation on concourse.ci, couldn’t find any comprehensive build guides.  Knew for certain I wasn’t going to use VirtualBox.  So, having worked it out; thought I’d share what I went through to get to a working system.

WARNING

It has been brought to my attention that CentOS does not have a compatible Linux kernel, so I’ve redone this post using Ubuntu instead.

Starting Position
I’m starting with a freshly-deployed CentOS 7 VM. I use Simon’s template build, so it comes up quickly and reliably.  Logged on as root.

Prep CentOS
Not a lot we have to do, but still pretty important:

1. Open firewall post for concourse
   

   firewall-cmd --add-port=8080/tcp --permanent
firewall-cmd --reload

   optionally, you can open 5432 for postgres if you feel like it

2. Update and make sure wget is installed
   

   yum update
yum install wget

Postgresql
Concourse expects to use a postgresql database, I don’t have one standing by, so let’s install it.

1. Pretty straightforward on CentOS:
   

   yum install postgresql-server postgresql-contrib

   Enter y to install the bits.

2. When that step is done, we’ll set it up with this command:
   

   sudo postgresql-setup initdb

3. Next, we’ll update the postgresql config to allow passwords. Use your favorite editor to open /var/lib/pgsql/data/pg_hba.conf We need to update the value in the method column for IPv4 and IPv6 connections from “ident” to “md5” then save the file.
   

   Before

   

   After

4. Now, let’s start postgresql and set it to run automatically
   

   sudo systemctl start postgresql
sudo systemctl enable postgresql

5. Ok, now we have to create an account and a database for concourse. First, lets create the linux account. I’m calling mine “concourse” because I’m creative like that.
   

   adduser concourse
passwd concourse

   

6. Next, we create the account (aka “role” or “user”) in postgres via the createuser command. In order to do this, we have to switch to the postgres account, do that with sudo:
   

   sudo -i -u postgres

   Now, while in as postgres we can use the createuser command

   createuser –interactive

   You’ll enter the name of the account, and answer a couple of special permissions questions.

7. While still logged in as postgres, run this command to create a new database for concourse. I’m naming my database “concourse” – my creativity is legendary. Actually, I think it makes life easier if the role and database are named the same
   

   createdb concourse

8. Test by switching users to the concourse account and making sure it can run psql against the concourse databaseWhile in psql, use this command to set the password for the account in postgress
   

   ALTER ROLE concourse WITH PASSWORD 'changeme';

9. Type \q to exit psql

Concourse
Ok, we have a running postgresql service and and account to be used for concourse. Let’s go.

1. Create a folder for concourse. I used /concourse, but you can use /var/lib/whatever/concourse if you feel like it.
2. Download the binary from concourse.ci/downloads.html into your /concourse folder using wget or transfer via scp.
3. Create a symbolic link named “concourse” to the file you downloaded and make it executable
   

   ln -s ./concourse_linux_amd64 ./concourse
chmod +x ./concourse_linux_amd64

4. Create keys for concourse
   

   cd /concourse

   mkdir -p keys/web keys/worker

   ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N ”
   ssh-keygen -t rsa -f ./keys/web/session_signing_key -N ”
   ssh-keygen -t rsa -f ./keys/worker/worker_key -N ”
   cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
   cp ./keys/web/tsa_host_key.pub ./keys/worker

5. Create start-up script for Concourse. Save this as /concourse/start.sh:
   

   /concourse/concourse web \
   –basic-auth-username myuser \
   –basic-auth-password mypass \
   –session-signing-key /concourse/keys/web/session_signing_key \
   –tsa-host-key /concourse/keys/web/tsa_host_key \
   –tsa-authorized-keys /concourse/keys/web/authorized_worker_keys \
   –external-url http://192.168.103.81:8080 \
   –postgres-data-source postgres://concourse:changeme@127.0.0.1/concourse?sslmode=disable

   /concourse/concourse worker \
   –work-dir /opt/concourse/worker \
   –tsa-host 127.0.0.1 \
   –tsa-public-key /concourse/keys/worker/tsa_host_key.pub \
   –tsa-worker-private-key /concourse/keys/worker/worker_key

   The items in red should definitely be changed for your environment. “external_url” uses the IP address of the VM its running on. and the username and password values in the postgres-data-source should reflect what you set up earlier. Save the file and be sure to set it as executable (chmod +x ./start.sh)

6. Run the script “./start.sh”. You should see several lines go by concerning worker-collectors and builder-reapers.
   • If you instead see a message about authentication, you’ll want to make sure that 1) the credentials in the script are correct, 2) the account has not had it’s password set in linux or in postgres and 3) the pg_hba.conf fie has been updated to use md5 instead of ident
   • If you instead see a message about the connection not accepting SSL, be sure that the connection string in the script includes “?sslmode=disable” after the database name
7. Test by pointing a browser at the value you assigned to the external_url. You should see “no pipelines configured”
   

   Success!

8. Back in your SSH session, you can kill it with <CRTL>+X

Finishing Up
Now we just have to make sure that concourse starts when the system reboots. I am certain that there are better/safer/more reliable ways to do this, but here’s what I did:

echo "/concourse/start.sh" >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local

Now, reboot your VM and retest the connectivity to the concourse page.

Thanks

EMC ECS Community Edition project for how to start the script on boot.

Mitchell Anicas’ very helpful post on setting up postgres on CentOS.

Concourse.ci for some wholly inadequate documentation