Set up GoCD Server and Agent with AWS OpsWorks

GoCD team is going to opensource their Chef cookbooks soon that used to manage their build infrastructure.  Before the cookbooks are available for public, I have taken few of them and create a private GitHub repository and used it to setup GoCD server and agent with AWS OpsWorks.

* This is a non-SSL GoCD Server setup.  For SSL (Apache fronted) configuration, it will be on my next post.

** I will update this post when GoCD team makes their cookbooks available for public

Before we create a new stack, we need to define two new security groups in order for Go agent to communicate with Go server through the internal network.

Screen Shot 2016-01-08 at 1.56.06 PM.pngScreen Shot 2016-01-08 at 1.58.43 PM.png

Screen Shot 2016-01-08 at 1.58.48 PM.png

Screen Shot 2016-01-09 at 9.24.57 AM.png

Screen Shot 2016-01-09 at 9.25.07 AM.png

After the two security groups are created, go to OpsWorks and create a new Chef 12 stack

Screen Shot 2016-01-08 at 2.00.45 PM.png

Screen Shot 2016-01-09 at 9.26.28 AM.png

After the stack is created, let’s create a new layer called “GoCD Server”

Screen Shot 2016-01-08 at 2.01.05 PM.png

In the “General Settings” of GoCD Server layer, we will provide few attributes in JSON that will be used by the GoCD cookbooks to configure the GoCD server

Screen Shot 2016-01-08 at 2.01.50 PM.png

Screen Shot 2016-01-08 at 11.43.52 PM.png

Then add three recipes, “yum-pgdg::default”, “yum-gocd::default” and “go-server-wrapper::default”

Screen Shot 2016-01-08 at 11.20.53 PM.png

Under “Security”, we will add three security groups to the GoCD Server layer, “AWS-OpsWorks-Default-Server”, “AWS-OpsWorks-GoCD-Server” and “AWS-OpsWorks-Web-Server”

Screen Shot 2016-01-08 at 11.22.31 PM.png

After GoCD Server Layer is ready, we can add a new instance called “gocd-server”Screen Shot 2016-01-08 at 8.32.25 PM.png

Wait for the new instance to be ready.

Screen Shot 2016-01-08 at 8.39.51 PM.png

Find the Public DNS of this GoCD Server instance

Screen Shot 2016-01-08 at 8.39.56 PM.png

Open your browser and go to that URL, you GoCD Server is ready.Screen Shot 2016-01-08 at 8.40.05 PM.png

Let’s enable Go agent auto registration on the Go Server.

Screen Shot 2016-01-08 at 10.31.04 PM.png

To setup GoCD agent, we will define a new layer called “GoCD Linux Dev Agent”

Screen Shot 2016-01-08 at 8.41.25 PM.png

Screen Shot 2016-01-08 at 8.41.30 PM.png

In the “General Settings” of GoCD Linux Dev Agent layer, we will provide few attributes in JSON that will be used by the GoCD cookbooks to configure the GoCD Linux Dev Agent.

Screen Shot 2016-01-08 at 8.42.40 PM.png

Screen Shot 2016-01-08 at 11.45.46 PM.png

Then add three recipes, “yum-repoforge::default”, “yum-gocd::default” and “go-agent::default”

Screen Shot 2016-01-08 at 8.43.32 PM.png

Apply the two Security Groups “AWS-OpsWorks-Default-Server” and “AWS-OpsWorks-GoCD-Agent” to this layer.

Screen Shot 2016-01-08 at 8.44.00 PM.png

Add a new instance in this layer and start the instance

Screen Shot 2016-01-08 at 8.44.24 PM.png

Screen Shot 2016-01-08 at 8.44.31 PM.png

When this instance is ready, you will see the agent automatically register on Go Server.

Screen Shot 2016-01-08 at 10.49.46 PM.png

Screen Shot 2016-01-08 at 10.49.52 PM.png

 

 

This entry was posted in GoCD and tagged , . Bookmark the permalink.

5 Responses to Set up GoCD Server and Agent with AWS OpsWorks

  1. @barrow do you know if gocd team has published their chef cookbooks?

  2. barrowkwan says:

    Yes they do. Search Chef’s Supermarket

  3. John Kordyback says:

    Thanks Barrow, this is terrific.

  4. lynndupree06 says:

    I have been having trouble trying to set this up properly. I followed each step as close as possible, but I did a few things differently because I could see that some of the things in this post are outdated:

    I set the source of the custom Chef cookbooks to the following public repo: https://github.com/gocd/go-cookbook.git
    Because I was using a different repo than in the example, based off of what I saw in the code (i.e. The metadata.rb file), I figured that the only custom chef recipe I needed to setup the GoCD Server layer was good::server, and gocd::agent for the GoCD Agent layer
    I also used the latest Amazon Linux version

    However, I ran into these issues:

    When I start the good-server instance, it runs the cookbook good::server and I get the following error: No such cookbook: apt

    I added the following (which I derived from details in the README of the repo) to the Custom JSON of the GoCD Server layer in hopes to fixing this error, but this did not work.

    {
      ...
      "gocd": {
        ...
        "repository": {
          "apt": {
            "uri": "https:\/\/download.gocd.io\/",
            "components": [
              "\/"
            ],
            "distribution": "",
            "keyserver": "pgp.mit.edu",
            "key": "https:\/\/download.gocd.io\/GOCD-GPG-KEY.asc"
          }
        }
      }
    }
    

    I was not able to find anything else online to help me solve or fix this error, so I eventually just resulted in removing the custom cookbook and manually installing the GoCD server using the steps outlined in the documentation here.

    I was not able to access the server due to the security group only taking inbound requests from IP address in the GoCD Agent security group. So I had to change the source to Anywhere, and I was able to navigate to the GoCD server through the browser and start to configure it.

    When I start the gocd-linux-dev-agent1 instance, it runs the cookbook good::agent and I get the following error: No such cookbook: apt

    Because it has the same issue as the GoCD Server layer, I went straight to trying to install it manually on the machine using the documentation here.

    After setting the environment variable GO_SERVER_URL in the /etc/default/go-agent file on the GoCD Agent machine the agent started successfully, however, in the logs (found in /var/log/go-agent/go-agent-bootstrapper.out.log) I received the following error: failed: connect timed out. So, this also lead me to have to change the GoCD Server security group to accept any inbound IP address.

    Once I got the go-agent to start, I was not able to see it as a listed agent on the GoCD Server.
    I also tried to set the agent.auto.register.key inside of the /var/lib/go-agent/config/autoregister.properties (I wasn’t sure if that was the correct path that file should be in) with the same agentAutoRegisterKey I saw in the GoCD Server configuration file. It didn’t work, even after restarting the server.

    I do not know what I am doing wrong. The farthest I have gotten is having a GoCD Server up and running, but with no agents. Do you have some insight on what I could be doing wrong? Thanks.

  5. barrowkwan says:

    Hi, I am working on a new instruction with some new code book to simplify the setup. I should have it ready in a day or two.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s