Go (Language – Golang) with GoCD – Part Two

1. A standalone app with unversioned external/third parties library

In this example, the app required external libraries.  In this example, my app use “stringutil” from golang example.  When I said “unversioned”, I mean my app does not care which version ( version from a year ago or latest ) of stringutil GoCD will use to build my app, it will use “go get” to pull the latest version as of the build run.

Create a new pipeline in GoCD called “App_with_Unversioned_Libraries”

Screen Shot 2016-10-07 at 8.09.10 PM.png

Screen Shot 2016-10-07 at 8.27.40 PM.png

Similar to the “Standalone” app from my previous post,  check out the repos to src/github.com/barrowkwan/gocd_golang_hello_ext.  If Go agent’s root folder is /var/lib/go-agent/pipelines, you will find the repos checkout at /var/lib/go-agent/pipelines/App_with_Unversioned_Libraries/src/github.com/barrowkwan/gocd_golang_hello_ext.

** You cannot specify “Destination Directory” in the “Add Pipeline” screen, you will need to save the pipeline and use “Edit Material” to update “Destination Directory”.

Screen Shot 2016-10-07 at 8.29.08 PM.png

Create a stage and a job in that stage, then create job that run the build script ( src/github.com/barrowkwan/gocd_golang_hello_ext/build.sh )

Screen Shot 2016-10-07 at 8.29.44 PM.png

When Go run this build script, it will use Go Agent’s root folder  ( ie /var/lib/go-agent/pipelines/App_with_Unversioned_Libraries ) as the working directory.  Since our repos has checkout under /var/lib/go-agent/pipelines/App_with_Unversioned_Libraries as src/github.com/barrowkwan/gocd_golang_hello_ext, if we configure GOPATH to /var/lib/go-agent/pipelines/App_with_Unversioned_Libraries, “go” will be able to look up our app or any other external libraries in the src folder.

Screen Shot 2016-10-07 at 8.36.25 PM.png

In my build.sh script, I use ‘pwd’ to set up environment variable GOPATH.  Unless you configure your job in GoCD to run your script from a different working directory; otherwise, the script will run from Go agent’s root folder ( ie /var/lib/go-agent/pipelines/App_with_Unversioned_Libraries ).  Since this app needs stringutil, when the build script called “go get”, it will download latest version of stringutil from github to src folder ( ie /var/lib/go-agent/pipelines/App_with_Unversioned_Libraries/src/github.com/golang/example/stringutil )

When build.sh run “go build”, it will build my app with the version checkout by GoCD under src/github.com/barrowkwan/gocd_golang_hello_ext and an “unversioned” stringutil under src/github.com/golang/example/stringutil.

Posted in GoCD, Golang | Leave a comment

Go (Language) with GoCD

I started learning Go about two to three months ago but didn’t see people use it in CI like GoCD.  Since Go has different file structure layout than other programming language like Ruby, Python or Java , how to run test and build the app in GoCD is slightly different.  Here are three different scenarios that might help people to setup their pipeline in GoCD with their Golang project.  This examples will give you a basic idea on how to setup the pipeline like how to build the file structure, what to set on various GO related environment variables like GOPATH, GOBIN, etc.

1. A standalone app without any external/third parties library

In this example, the app only use Go built-in libraries.  It doesn’t need “go get” to pull any external/third library.  My sample app is on GitHub (https://github.com/barrowkwan/gocd_golang_hello.git)

Create a new pipeline in GoCD called “Standalone”

Screen Shot 2016-08-09 at 10.56.12 PMCheck out the repos to src/github.com/barrowkwan/gocd_golang_hello.  If you are familiar with Go directory structure, you will know why the repos need to check out in this location.  If Go agent’s root folder is /var/lib/go-agent/pipelines, you will find the repos checkout at /var/lib/go-agent/pipelines/Standalone/src/github.com/barrowkwan/gocd_golang_hello.

Screen Shot 2016-08-09 at 10.56.38 PM

Create a stage and a job in that stage, then create job that run the build script ( src/github.com/barrowkwan/gocd_golang_hello/build.sh )

Screen Shot 2016-08-09 at 10.56.47 PM

When Go run this build script, it will use Go Agent’s root folder  ( ie /var/lib/go-agent/pipelines/Standalone ) as the working directory.  Since our repos has checkout under /var/lib/go-agent/pipelines/Standalone as src/github.com/barrowkwan/gocd_golang_hello, if we configure GOPATH to /var/lib/go-agent/pipelines/Standalone, “go” will be able to look up our app or any other external libraries.

Screen Shot 2016-08-09 at 10.56.52 PM

In my build.sh script, I use ‘pwd’ to set up environment variable GOPATH.  Unless you configure your job in GoCD to run your script from a different working directory; otherwise, the script will run from Go agent’s root folder ( ie /var/lib/go-agent/pipelines/Standalone ).  If you run “go get <other external library>” in your script, it will download the library to /var/lib/go-agent/pipelines/Standalone/src.  I also setup environment variable GOBIN so that “go get” or “go install” will install the binary under /var/lib/go-agent/pipelines/Standalone/bin instead of the default OS location ( eg /opt/go/bin )

When build.sh run “go build”, it will build my app with the version checkout by GoCD under src/github.com/barrowkwan/gocd_golang_hello.

 

** I will post two other scenarios on how to build your Golang application with GoCD soon

 

Posted in GoCD, Golang | 2 Comments

OpenStack migration from LDAP assignment backend to SQL backend

The migration is tricky and I don’t think OpenStack has any tool for this kind of migration.  Since this is only a one time thing, I just generate the SQL to replace all user_id and project_id in all tables ( nova, glance, cinder, neutron ).  Neutron is still reference the tenant model , so it has the tenant_id column in the database instead of project_id.  After this migration, our OpenStack will be running Liberty with multi-domain support, yah!

Posted in Openstack | Tagged , | Leave a comment

OpenStack Multi-Domain with LDAP backend

If you are using LDAP for authentication with single domain and want to move to multi-domain, you have to be careful with the user id and project id in OpenStack database.

If “openstack user list” show this before you migrate to multi-domain

+——————————————+———————-+
| ID | Name |
+——————————————+———————-+
| mmouse | Mickey Mouse |
| dduck  |Donald Duck |
…..

“openstack user list” will show this after the migration

+——————————————+———————-+
| ID | Name |
+——————————————+———————-+
| 607fba330dce552bcc34f0f821dabfcddbdf49f2635a9d8d372b5f0cad8ec00b |Mickey Mouse |
| 1b8a590adab6b707cae2a43d64552a1d0b000fe05431fcfde1c48742f966a1c3 |Donald Duck |

….

Because OpenStack use “ID” in database to associate all resources like cinder volume,  glance image, vm,  you will have to update all database that reference, eg “mmouse” to “607fba330dce552bcc34f0f821dabfcddbdf49f2635a9d8d372b5f0cad8ec00b”.  I don’t know if there is any migration tools available.

 

 

Posted in Uncategorized | Tagged , , | Leave a comment

Set up GoCD Server and Agent with AWS OpsWorks

GoCD team is going to opensource their Chef cookbooks soon that used to manage their build infrastructure.  Before the cookbooks are available for public, I have taken few of them and create a private GitHub repository and used it to setup GoCD server and agent with AWS OpsWorks.

* This is a non-SSL GoCD Server setup.  For SSL (Apache fronted) configuration, it will be on my next post.

** I will update this post when GoCD team makes their cookbooks available for public

Before we create a new stack, we need to define two new security groups in order for Go agent to communicate with Go server through the internal network.

Screen Shot 2016-01-08 at 1.56.06 PM.pngScreen Shot 2016-01-08 at 1.58.43 PM.png

Screen Shot 2016-01-08 at 1.58.48 PM.png

Screen Shot 2016-01-09 at 9.24.57 AM.png

Screen Shot 2016-01-09 at 9.25.07 AM.png

After the two security groups are created, go to OpsWorks and create a new Chef 12 stack

Screen Shot 2016-01-08 at 2.00.45 PM.png

Screen Shot 2016-01-09 at 9.26.28 AM.png

After the stack is created, let’s create a new layer called “GoCD Server”

Screen Shot 2016-01-08 at 2.01.05 PM.png

In the “General Settings” of GoCD Server layer, we will provide few attributes in JSON that will be used by the GoCD cookbooks to configure the GoCD server

Screen Shot 2016-01-08 at 2.01.50 PM.png

Screen Shot 2016-01-08 at 11.43.52 PM.png

Then add three recipes, “yum-pgdg::default”, “yum-gocd::default” and “go-server-wrapper::default”

Screen Shot 2016-01-08 at 11.20.53 PM.png

Under “Security”, we will add three security groups to the GoCD Server layer, “AWS-OpsWorks-Default-Server”, “AWS-OpsWorks-GoCD-Server” and “AWS-OpsWorks-Web-Server”

Screen Shot 2016-01-08 at 11.22.31 PM.png

After GoCD Server Layer is ready, we can add a new instance called “gocd-server”Screen Shot 2016-01-08 at 8.32.25 PM.png

Wait for the new instance to be ready.

Screen Shot 2016-01-08 at 8.39.51 PM.png

Find the Public DNS of this GoCD Server instance

Screen Shot 2016-01-08 at 8.39.56 PM.png

Open your browser and go to that URL, you GoCD Server is ready.Screen Shot 2016-01-08 at 8.40.05 PM.png

Let’s enable Go agent auto registration on the Go Server.

Screen Shot 2016-01-08 at 10.31.04 PM.png

To setup GoCD agent, we will define a new layer called “GoCD Linux Dev Agent”

Screen Shot 2016-01-08 at 8.41.25 PM.png

Screen Shot 2016-01-08 at 8.41.30 PM.png

In the “General Settings” of GoCD Linux Dev Agent layer, we will provide few attributes in JSON that will be used by the GoCD cookbooks to configure the GoCD Linux Dev Agent.

Screen Shot 2016-01-08 at 8.42.40 PM.png

Screen Shot 2016-01-08 at 11.45.46 PM.png

Then add three recipes, “yum-repoforge::default”, “yum-gocd::default” and “go-agent::default”

Screen Shot 2016-01-08 at 8.43.32 PM.png

Apply the two Security Groups “AWS-OpsWorks-Default-Server” and “AWS-OpsWorks-GoCD-Agent” to this layer.

Screen Shot 2016-01-08 at 8.44.00 PM.png

Add a new instance in this layer and start the instance

Screen Shot 2016-01-08 at 8.44.24 PM.png

Screen Shot 2016-01-08 at 8.44.31 PM.png

When this instance is ready, you will see the agent automatically register on Go Server.

Screen Shot 2016-01-08 at 10.49.46 PM.png

Screen Shot 2016-01-08 at 10.49.52 PM.png

 

 

Posted in GoCD | Tagged , | 5 Comments

Create CentOS 7 image for Vagrant / VMware Fusion

  1. Generate SSH key pair without a passphrase to be used by Vagrant
    • eg.  ssh-keygen -t rsa -C “barrowkwan@yahoo.com” -f ~/.ssh/vagrant_rsa
    • Screen Shot 2015-04-28 at 11.34.10 AM
  2. Create a github repository and upload your public key
  3. Folk my Kickstart script from github and change line 60 in centos7.txt file to point to your raw SSH key on github.
    • Screen Shot 2015-04-28 at 11.34.57 AM
  4. Download CentOS 7 DVD ISO from CentOS site
  5. Create a new Virtual Machine in Fusion and choose “Install from disc or image”
    • Screen Shot 2015-04-28 at 11.41.39 AM
  6. Choose the ISO file you have download from CentOS site
    • Screen Shot 2015-04-28 at 11.42.32 AM
  7. Uncheck “Use Easy Install”
    • Screen Shot 2015-04-28 at 11.44.13 AM
  8. If you want to change the name of your Virtual Machine, click “Customize Settings”.
    • Screen Shot 2015-04-28 at 11.46.38 AM
  9. Remove “Printer Port”, “Sound Card” and “Camera”
  10. Disable “Share Bluetooth devices with Linux”
    • Screen Shot 2015-04-28 at 11.50.07 AM
  11. Under “Isolation”,  disable “Drag and Drop” and “Copy and Paste”
    • Screen Shot 2015-04-28 at 11.51.46 AM
  12. Under “Hard Disk”, uncheck “Split into multiple files” and change the disk size to 40G.  ( Make sure to click “Apply” after these changes )
    • Screen Shot 2015-04-28 at 11.53.15 AM
  13. Change the “Startup Disk” to “CD/DVD”, then click “Restart”
    • Screen Shot 2015-04-28 at 11.55.07 AM
  14. Press “Tab” in the “CentOS 7” screen and add “noverifyssl ks=https://raw.githubusercontent.com/barrowkwan/linux_kickstart_script/master/centos7.txt” at the end of “vmlinuz” line and hit “Enter”.  ( Make sure you change the URL to point to the github repos you forked ).
    • Screen Shot 2015-04-28 at 11.59.08 AM
  15. After the installation is done, the VM will shutdown.  In Settings menu, double click “CD/DVD (IDE)” and uncheck “Connect CD/DVD Drive”
    • Screen Shot 2015-04-28 at 1.16.52 PM
  16. In Settings menu, double click “Startup Disk” and change the device to “Hard Disk (SCSI)” and click “Restart”
    • Screen Shot 2015-04-28 at 1.07.11 PM
  17. After VM boot, login as root ( password is vagrant )
  18. The VM should be up-to-date but just in case, run “yum update”
    • Screen Shot 2015-04-28 at 1.12.24 PM
  19. Go to menu “Virtual Machine” and choose “Install VMware Tools”
    • Screen Shot 2015-04-28 at 1.13.31 PM
  20. Execute “mount /dev/cdrom /mnt” to mount the VMware Tools CD
    • Screen Shot 2015-04-28 at 1.15.28 PM
  21. Exceute “tar zxvf /mnt/VMwareTools-9.9.2-2496486.tar.gz -C /tmp”  ( Depends on the version of your Fusion, VMwareTools might have a different filename )
    • Screen Shot 2015-04-28 at 1.23.46 PM
  22. Change the execution bit of vmware installation script and run the script
    • chmod 755 /tmp/vmware-tools-distrib/vmware-install.pl
    • /tmp/vmware-tools-distrib/vmware-install.pl
    • Screen Shot 2015-04-28 at 1.25.24 PM
  23. Use default for all questions asked by the script except “Would you like to enable VMware automatic kernel modules”, type “yes” for this option.
    • Screen Shot 2015-04-28 at 1.28.02 PM
  24. After the install is done, you will need to run the vmware-config-tools.pl again to compile the HGFS kernel module.   Use default for all questions asked by the config script.
    • Screen Shot 2015-04-28 at 1.31.30 PM
  25. After the script is done, clean up the vm and shut it down.
    • rm -rf /tmp/* /var/log/wtmp /var/log/btmp
    • history -c
    • Screen Shot 2015-04-28 at 1.34.47 PM
  26. Open a terminal on your Mac and go to the directory where your VM is installed. ( eg /Users/barrowkwan/Documents/Virtual Machines.localized/CentOS 7 Vagrant 64-bit.vmwarevm )
  27. Execute the following two commands in order
    • /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -d Virtual\ Disk.vmdk
    • /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -k Virtual\ Disk.vmdk
    • Screen Shot 2015-04-28 at 1.38.37 PM
  28. create a file called “metadata.json” with the content below:
    • { “provider” : “vmware_fusion” }
    • Screen Shot 2015-04-28 at 1.41.01 PM
  29. Remove all vmware log and create vagrant image
    1. rm -rf vmware*.log
    2. tar zcvf ../centos7.box ./*
    3. Screen Shot 2015-04-28 at 1.45.24 PM
  30. Add the image to vagrant
    1. vagrant box add centos7-vagrant-x86_64 ../centos7.box
    2. Screen Shot 2015-04-28 at 1.47.59 PM
Posted in CentOS, Vagrant, VMware Fusion | Tagged , , , , | Leave a comment

Working with Vagrant+Berkshelf to include local cookbook

When we use Vagrant+Chef+Berkshelf, you can manage cookbook by adding 

cookbook ‘ntp’

in Berksfile.  However, local developed cookbook that is not in the ~/.berkshelf folder will be ignored.   You can specify full path of the cookbook in Berksfile

cookbook ‘ntp’, path: ‘/Users/barrowkwan/chef-work/cookbooks/ntp’

Fortunately, Berksfile is executed as Ruby, you can leverage the following code to include or all selected cookbooks

Dir.glob[‘/Users/barrowkwan/chef-work/cookbook/**’].each do |mycookbook|

  cookbok File.basename(mycookbook), path: path

end

 

 

Posted in Uncategorized | Tagged , , , | Leave a comment