Example Pipeline Overview

  • There's a spreadsheet describing the Cadogan GoCD pipelines in detail here (only available to Clare).
  • Generally pipelines are initially copying stuff into S3 artefacts buckets, and then copying from there into final buckets, which are pointed at either by Cloudfront distributions or EBS environments.

    • The pipelines which deploy to artefacts buckets are automatically triggered
    • The pipelines which copy from artefacts into main buckets are mostly manual (apart from dev)
    • QA, Dev and Showcase will be triggered by / pull stuff from master branch.
    • Preproduction and Production will be triggered by / pull stuff from release branch.
  • Branches:

    • Typically we only have 2 branches: master and release

      • Master is used for dev, qa and showcase
      • Release is used for pre-prod and prod
      • Regression tests are done on pre-prod
    • Backend only has 3 envs: dev, preprod and prod

    • Frontend always has 5 envs: dev, qa, showcase, preprod, prod

    • Pipelines are grouped into pipeline groups

    • Jobs always run in parallel

      • They are the most granular level
    • See pipeline | settings | automatic pipeline scheduling to tell whether a pipeline is automatic or not

      • Also see Settings | Materials | any git repos | “Poll for new changes”

Showing / Hiding Pipelines

  • To see hidden pipelines, use the Personalise menu in Go (top right)
  • It’s a good idea to hide old pipelines so you don’t use them by accident – but even better is to delete them!

Adding New Go Users / Removing Users

  • SSh into the Go server
    • See SSH Access to Go Server, below
  • Open the user file:
    • First you have to open a new shell with admin privileges: sudo su
    • Then go to the right folder: cd /var/go
    • Now open the user file: vi user_file
  • Now, in a separate Terminal tab (Cmd + T) on your main Mac, create an SHA for your password (replace “password” with the actual password):
    • Like this: slappasswd -h {SHA} -s password
  • Now add your user name and password SHA to the user file
    • Press i to get into insert mode
    • Then add your name, a colon and your SHA, like this: clare:{SHA}3vKr3vKr3vKr3vKr3vKr3vKr3vK=
      • (that’s not a real sha)
    • To paste from system clipboard:
      • make sure you used vi rather than VIM
      • Use Cmd + Shift + V
  • Finally, to save the user file: First hit esc, then type :wq and Enter
  • Once you’ve added the user, you need to grant them roles:

SSH Access to Go Server

  • To ssh into the server:
  • Get yourself into the folder where your pem file lives
  • Terminal: ssh -i "go-server.pem" ec2-user@54.171.48.173

Security Access to Go

Go Servers and Agents

  • AWS (EC2) contains one server which also acts as an agent, plus one other agent
  • Micro instance of Go on EC2
    • Commands on go agent when ssh-ed in:
      • /var/log/go
      • sudo service g-server status
      • /go/go-agent/^c
      • sudo su –l go
    • Go server was manually installed on this server
    • 2 agents there for every job – backend, fronted, mocks
    • Getting a Go agent up on ec2:
      • create ec2 instance
      • install go agent
      • configure GO_SERVER url
      • start go agent
      • ssh keys generate
      • add it to aws new user
      • copy access keys
      • install git
      • change ssh config permissions, chmod 600 ~/.ssh/config
      • install jdk, yum install java-1.8.0-openjdk-devel
      • npm install -g npm
      • install and configure aws cli
      • install docker (yum install)
      • give go user permissions of docker or add to the user group (sudo usermod -a -G docker go)
    • In EC2, Go Agent 1 and Go Server
      • You might want to ssh in if you want to check resources, or if git repo unable to clone
      • Typical use case is to view server logs
        • To see logs:
        • ls –la /go/go-agent/log
        • ls –la /go/go-server/artifacts
        • And you might want to look at previous artefacts, or manually remove files
        • ls –la /storage/
        • To ssh out again, type logout
        • Less /etc/default and /etc/default/go-agent
          • To see port settings etc
          • Go_server_port=8153
      • Go Agent 1 doesn’t have go-server, only has go-agent

Docker image for Go Agent

  • Go-cd have always published their go server and agent images
  • This is the docker image that one of our colleagues created to make go agent: Infra/dockerfile
    • Get gocd image
    • Update
    • Install git
    • Get jdk
    • Get bzip2
    • Switch user
    • Install nvm
    • Source nvm and run nvm install
    • Change to root
    • Run docker
    • Probably also have to add docker, mocks + dependencies, galen
    • We would also need ssh keys and credentials – don’t mention them in the containers
    • Sometimes docker takes time for daemon to load – be patient
  • Basic image is always ubuntu
  • Get package manager there
  • Command to run up image is this: docker run -v ~/.ssh/:/home/go/.ssh/ -d -e GO_SERVER_URL="https://54.171.48.173:8154/go" --name='gocd-agent' gocd-agent
  • One of our colleagues created a docker image which could be used to create the Go agent:
    • Go to dockerhub:
    • amiedeep/gocd-agent
    • docker run -v ~/.ssh/:/home/go/.ssh/ -d -e GO_SERVER_URL="https://54.171.48.173:8154/go" --name='gocd-agent' gocd-agent

Setting up the Local Go Agent

  • Our notes from when we did this:
  • You need nodejs set up - I used homebrew to install this.
    • Troubleshooting: The previous team used node version 6.11.3 so if you run into trouble with the version you have installed use a tool like nvm (https://github.com/creationix/nvm) to switch versions.
    • N.B. This may assume that your nodejs installation is managed by brew and not independently installed.
  • You should have node installed and ensure that /usr/local/bin is in your $PATH - this can be done by editing your `~/.bash_profile` and adding: export PATH="/usr/local/bin:$PATH"
  • Ensure you have the right components installed globally by running: npm install -g bower ember-cli galen
  • Download go-agent from web site, install into the Applications folder as per a standard mac app.
    • The go-agent server config should go to https on port 8154 (not 8153, that’s http)
    • Configure this by running the go-agent app and heading to Go Agent > Preferences
    • https://54.171.48.173:8154/go
    • [option SSL Mode: No Verification should be selected]
  • Buildreactor dashboard is used in chrome for the build display - uses the (non-admin) user.  A new user can be setup with a new password and non-admin permissions to replace this.
  • Build assets typically end up at a path like:

Deployment environments

  • Gulpfile.js references deployEnv:
    • See GoCD for env var which then finds its way into gulpfile.js
    • Build_and_publish.sh (which is in infra) also takes the var from Go pipeline.

Configuring jobs to run on particular Go Agents

  • Note: It's possible to set up individual pipeline jobs to run on specific Go agents.
  • You do this using resources, which are configured under Job Settings in pipelines and also under Resources on each agent.
  • Full documentation here (under “Matching jobs to agents”): https://docs.gocd.org/current/configuration/managing_a_build_cloud.html
  • For an example in Cadogan:
    • Select Agents in Go (top navbar)
    • Select the checkbox next to an agent
    • Select the Resources dropdown, top right
    • Now go into the settings for the product pipeline
    • Select the Deploy job, under the deploy_showcase stage (over on the left)
    • Select job Settings
    • You can see the resource(s) specified under Resources. Each of these resources can then also be attached to one or more Go agents (see above).
  • Click into the settings (cog icon) for the Backend-api pipeline.
  • Select the top level on the left
  • Select the Materials tab
  • Click on git
  • You'll see destination directory is backend folder
  • To find out which Go agent is used for this pipeline:
    • Drill down into an individual job on the left (eg Backend_api, Build_Test, compile_test)
    • Select Job Settings, check Resources.
    • Now go to Agents (top menu) and you'll see which agents handle jobs with those resources.
    • In this case the resource is backend, and one of the agents handling backend is ip-172-31-23-137.
  • This is the Go server. In the AWS management console under EC2 instances, you can see it has that private IP address
  • If you ssh into the Go server, you can look for the destination directory referred to above. This will give you a bit of a feel for what happens on a Go agent during a deployment.
  • Note that you need the sudo su command to open up a shell with the correct privileges

Renaming / Top tip for checking connections between things in Go

  • Select Admin | Config XML
  • Search for a string (eg the name of a resource)
  • You'll find all the elements of Go config that reference that string.
  • This is particularly useful if you want to rename something: You can use this technique to check whether it’s being referred to anywhere else.