Pharo

Deploying Pharo builds from Travis over ssh

Some time ago I wrote how you can Test Pharo projects with Travis; but what if you wanted to deploy the created image? That’s what this post is about… or rather one of the ways.

Travis offers native support for deploying to thirty or so cloud solutions, and there has been an experimental support for BinTray in SmalltalkCI, so if one of those solutions is sufficient for you, go with it.

We however required a bit more powerful solution with the option to do some additional processing of the builds on the server, namely bundling VMs for specific platforms and custom launchers (doing so on Travis is quite a lot of network traffic), blessing specific versions, etc. So we chose a custom solution of deploying the built images from Travis to our server over ssh.

TL;DR

The setup may seem complex at first, but once you are familiar with it, you can get a new project up and running in five minutes.

1
2
3
4
5
6
# generate a new ssh key
ssh-keygen -t rsa -b 4096 -C "travis deployment" -f travis_key
# install travis client 
sudo gem install travis
# encrypt your key (the key must be in the repository folder)
travis encrypt-file travis_key

Move travis_key out of the repo, and leave there just the travis_key.enc.

Append your travis_key.pub to the server’s ~/.ssh/authorized_keys. (Also don’t leave your public key in the repo.)

Add addons, before_install, and deploy sections to .travis.yml

1
2
3
4
5
6
7
8
9
10
11
12
addons:
  ssh_known_hosts: your-server.com
before_install:
  - openssl ... # whatever travis encrypt-file gave you ...
  - chmod 0400 travis_key
  - mv travis_key ~/.ssh/id_rsa
deploy:
  provider: script
  skip_cleanup: true
  script: scripts/deploy.sh

Copy the deploy script to scripts/deploy.sh and make it executable chmod +x scripts/deploy.sh.

Getting image out of Travis

To first thing we need to do is get the built image out of Travis to our ssh server.

For that we need to commit our private ssh key to the github repo so Travis can use it. Of course the repo is public and we don’t to share our access keys with everyone — for this reason Travis offers to encrypt your key so it is safe. Of course it is always better to create a separate account for the deployment that has an appropriately limited access to your server.

Preparing the ssh key

First you need to create a new ssh key; you can leave the passphrase empty, otherwise you will have extra work of encrypting not just the file, but also the passphrase.

generating a new ssh key
1
ssh-keygen -t rsa -b 4096 -C "travis deployment" -f travis_key

The command will create two files travis_key.pub (your public key) and travis_key (your private key). Append your public key to ~/.ssh/authorized_keys of the target server.

Installing travis client

Next you need to install the travis client. It will be needed for encrypting your key, but you might find it handy for restarting your builds, polling them etc.

1
2
3
4
5
6
7
"install travis system wide"
sudo gem install travis
"or"
"install just for your local user"
gem install --user-install travis

Since I recently reinstalled my machine, I was missing ruby-dev, so I had to install that too.

missing ruby dev envstackoverflow.com
1
sudo apt-get install ruby-dev

If you have any further issues, consult the travis client readme.

Encrypting your private key

Now move your key to the git repo and encrypt it.

1
travis encrypt-file travis_key

You will be given instructions to add a command to before_install section of your .travis.yml, so do that.

travis encrypt-file will kindly offer you to automatically add it to your .travis.yml. Doing so will however remove any comments from the file and break the formatting…

Don’t forget to move your private key out of the repo, you don’t want to publish that.

Configuring the ssh key

Couple more things are needed in your .travis.yml for a successful login.

.travis.yml excerpt
1
2
3
4
5
6
7
addons:
  ssh_known_hosts: your-server.com
before_install:
  - openssl ... -out travis_key -d
  - chmod 0400 travis_key
  - mv travis_key ~/.ssh/id_rsa
  • ssh_known_hosts is the name of your server
  • chmod 0400 will set the proper permissions; if they are too loose, Travis will reject your key
  • ~/.ssh/id_rsa is a default location for a key; that way you don’t need to specify the key in your ssh/scp commands

If you want to keep your .travis.yml short(er), you can always move those commands to a separate bash file and execute that instead.

Deployment itself

Now we can finally get to the deployment itself. To specify the target script, add the following to the .travis.yml

1
2
3
4
deploy:
  provider: script
	skip_cleanup: true
  script: scripts/deploy.sh

And now create scripts/deploy.sh and write whatever you want to happen. You should be aware of at least some of the environmental variables provided by Travis and SmalltalkCI:

Travis (see the full list):

  • TRAVIS_BUILD_NUMBER - the number of the currently executed build
  • TRAVIS_BRANCH - the branch travis is building
  • TRAVIS=true - handy if you want to test your scripts locally

SmalltalkCI:

  • SMALLTALK_CI_HOME - folder containing the SmalltalkCI builder
  • SMALLTALK_CI_BUILD - folder containing the built image
  • SMALLTALK_CI_IMAGE - path to the built .image
  • SMALLTALK_CI_CHANGES - path to the .changes of the built image
  • SMALLTALK_CI_VMS - path to the folder containing VMs
  • SMALLTALK_CI_VM - path to the Pharo VM This variable is broken and there’s a pending bug

scripts/deploy.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/bin/bash
# exit on first encountered error
set -o errexit
# wherever you'll be ssh-ing into user@machine
readonly TARGET_MACHINE="username@example.com"
# customize the name of the Pharo image you will be deploying
readonly PROJECT_NAME="my-project"
# customize the name of the build folder
readonly ARTIFACT_DIR="${PROJECT_NAME}-${TRAVIS_BUILD_NUMBER}"
# Rename the image, zip it, and send it to a server
deploy-simple() {
	mkdir "$ARTIFACT_DIR"
	cp "$SMALLTALK_CI_IMAGE" "${ARTIFACT_DIR}/${PROJECT_NAME}.image"
	cp "$SMALLTALK_CI_CHANGES" "${ARTIFACT_DIR}/${PROJECT_NAME}.changes"
	local build_zip="${ARTIFACT_DIR}.zip"
	zip -qr "$build_zip" "$ARTIFACT_DIR"
	scp -rp "$build_zip" "$TARGET_MACHINE:~/builds/"
	# I have a server-side post-processing script that bundles VMs into the build
	#ssh "$TARGET_MACHINE" "~/post-process.sh ${TRAVIS_BUILD_NUMBER}"
}
main() {
	deploy-simple
	echo "Build ${ARTIFACT_DIR} deployed."
}
if [[ "$TRAVIS_BRANCH" == "master" ]]; then
	main
fi

Final thoughts

If you’ve done everything correctly, you should have a new my-project-NUMBER.zip file in your builds folder every time the master branch is successfully built.

There is one more thing I would like to address in one of the next posts — bundling the images with VMs for specific platforms and custom launchers… but that’s next time.