Steve Bennett blogs

…about maps, open data, Git, and other tech.

Getting started with Chef on the NeCTAR Research Cloud


Opscode Chef is a powerful tool for automating the configuration of new servers, and indeed, entire clouds, multi-server architectures etc. We now use it to deploy MyTardis. It’s quite daunting at first, so here’s a quick guide to the setup I use. You’ll probably need to refer to more complete tutorials to cover gaps (eg  OpsCode’s Fast Start Guide, the MyTardis chef documentation and this random other one.) We’ll use installing MyTardis as the goal.

  1. Sign up to Hosted Chef. That gives you a place where Chef cookbooks, environments, roles etc will live.
  2. Install Chef (client) locally.
  3. Get the MyTardis Chef cookbook:
    cd ~/mytardis
    git clone https://github.com/mytardis/mytardis-chef
  4. Make a directory, say ~/chef, and download the “…-validator.pem” and knife.rb files that Opscode gives you to there. The knife.rb file contains settings for Knife, like directories. Modify it so it points to ~/mytardis/mytardis-chef. Mine looks like this
    current_dir = File.dirname(__FILE__)
    log_level :info
    log_location STDOUT
    node_name "stevage"
    client_key "#{current_dir}/stevage.pem"
    validation_client_name "versi-validator"
    validation_key "#{current_dir}/versi-validator.pem"
    chef_server_url "https://api.opscode.com/organizations/versi"
    cache_type 'BasicFile'
    cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
    cookbook_path ["#{current_dir}/../mytardis/mytardis-chef/site-cookbooks", "#{current_dir}/../mytardis/mytardis-chef/cookbooks"]
  5. You now need to upload these cookbooks and assorted stuff to Chef Server. This part of Chef is really dumb. (Why isn’t there a single command to do this – or why can’t knife automatically synchronise either a local directory tree or git repo with the Chef server. Why is one command ‘upload’ and another ‘from file’…? Why are cookbooks looked for in the ‘cookbooks path’ but roles are sought under ./roles/?)
    knife cookbook upload -a
    knife role from file ../mytardis/mytardis-chef/roles/mytardis.json
  6. Get a NeCTAR Research Cloud VM. Launch an instance of “Ubuntu 12.04 LTS (Precise) amd64 UEC”. Call it “mytardisdemo”, give it your NeCTAR public key, and do what you have to (security groups…) to open port 80.
  7. Download your NeCTAR private key to your ~/chef directory.
  8. Now, to make your life a bit easier, here are three scripts that I’ve made to simplify working with Knife:bootstrap.sh:
    #!/bin/bash -x
    # Bootstraps Chef on the remote host, then runs its runlist.
    source ./settings$1.sh
    knife bootstrap $CHEF_IP -x $CHEF_ACCOUNT --no-host-key-verify -i nectar.pem --sudo $CHEF_BOOTSTRAP_ARGS

    update.sh:

    #!/bin/bash -x
    # Runs Chef-client on the remote host, to run the latest version of any applicable cookbooks.
    source settingsi$1.sh
    knife ssh name:$NECTAR_HOST -a ipaddress -x $NECTAR_ACCOUNT -i nectar.pem -o "VerifyHostKeyDNS no" "sudo chef-client"

    connect.sh:

    #!/bin/bash -x
    # Opens an SSH session to the remote host.
    source ./settings$1.sh
    ssh $CHEF_ACCOUNT@$CHEF_IP -i nectar.pem -o "VerifyHostKeyDNS no"

    (We disable host key checking because otherwise SSH baulks every time you allocate a new VM with an IP that it’s seen before.)With these, you can have several “settings…sh” files, each one describing a different remote host. (Or just a single settings.sh). So, create a “settings-mytardisdemo.sh” file, like this:

    # Settings for Ubuntu Precise NeCTAR VM running mytardis as a demo
    export CHEF_HOST=mytardisdemo
    export CHEF_IP=115.146.94.53
    export CHEF_ACCOUNT=ubuntu
    export CHEF_BOOTSTRAP_ARGS="-d ubuntu12.04-gems -r 'role[mytardis]'"
  9. Now you have it all in place: A chef server with the cookbook you’re going to run, a server to run it on, and the local tools to do it with. So, bootstrap your instance:
    ./bootstrap.sh -mytardisdemo

    If all goes well (it won’t), that will churn away for 40 minutes or so, then MyTardis will be up and running.

  10. If you change a cookbook or something (don’t forget to re-upload) and want to run chef again:
    ./update.sh -mytardisdemo
  11. And to SSH into your box:
    ./connect.sh -mytardisdemo
About these ads

One response to “Getting started with Chef on the NeCTAR Research Cloud

  1. Pingback: A pattern for multi-instrument data harvesting with MyTardis « Steve Bennett blogs

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 431 other followers

%d bloggers like this: