…about maps, open data, Git, and other tech.
Opscode Chef is a powerful tool for automating the configuration of new servers, and indeed, entire clouds, multi-server architectures etc. We now use it to deploy MyTardis. It’s quite daunting at first, so here’s a quick guide to the setup I use. You’ll probably need to refer to more complete tutorials to cover gaps (eg OpsCode’s Fast Start Guide, the MyTardis chef documentation and this random other one.) We’ll use installing MyTardis as the goal.
cd ~/mytardis git clone https://github.com/mytardis/mytardis-chef
current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "stevage"
client_key "#{current_dir}/stevage.pem"
validation_client_name "versi-validator"
validation_key "#{current_dir}/versi-validator.pem"
chef_server_url "https://api.opscode.com/organizations/versi"
cache_type 'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path ["#{current_dir}/../mytardis/mytardis-chef/site-cookbooks", "#{current_dir}/../mytardis/mytardis-chef/cookbooks"]
knife cookbook upload -a knife role from file ../mytardis/mytardis-chef/roles/mytardis.json
#!/bin/bash -x # Bootstraps Chef on the remote host, then runs its runlist. source ./settings$1.sh knife bootstrap $CHEF_IP -x $CHEF_ACCOUNT --no-host-key-verify -i nectar.pem --sudo $CHEF_BOOTSTRAP_ARGS
update.sh:
#!/bin/bash -x # Runs Chef-client on the remote host, to run the latest version of any applicable cookbooks. source settingsi$1.sh knife ssh name:$NECTAR_HOST -a ipaddress -x $NECTAR_ACCOUNT -i nectar.pem -o "VerifyHostKeyDNS no" "sudo chef-client"
connect.sh:
#!/bin/bash -x # Opens an SSH session to the remote host. source ./settings$1.sh ssh $CHEF_ACCOUNT@$CHEF_IP -i nectar.pem -o "VerifyHostKeyDNS no"
(We disable host key checking because otherwise SSH baulks every time you allocate a new VM with an IP that it’s seen before.)With these, you can have several “settings…sh” files, each one describing a different remote host. (Or just a single settings.sh). So, create a “settings-mytardisdemo.sh” file, like this:
# Settings for Ubuntu Precise NeCTAR VM running mytardis as a demo export CHEF_HOST=mytardisdemo export CHEF_IP=115.146.94.53 export CHEF_ACCOUNT=ubuntu export CHEF_BOOTSTRAP_ARGS="-d ubuntu12.04-gems -r 'role[mytardis]'"
./bootstrap.sh -mytardisdemo
If all goes well (it won’t), that will churn away for 40 minutes or so, then MyTardis will be up and running.
./update.sh -mytardisdemo
./connect.sh -mytardisdemo
OData’s shiny logo.
Ok, now that I have your attention, let me start backpedalling. In the Australian e-Research sector, I’ve worked on problems of research data management (getting researchers’ data into some kind of database) and registering (recording the existence of that data somewhere else, like an institutional store, or Research Data Australia). There hasn’t been a huge amount of discussion about the mechanics of reusing data: the discussion has mostly been “help researcher A find researcher B’s data, then they’ll talk”.
Now, I think the mechanics of how that data exchange is going to take place will become more important. We’re seeing the need for data exchange between instances of MyTardis (eg, from a facility to a researcher’s home institution) and we will almost certainly see demand for exchange between heterogeneous systems (eg, MyTardis to/from XNAT). And as soon as data being published becomes the norm, consumers of that data will expect it to be available in easy to use formats, with standard protocols. They will expect to be able to easily get data from one system into another, particularly once those two systems are hosted in the cloud.
Until this week, I didn’t know of a solution to these problems. There is the Semantic Web aka RDF aka Linked Data, but the barrier to entry is high. I know: I tried and failed. Then I heard about OData from Conal Tuohy.
You have OData producers (or “services”) and consumers. A consumer could be: a desktop app that lets a user browse data from any OData site; a web application that mashes up data between several producers; another research data system that harvests its users’ data from other systems; a data aggregator that mirrors and archives data from a wide range of sites. Current examples of desktop apps include a plugin to Excel, a desktop data browser called LINQPad, a SilverLight (shudder) data explorer, high-powered business analytics tools for millionaires, a 3.5 star script to import into SQL Server…
Perhaps you already have a set of webservices with names like “ListExperiments”, “GetExperimentById”, “GetAuthorByNLAID”. You’ve documented your API, and everyone loves it. Here’s why OData might be a smarter choice, either now, or for next time:
To turn an existing data management system (say, PARADISEC’s catalogue of endangered language field recordings) into an OData producer, you bolt an Atom endpoint onto the side. Much like many systems have already done for RIF-CS, but instead of using the rather boutique, library-centric standards OAI-PMH and RIF-CS to describe collections, you use AtomPub and HTTP to provide individual data items.
Specifically, you need to do this:
There are libraries for a number of technologies, but if your technology doesn’t have an OData library, it probably has libraries for components like Atom, HTTP, JSON, and so on.
As an example, Tim Dettrick (UQ) and I have built an Atom dataset provider for MyTardis – before knowing about OData. To make it OData compliant probably requires:
Linked Data aka RDF aka Semantic Web is another approach that has been around for many years. Both use HTTP with content negotiation, an entity-attribute-value data model (ie, equivalent to XML or JSON), stable URIs and serialise primarily in an XML format. Key differences (in my understanding):
It is highly implausible that Google will be beaten by Microsoft in such an important area as data sharing. And yet Google Data Protocol (aka GData) is not yet a contender: it is controlled entirely by Google, and limited in scope to accessing services provided by Google or by the Google App engine. A cursory glance suggests that it is in fact very similar to OData anyway: AtomPub, REST, JSON, batch processing (different syntax), a filterying/query syntax. I don’t see support for a data model discovery, however: there’s no $metadata.
Git is the source code version control system that is rapidly becoming the standard for open source projects. It has a powerful distributed model which allows advanced users to do tricky things with branches, and rewriting history. What a pity that it’s so hard to learn, has such an unpleasant command line interface, and treats its users with such utter contempt.
The information model is complicated – and you need to know all of it. As a point of reference, consider Subversion: you have files, a working directory, a repository, versions, branches, and tags. That’s pretty much everything you need to know. In fact, branches are tags, and files you already know about, so you really need to learn three new things. Versions are linear, with the odd merge. Now Git: you have files, a working tree, an index, a local repository, a remote repository, remotes (pointers to remote repositories), commits, treeishes (pointers to commits), branches, a stash… and you need to know all of it.
The command line syntax is completely arbitrary and inconsistent. Some “shortcuts” are graced with top level commands: “git pull” is exactly equivalent to “git fetch” followed by “git merge”. But the shortcut for “git branch” combined with “git checkout”? “git checkout -b”. Specifying filenames completely changes the semantics of some commands (“git commit” ignores local, unstaged changes in foo.txt; “git commit foo.txt” doesn’t). The various options of “git reset” do completely different things.
The most spectacular example of this is the command “git am”, which as far as I can tell, is something Linus hacked up and forced into the main codebase to solve a problem he was having one night. It combines email reading with patch applying, and thus uses a different patch syntax (specifically, one with email headers at the top).
The man pages are one almighty “fuck you”. They describe the commands from the perspective of a computer scientist, not a user. Case in point:
git-push – Update remote refs along with associated objects
Here’s a description for humans: git-push – Upload changes from your local repository into a remote repository
Update, another example: (thanks cgd)
git-rebase – Forward-port local commits to the updated upstream head
Translation: git-rebase – Sequentially regenerate a series of commits so they can be applied directly to the head node
Remember the complicated information model in step 1? It keeps growing, like a cancer. Keep using Git, and more concepts will occasionally drop out of the sky: refs, tags, the reflog, fast-forward commits, detached head state (!), remote branches, tracking, namespaces
Most of the power of Git is aimed squarely at maintainers of codebases: people who have to merge contributions from a wide number of different sources, or who have to ensure a number of parallel development efforts result in a single, coherent, stable release. This is good. But the majority of Git users are not in this situation: they simply write code, often on a single branch for months at a time. Git is a 4 handle, dual boiler espresso machine – when all they need is instant.
Interestingly, I don’t think this trade-off is inherent in Git’s design. It’s simply the result of ignoring the needs of normal users, and confusing architecture with interface. “Git is good” is true if speaking of architecture – but false of user interface. Someone could quite conceivably write an improved interface (easygit is a start) that hides unhelpful complexity such as the index and the local repository.
The fundamental promise of any version control system is this: “Once you put your precious source code in here, it’s safe. You can make any changes you like, and you can always get it back”. Git breaks this promise. Several ways a committer can irrevocably destroy the contents of a repository:
In the traditional open source project, only one person had to deal with the complexities of branches and merges: the maintainer. Everyone else only had to update, commit, update, commit, update, commit… Git dumps the burden of understanding complex version control on everyone – while making the maintainer’s job easier. Why would you do this to new contributors – those with nothing invested in the project, and every incentive to throw their hands up and leave?
The primary output of development work should be source code. Is a well-maintained history really such an important by-product? Most of the arguments for rebase, in particular, rely on aesthetic judgments about “messy merges” in the history, or “unreadable logs”. So rebase encourages you to lie in order to provide other developers with a “clean”, “uncluttered” history. Surely the correct solution is a better log output that can filter out these unwanted merges.
If your changes involve creating new files, there’s a tricky extra step:
For a Github-hosted project, the following is basically the bare minimum:
In reality though, the maintainer of that Github-hosted project will probably prefer your changes to be on feature branches. They’ll ask you to work like this:
As an added bonus, here’s a diagram illustrating the commands a typical developer on a traditional Subversion project needed to know about to get their work done. This is the bread and butter of VCS: checking out a repository, committing changes, and getting updates.
“Bread and butter” commands and concepts needed to work with a remote Subversion repository.
And now here’s what you need to deal with for a typical Github-hosted project:
The “bread and butter” commands and concepts needed to work with a Github-hosted project.
If the power of Git is sophisticated branching and merging, then its weakness is the complexity of simple tasks.
This post has obviously struck a nerve, and gets a lot of traffic. Thought I’d address some of the most frequent comments.
A few bonus command inconsistencies:
Reset/checkout
To reset one file in your working directory to its committed state:
git checkout file.txt
To reset every file in your working directory to its committed state:
git reset --hard
Remotes and branches
git checkout remotename/branchname git pull remotename branchname
There’s another command where the separator is remotename:branchname, but I don’t recall right now.
Command options that are practically mandatory
And finally, a list of commands I’ve noticed which are almost useless without additional options.
| Base command | Useless functionality | Useful command | Useful functionality |
| git branch foo | Creates a branch but does nothing with it | git checkout -b foo | Creates branch and switches to it |
| git remote | Shows names of remotes | git remote -v | Shows names and URLs of remotes |
| git stash | Stores modifications to tracked files, then rolls them back | git stash -u | Also does the same to untracked files |
| git branch | Lists names of local branches | git branch -rv | Lists local and remote tracking branches; shows latest commit message |
| git rebase | Destroy history blindfolded | git rebase -i | Lets you rewrite the upstream history of a branch, choosing which commits to keep, squash, or ditch. |
| git reset foo | Unstages files | git reset –hard git reset –soft |
Discards local modifications Returns to another commit, but doesn’t touch working directory. |
| git add | Nothing – prints warning | git add . git add -A |
Stages all local modifications/additions Stages all local modifications/additions/deletions |
A few interesting links:
I seem to have picked up the habit of using semantic keywords in my Google searches. I have no idea if Google actually supports any of them, but, if they don’t, they should!
Consider an action like creating a new “space” in Confluence. Because of the visibility of this action, people want it restricted. Controlled. Managed. So Atlassian makes it only available to people with administrator rights. Which seems ok, until you realise that the workflow of the non-administrator ends up looking like this:
For the non-administrator:
Since creating a space has very limited potential for destruction if it’s hidden, how about this:
For the administrator:
Most users aren’t destructive. And especially in professional environments, virtually all users can be trusted to act in good faith. (Managers strangely predict “chaos” if many users are authorised). So, let them do it:
So you’ve found the thing you want, and you already have both an eBay account and a PayPal account. You don’t want to do anything complicated – send this item to me, and charge my (already stored) card appropriately.
Here’s what I hate about recipes:
I can’t fix all of that. But here goes:
A cauliflower dish I made up the other night, seen by an engineer.
Here’s what a conventional version of that dish might look like:
Ingredients:
1/2 cauliflower, chopped
1 bok choi
1 tbsp coriander, chopped
1/2 cup slivered almonds
1 tbsp ground turmeric
1 tbsp mustard seeds
1-4 tsp chilli powder, to taste
Put 2cm of water in a saucepan and bring it to the boil. Simmer the cauliflower 2 minutes, then drain, discarding the water. Add half the sesame oil, and return the cauliflower to the saucepan.
Meanwhile, heat the rest of the sesame oil in a frying pan. When hot, add the slivered almonds and remove from the heat, stirring continuously. Once browned, add the almonds to the frying cauliflower, and add the spices. After a few minutes, add the bok choi. Stir frequently until the bok choi is soft, then serve with rice and roti bread.
Here are the rules:
Explaining step 7: the diagram above could have shown a colander as another column, with cauliflower transferred from the saucepan to the colander and back. But why would you do that?
Normal recipe format works pretty well for most people, but I find it takes multiple readings before I can start. Effectively, I’m constructing this kind of flow chart in my head, working out what gets transferred from where, to where. I hope that a refined version of this diagramming methodology would allow you to confidently dive straight in, with no nasty surprises.
On the downside, diagrams are hard to manage. I realise I forgot an ingredient (crushed garlic and ginger paste), but it’s time-consuming to modify the diagram and re-upload.
And lastly, yes, it’s a bit facile to call this “cooking for engineers”. Probably real engineers want precise measurements, correct use of flow control symbols and so forth.
Ok, Gmail, you’re halfway there: you’ve got labels, and you can detect mailing lists (Google Groups ones, anyway). Now, take it a bit further:
For extra credit:
4. Detect other sources of regular mail which are not “mailing lists” as such. Newsletters from the bank. Quarterly updates from my alma mater. Treat them exactly the same.
Most of the features are already there, but it’s so tedious having to set up a special rule and label for every single list.
swoopo, bidray, bidstick (bids tick, apparently), bidrivals and dozens of others are running what we’ll call “penny auctions“. Using bidrivals.com as the example, they all work on the following principals:
At first glance, the auction looks great – buy a phone for $20! Buy a plasma tv for $1.53!
But not so fast. A couple of things that are not obvious to the beginner:
So, is it a scam? Well, there are really two quesions:
I see very little to distinguish these penny auctions from gambling:
There is quite a bit of antipathy towards these sites: Washington Post, Jeff Atwood, Ed Oswald.
There are two main risks:
Fortunately, there are sites on the look out for this kind of thing, such as pennyauctionwatch.com. There is evidence of dodgy sites, such as fake testimonials.
So, what are the incentives for a site to use shill bidding? Well, as we saw above, the difference between a $1000 item selling for $1 and $25 doesn’t look like much, but it’s the difference between breaking even and posting a big loss. Imagine there is fairly steady bidding activity, but there are just a few gaps before that $25 mark. If the site could shill just a few times, they would massively increase their profitability.
But how much should they shill? Consider two strategies:
In 1), the shill bids guarantee almost any asking price, as long as there is still some demand. This has the potential to greatly increase profit, and decrease variance.
In 2), half the bids end up being shill bids. This causes two problems: first, you’re directly losing one bid fee for every shill bid. Second, by inflating the price, you’re accelerating reaching the point at which people no longer want to bid, because the prize at stake is shrinking. So if people might normally bid strongly up to half the value of the item, then shilling along the way is just replacing paying bids with free ones. You might even decrease the final sale value, and every dollar of sale value lost is $40 of bidding fees lost.
Conclusion: shill bidding seems likely to occur, in small doses, because the incentive is just so strong.
Can you beat them?
Probably not. It’s been tried. To beat it:
If it is beatable, you’d think people would have done it. And, since they’re capped at 4 wins per month generally, there would not be much harm in them sharing their secret. Unless they have a network of penny auction-beating bots. There’s a thought.
But just in case you wanted to try:
Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!
Steve Bennett blogs 
Recent Comments