Recent Comments
 | arnebab on 10 things I hate about Gi… |
| arnebab on 10 things I hate about Gi… |
| arnebab on 10 things I hate about Gi… |
 | Smitty on 10 things I hate about Gi… |
 | Igor on 10 things I hate about Gi… |
Steve Bennett blogs
…about maps, open data, Git, and other tech.
One week of Salt: frustrations and reflections.
Posted by on February 17, 2014
Reflecting on Salt. (Credit: Psyberartist, CC-BY 2.0)
Salt, or SaltStack, is an automatic deployment tool for clusters of machines, in the same category as Chef, Puppet and Ansible. I’ve previously used Chef, found the experience frustrating, and thought I’d give Salt a go. The task: converting my tilemill-server bash scripts, which set up a complete TileMill server stack, with OpenStreetMap data, Nginx and a couple of other goodies.
The goals in this kind of automation for me are threefold:
- So I can quickly build new servers for Mapmaking for Academics workshops.
- So I can manage other, similar (but slightly different) servers, such as cycletour.org and gis.researchmaps.net. “Manage” here means documenting and controlling the slight configuration differences, being able to roll out further changes (such as web passwords) without having to log in.
- So other people can do this too. (That is, the solution has to be simple enough to explain that other people can do it).
The product of my labours: https://github.com/stevage/saltymill
Disclaimers and apologies
A week is too soon to feel the real benefits of automatic deployment and configuration management, but plenty long enough for frustrations and annoyances. There are lots of good things about Salt, but those will go in another post.
I’ve intentionally written this post as I go, to document things that are surprising, counter-intuitive or bothersome to the newcomer. Once you become familiar with a system, it’s much harder to remember what it was like as a newbie, and easier to make allowances for it. So experienced Salt users will probably rankle at the wilful ignorance displayed herein.
A few things I quickly appreciated, compared to Chef:
- Two parties, not three. Chef has the client (your laptop), the Chef Server (which I never used because it was too complicated) and the target machine itself. Salt doesn’t have the client, so you do everything either on the master (“salt ‘*’ state.highstate) or on the target machine in masterless mode (“salt-call –local state.highstate)
- No need for “knife upload”: the Salt Master just looks for your files in /srv/salt, so you can use any method you like for getting them there. I always hated having to upload recipes, environments etc individually – you always forgot something.
- YAML rather than Ruby.
- A nifty command line tool that lets you try states out on the fly, or run commands remotely on several servers at once. (Chef’s shell, formerly ‘shef’, never really worked out for me.)
Mediocre documentation
Automatic deployment systems are complicated, and you’d love some clear mental concepts to hang on to. Unfortunately, Salt’s chosen metaphor is pretty clumsy (salt grains, pillars, mines…meh), and they’re not well explained. Apparently the concept of a “state” is pretty important to Salt, but even after reading much documentation and quite a reasonable amount of hacking away, I still have only the vaguest notion of what a Salt state is. What’s a “high state”? A “low state”? How can I query the “state” of a minion? No idea. It’s hard to tell if “states” are an implementation magic that makes the whole thing work, or if I, the user am supposed to know about them.
(After more digging around, I think “state” has several meanings, of which one is just one of the actions defined in a SLS file.)
And then, strangely, how do you refer to the most important objects that you work with, the actual code that specifies what you want done? “SLS files”. Or “state files”. Or “SLS[es]”. Or “Salt States”. Or “Salt state files”. Those terms are all used in the docs. (I thought they were also called “formulas“, but those are apparently only pre-canned state files.)
YAML headaches
It turns out YAML has some weird indenting and formatting rules that occasionally ruin your day. Initially it’s especially tedious having to learn this extra language and its subtleties (> vs |, multi-line hyphenated lists versus single line square-bracketed lists) when you just want to get started – it steepens the learning curve.
IDs, names
The behaviour of statement IDs is cute at first, but not well explained, and becomes pretty tiresome. The idea is you can write this:
httpd: pkg.installed
Succinct, huh? That’s actually shorthand for this:
install_apache_please:
pkg:
- name: httpd
- installed
That is, the “id” (httpd) is also the default value for “name”. Secondly, you can compress one item of the following list (“installed”) onto the function name (“pkg”) with a dot.
So far, so good. But it quickly leads to this kind of silliness:
"{{ pillar.installdir}}/myfile.txt":
file.managed:
- unless: ...
another_command:
cmd.wait:
- watch: [ "{{ pillar.installdir}}/myfile.txt" ]
Free-text strings don’t make great IDs. On the whole, I’d prefer a syntax which is robustly readable and predictable, rather than one which is very succinct in special cases, but mostly less so.
The next problem is that all those IDs have to be unique across all your .sls files. That’s tedious when you have repetitious little actions that you can’t be bothered naming properly.
Another annoyance is that Salt’s data structure sometimes requires lists, and sometimes requires dicts, and it’s hard to remember which is which. For example:
finishlog:
cmd.run:
- name: |
echo "All done! Enjoy your new server.<br/>" >> /var/log/salt/buildlog.html
file.managed:
- name: /var/log/salt/index.html
- source: salt://initindex.html
- template: jinja
- context:
buildtitle: "Your server is ready!"
buildsubtitle: "Get in there and make something."
buildtitlecolor: "hsl(130,70%,70%)"
Notice how the first level of indentation doesn’t have hyphens, the second level does, then the third level doesn’t? I’m not sure what the philosophy is here – it looks to me like each level is just a list of key/value pairs.
Jinja templating logic
Jinja is a fantastic templating language. But Salt uses this template language as its core programming logic. That’s like writing a C program using macros all over the place. It’s sort of ok as a way to access data values:
update_data:
cmd.run:
- cwd: {{ pillar['tm_dir'] }}
Here, the {{ … }} bit is a Jinja substitution, so the actual YAML code that gets parsed and executed looks like this:
update_data: cmd.run: - cwd: /mnt/saltymill
As most people who have dealt with macro preprocessors know, they rapidly get complex and very hard to debug. The error you see is a YAML parse error, but is the problem in your YAML code, or in what got substituted by Jinja?
It’s certainly possible to use the full range of Jinja functions (that is, {% … %} territory) inside Salt formulas, but for sanity I’m trying to avoid it. On the flip side, certain things that I expected to be easy, like defining a variable in a state {% set foo = blah %} and then being able to access it from inside any other template, turn out to not work. You need to explicitly pass context variables around, or import state files, and it becomes quite cumbersome.
Help may be at hand. Evan Borgstrom also noticed that “simple readability of YAML starts to get lost in the noise of the Jinja syntax” and is working on a new, pure Python syntax called NaCl.
Many ordering options
There are about 4 competing systems simultaneously operating to determine which order actions get executed in, ranked here from most important to least important.
- Requisite order: The “preferred”, declarative system, where if X requires Y and Z requires X, then Salt just figures out that Y has to happen first.
- Explicit order: you can set an “order flag” on an action, like “order: 1” or “order: last”. Basically, if you’re tearing your hair out with desperation, I think. (I tried it once, and it seemed to have no effect.)
- Definition order: things happen in the order they’re declared in. (By far my preferred option). This doesn’t always seem to work, in the top.sls, for reasons I don’t understand.
- Lexicographical order (yes, really): actions are executed in alphabetical order of their state name. (Apparently this was a good thing because at least it was consistent across platforms.)
The requisite ordering system sounds good, but in practice it has two shortcomings:
- Repetitiveness: let’s say you have psql 10 commands that all strictly depend on Postgresql being installed. It’s very repetitive to state that dependency explicitly, and much more efficient to simply install Postgresql at the top of the formula, and assume its existence thereafter.
- Scope: Requiring actions in other files seems to prevent the running of just this file. That is, if in load_db.sls you do “require: [ cmd: install_postgres ]”, then this command line no longer works: salt ‘*’ state.sls load_db
I’m yet to see any advantages to a declarative style. I like the certainty of knowing that a given sequence of steps works. The declarative model implies that one day the steps might be rearranged based only on my statement of what the dependencies are.
Arbitrary rules
There’s a rule that says you can’t have the same kind of state multiple times in a statement. This is invalid:
/var/log/salt/buildlog.html: file.managed: - source: salt://initlog.html file.append: - text: Commencing build...
Why? I don’t know. The rules dictate that you have to do this:
/var/log/salt/buildlog.html: file.managed: - source: salt://initlog.html append_to_that_file: file.append: - name: /var/log/salt/buildlog.html - text: Commencing build...
Notice the cascade of consequences as this fragile “readability” tower crashes down:
- You can’t have two states of the same type in a statement, so we move the second state to a new statement; but:
- You can’t have two statements with the same ID, so we give the second statement a new ID; meaning:
- We now have to explicitly define the “name” of the file.append function.
Clumsy bootstrapping
I miss Chef’s elegant “knife bootstrap” though. In one command line command, Chef:
- Defines properties about the node, like its environment and role.
- Connects to the node
- Installs Chef
- Registers the node with the Chef server, or your client, or both.
- Starts a deployment, as required.
With Salt, these all seem to be manual steps:
- I SSH to the node
- I download and install Salt (using one-liner bootstrap script)
- I write ‘grains’ to the newly created /etc/salt/grains file (yes, I think I’m doing this wrong – roles should be defined through pillars maybe)
- The node then attempts to register itself with the Salt Master, but because the connection is insecure:
- I have to go to the Salt Master and accept the pending key: yes | salt-key -A [and the docs suggest you’re supposed to manually verify the keys!]
- Now I can launch a deployment: salt-master <nodename> state.highstate
When I rebuild a server, the above are preceded by these steps:
- Log in to OpenStack Dashboard, click ‘rebuild’ on the instance.
- Edit my ~/.ssh/known_hosts to remove the old SSH key.
- On the Salt Master, delete the old key: yes | salt-key <node> -d
There are probably ways of streamlining this process. I hope so, anyway, because it’s pretty clumsy. I don’t know yet whether the SaltMaster can automatically launch deployments when new nodes register.
Steve Bennett blogs 
Learning salt over the last month as well, and I agree with a lot of this. I feel like some of that may be me, and some a result of learning new syntax of several languages at once – yaml, jinja, python, and salts own expectations. If there was the intro course, salt for programmers which could make broad statements like – every time you want to define a variable, instead do {x}, or paired most actions to a validation command that could tell they worked correctly.
From what I’ve seen salt-cloud (now / in the next release) folded into salt itself is the path around your final comments – if salt handles the initial creation of your instance, then the naming and ssh handshake/salt-key steps are done for you.
I could not agree more !! It’s now been a long month since I started to use salt and the worst pain in the ass that I have is to clearly understand the purpose of the pillars/grains/states and filesystem hierarchy depending of.
I do love salt regarding any other DevOPS tool like Chief/Ansible/Puppet and so on, and I’m not disturbed by jinja/YAML/Python as I’m a Google Cloud plateform user which is heavily using them.
I do love salt command line simplicity and security provided by the SSL line.
But I’m also pissed of by how complex it is to set it correctly for large scale environnement and the lack of features regarding bootstrapping and PKI/OpenLDAP link.
I have been learning and playing with SALT whenever I have time. Really cannot stand maintaining Puppet manifests anymore. I must say the SALT documentation is completely horrible to me. Concepts are not clearly defined. The most important concept of a state is truly confusing. Often I must read the document a few times to know in which files to add example codes. However, at this point, I feel SALT is still easier to use than Puppet.
From reading and poking around, grains are just facts from facter, pillar items are like those properties that can be looked up using extlookup, sls files are like Puppet manifests and templates corresponds to erb files (but more powerful).
It is very easy to trigger minion (client machine) updates from the master itself rather than login to client machine and run puppet agent.
In my opinion, the real problem with Saltstack is how ridiculously hard it is to debug. Right now, I’m trying to set up nginx to serve a domain. This should be easy. I should just have to drop a file into /etc/nginx/sites-enabled and have Saltstack watch the file. But nope. nginx doesn’t get restarted when the file gets dropped in. So I have to manually restart nginx. The logs are no help, at all. Silent errors aren’t fun.
This is my biggest issue as well. The end result is always drop in to the #salt irc channel and ask for help. That method has worked every time for me.
I’ve not had any problem debugging salt. Quite the contrary, I just stop the salt-minion service and run it myself with -l debug, then I run highstate on the master with -l debug as well. At this point, eEverything I need to debug was pretty much in either the master terminal or the minion’s.
If you put a watch requisite on your nginx service.running state, it will watch the file for changes and restart if any :)
I haven’t tested, but i think you could do:
/var/log/salt/buildlog.html: file: - managed: - source: salt://initlog.html - append: - text: Commencing build...I agree that the hash/list differences (and which is used where) is confusing and feels very arbitrary.
That’s a promising idea, but it doesn’t work (I just tried). The expansion of file.managed puts the “managed” part on the same level as source, so what you’re doing actually expands like this:
/var/log/salt/buildlog.html: file: - managed - source: salt://initlog.html - append - text: Commencing build...Which obviously won’t work. Trying to do it your way gives the error message “
No function declared in state "file" in sls fooo“.Getting started and feeling some of your pain. The docs could really benefit from some diagrams (page 35 of this helped: https://speakerd.s3.amazonaws.com/presentations/49d401c0a3d80131166e024e11a95d47/Getting_Started_with_Salt.pdf ). It took a few passes at it before it started clicking for me–woe to the visual learner. You may find some things in there that help regarding YAML: http://docs.saltstack.com/en/latest/topics/troubleshooting/yaml_idiosyncrasies.html
I definitely plan on using ID’s for logical names (and avoiding that shortcut). Regarding the data structure list/dictionary thing–if you can guess that a context param is likely a dictionary, then it’s more clear why the YAML is as it is (although maybe using the explicity { } ‘s per that link would good.
I’ve been running “salt-minion -l debug” and haven’t yet had a problem debugging (but just getting started).
Only ever tinkered with Puppet but starting to like Salt as a go from crawl to walk.
Thanks for the great comment. It’s actually not just YAML *itself* that causes me problems, it having to use YAML to express a data structure whose properties are a bit inconsistent and a bit hard to grasp. Ultimately you’re trying to build Python lists and dictionaries, but trying to remember which goes where is hard – and having to express them in YAML, and getting syntax errors in YAML just makes it a bit frustrating.
Regarding your comment about YAML, I just wanted to mention that YAML is a superset of JSON and you can use bracket lists instead of multi-line hyphenated lists. You can also quote all of the text items if you are glutton for punishment. :-)
Yeah, but it’s sensitive to whitespace in a way that JSON isn’t. It’s very easy (for me) to write JSON and be very confident that I have the syntax rules right, since there are so few of them. In JSON, you have [lists, of, things], and you have {dicts: “of things”} and you can embed either inside the other, and it’s all very simple. You can stick whitespace wherever you want (other than inside quoted strings, obviously), and nothing will ever break.
In YAML, everything is fragile. Too many spaces or two few, and it breaks. List items need to share an indent level, and be indented relative to their parent. You can wrap a line, if you’re really careful, and there’s at least five different syntaxes for that. (Yes, five: > then block, | then block, “block”, ‘block’, and terminating each line with \).
Good points on some limitations of salt. The yaml syntax can be messy.
I havent worked w chef or puppet, but I tried picking up puppet a few times and it was way over my head (our company’s main developer swears by Puppet though, but he also despises GUIs and anything visual). I found salt by comparison to be very straight forward.
As for documentation, I think Salt docs are pretty thorough but definitely lacking any diagrams or conceptual graphics. Im a visual guy and its much easier to understand how a Reactor works for example, via a graph. You will find tons of salt tutorials out there with good explanations. Heres a link to some of them,
https://sites.google.com/site/bladelogicwiki/salt-stack/guides/salt-tutorial-links
also I created a cheat sheet for basic commands that I use more and more now,
https://sites.google.com/site/bladelogicwiki/salt-stack/guides/salt-commands
The biggest problem is the documentation is horrible. At first glance it looks OK because there’s a lot of it and it’s nicely formatted. But when you try reading it, it’s clear that some tech writer who isn’t really familiar with the product wrote it. (It’s also full of misspellings and grammatical errors.)
Actually, it was all written by the developers themselves. It’s auto-generated by Sphinx from the comments in the code.
As someone slogging up the saltstack learning curve, couldn’t agree more with your post. The lack of clarity around salt’s definition of a “state” and how it relates to “modules” and their functions has been frustrating. Thanks for your post.
Pingback: Required Reading – 2015-05-18 | We Build Devops
Ive been using Salt for a year now and its been a boon to our web hosting business. in the process I learnt that you cant tackle complex ideas until your solid in the basics. For example, your file append issues and other operations is easy, state the file name and then list all the operations under it, dont split them apart (which seams logical but isnt how to do it). YAML is a pain in the arse if you forget to space things correctly, basically its either 2 or 4 spaces and any op that takes parameters usually ends in a colon. grains are easy – use the salt command line to set a grain using setval i.e. “salt ‘my.host.com.au’ grains.setvals “{‘php55′:’True’}” then you can use the “php55” grain to do other operations which works well when you are deploying different environments. Also, make a wiki page of your grain assignments so you do things logically.
last but not least here are some articles I wrote on Salt: https://z900collector.wordpress.com/linux/saltstack-handy-scripting-tips/ and https://z900collector.wordpress.com/linux/getting-started-with-saltstack/
Do a search for SALT and you will find more.
Salt is great but it takes Baby steps to become fluent in using it effectively, after 30 years in IT it is the one tool I think has made a real difference to managing and deploying code. Also see here: https://www.conetix.com.au/blog/software-deployment-saltstack
Sid
I forgot to add – use highstate to initially setup your target server, after that deploying using salt scripts is often done adhoc and so highstate can get out of state :) I rarely use the highstate after its been run the first time and I manage 1000+ servers.
It’s been 2016 and you know what… Not that much has changed! I spend most time to learn “getting things done the salt way”, rather than actually working on infrastructure… I think next time I’ll have a few tens of spare hours I’ll just write my own “orchestrating tool”…
I know the feeling, however I’ve also documented many handy commands and scripts here:
I expect it will take more than a week to get up to speed, but you will not look back :)
Sid
quote: and the docs suggest you’re supposed to manually verify the keys!
oh my god I have to enter the pin for my credit card … manually … why didn’t they just save it on the card?
Looks to me like the developers didn’t understand YAML themselves! I use it a lot and love it, and unless you absolutely cannot abide the indentation-defines-structure style which it shares with Python, there’s not much to dislike when it’s used properly.
You ask about the lines:
file.managed:
– name: /var/log/salt/index.html
– source: salt://initindex.html
…
Well that’s just totally stupid! It’s defining an array of single-item dictionaries! GOK why, but it just screams to me, “I don’t know what I’m doing but I’ve hacked it at random until it seems to work”
Bottom line? Developers are idiots, RUN AWAY FAST.