Spine: Overview

Spine is framework for managing configurations of large numbers of Unix machines. Currently, only Linux is fully supported, but other Unicies shouldn't be hard to make work. Support for different package managers will be the biggest hurdle there.

What do we mean by large? At least thousands. Ticketmaster manages their world-wide infrastructure of over 3000 systems with Spine. What do we mean by a framework? Spine isn't designed to out of the box allow you to say "hey this is a DNS server, go configure it." This would assume we know how you want to run your DNS servers. Instead, Spine allows you to define a set of roles, keys, and templates that define what a DNS server is to you, and describe what machines in your network play this role.

Spine handles tasks like package management (ensuring only the expected packages and their dependencies are installed, nothing else), config-file management, NFS-mount management, system hardening, restarting services (when their config files change), disabling / enabling services (based on the configuration), and local user management. Spine, for some groups, ties into their host IDS, among other things as well.

Spine excels at allowing administrators to manage a very large number of systems where those systems are either fairly homogeneous, or are a heterogeneous group of homogeneous systems. In other words, it assumes the fact that you at least have groups of similar systems. Maybe application servers, proxy servers, cache servers, and DNS servers, for example. While Spine is designed with flexibility in mind, it's main audience is not shops machines are all unique and share little or no configuration with any other machine.

Spine also assumes a fairly consistent environment. For example, it assumes some consistent way of pulling information about roughly what sort of groups a system belongs to. By default, it pulls this information by parsing the host name. Thus, a reasonable naming schema would is required. Spine could fairly trivially be adapted to pull this from somewhere else, but again, that source would need to be fairly structured and consistent. There are other examples of the way in which Spine assumes a fairly consistent environment such as the way it determines NFS mounts (it expects to have some sort of regex-able structure to match mounts to systems).

Spine is completely pluggable - if it doesn't do something you'd like it to do, writing a plugin for it is very easy. And since Spine, and it's plugins, pull nearly all of their configuration from a hierarchical configuration tree, you don't need to worry about how to configure the plugin.

What Spine Is Not

As mentioned before, because Spine is a management framework, it's not out of the box a management system. You must set it up with a lot of knowledge about your environment before it will be even remotely useful to you.

Spine is also not a provisioning system. However, Ticketmaster has released "provision", a provisioning tool similar in style to Spine.

Why Spine?

There are other configuration management system such as cfengine and puppet, why release another? Spine works on a different paradigm from anything else out there. As such, we feel it allows you to scale your admin-to-server ratio easier. Here are some of the key differences:

• It's hierarchical. This one is the big one. This means:
  • You don't need to define all your machines, much less try to keep spine in-sync with provisioned and de-provisioned machines
  • You don't need to define a config per-machine. A machine's policy (or "descend_order") is discovered via local system info, a dns schema, and whatever else you'd like to throw at it. Of course, if you need a specific config for a single host, that's possible.
• It has revisions
  • This means you can roll forward, roll back, freeze on versions, etc.
• It has templates
  • Leveraging the aforementioned hierarchical configuration tree, you define config files like /etc/hosts only ONCE... as a template that uses the config tree to determine the contents

There are others, but these three are the big ones.

A Few Notes On The State And Quality Of Spine

1. What we did right
   • Hierarchical configs
   • Easily pluggable
   • Revision control of configs
   • Templates
2. What we did wrong
   • Templates should define their location via... a key, or meta-data, not by location within a tree. This isn't nearly flexible enough
   • The configball publisher doesn't scale well
   • Rather than only enforcing the hierarchy (as useful as it is), we need to be able to specify configs that apply to arbitrary parts of the tree for times when the hierarchy isn't sufficient.
   • Redhat-isms (such as /sbin/service) and apt-rpm assumptions are made
   • Poor exception handling
3. What's coming
   • The ability to apply keys based on arbitrary criteria
   • A better configball publisher
   • A better (read: queriable) back end data source
   • Support yum for dependency resolution