Sysadmin Field Notes

November 16, 2004

Simon is talking about environmentally-friendly configuration within webapps.

For those of you who know me, This is something that I've long consider to be a mostly-unsung problem in developing web applications. Managing all that configuration information is tough; it's in different formats (regular property files, log4j properties, app-server specific deployment descriptors, app-server specific configuration files, shell scripts), it can be large (some of our applications have a couple of hundred properties that must be managed), and if you work at a big company (this is a big one), then different groups should probably own different parts of the settings. And most distressingly, the vendors provide very little support for handling any of this.

For example, say we're talking about the production database password. It usually goes into the app server configuration file somewhere. Some products will support easily pulling this from somewhere else, some will not. Taking the weblogic view, it's stored in the main config.xml file that contains the configuration for the whole server. Now, this file contains a bunch of stuff that developers care about; application component definitions, JNDI names of resources, etc.. However, the developers have no need (and in some places aren't allowed) to know what the production database password (or IP, etc.) is. So you can't just check it into CVS.

Plus, if the system administrators want to change it, they should be free to, whenever they want, without having to push an entire new build, built from CVS. If it's in CVS and the admins change it locally, then someone needs to propegate it back to CVS so it doesn't get overwritten, but the sysadmins might not have access to CVS, so they have to get a develoepr to do it. In the right property file of course.

There are a host of things like this; anything that's an IP, a username, password, any security keys, etc. should definitely all be managed by the admins. There are a few grey area things, like URL's and tuning settings (thread counts, connection pool sizing) where both the admins and developers probably need to talk to figure out the correct one.

I've rarely seen this separation done well. Most of the time there ends up being either a sysadmin with some development chops building the whole system and maintaining it, or vice-versa. What's really needed is a nice, complete configuration system from the app server vendor that allows you to seperate your config files into multiple parts. So the developers can put whatever they decide they need to manage into CVS (things that should require a build to change), and other stuff can be split out and left on the system itself, persistent from build to build, and managed by the admins (probably via rcs, or cfengine, whatever their normal workflow is.

I've tried to basically do that by wrapping similar functionality over all the config files via vpp and a lot of templates. I think this works the best so far, but there are a few problems with this method (I can detail them out if you're interested).

My other thought is to make people build the app more like shrinkwrap. Pretend you have no idea who's going to use it. Create a from-scratch configuration/install wizard as part of the code (or use an off the shelf one) that configures the app as the admin sets it up (it of course would have to support some sort of automated unattended install file). So when you drop a new build to the admins, it's a fully contained unit that you don't need to worry about configuration. Depending on how vertically siloed your admins are from the developers, this may or may not make sense for you. It does increase the testing burden; but it would be a nice, elegant solution.