Most BSD's have a useful notion of security levels, where certain operations are not permitted, even if the usual Unix permissions would allow them. Security levels are an important piece of a complete security puzzle (relying on it alone of course, like any security measure of course, gets you in trouble). One of these forbidden items in FreeBSD (4.2 at least) is the ability to directly open /dev/io. As far as I could tell in 15 minutes of research, XFree86 must have access to this device to work, otherwise you will get an ExtendedIO error message, complaining about opening /dev/io.
My workaround was just to lower the security level to zero. Not ideal, but should be fine for most workstations (and does your server really need to run an X server?)