Ick. Here are the results of bbouncer for SilverKeeper, which is what I use for backups:
Verifying: basic-permissions ... ok Verifying: timestamps ... Sub-test: modification time ... ok ok Verifying: symlinks ... ok Verifying: symlink-ownership ... FAIL Verifying: hardlinks ... FAIL Verifying: resource-forks ... ok Verifying: finder-flags ... ok Verifying: finder-locks ... ok Verifying: creation-date ... ok Verifying: bsd-flags ... FAIL Verifying: extended-attrs ... Sub-test: on files ... FAIL Sub-test: on directories ... FAIL Sub-test: on symlinks ... FAIL FAIL Verifying: access-control-lists ... Sub-test: on files ... FAIL Sub-test: on dirs ... FAIL FAIL Verifying: fifo ... FAIL Verifying: devices ... FAIL Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... FAIL Sub-test: lots of metadata ... FAIL FAIL
That's not particularly encouraging.
Jamie Zawinski is someone who understands backups.
Do exactly what he says in that post and you'll be fine.
Is Server Virtualization the New Snake Oil? I think it depends on why they are pushing it. If it some sort of notion that you get more horsepower with a similar amount of cash, power, or cooling I'd say yes. Virtualization to save servers takes advantage of the fact that most non-prod workloads aren't really maxing out the hardware. If your prod load isn't maxing out the server, and you're not getting hardware redundancy, why would you add another server?
If you do see a gain, I'd assume you could get the same gain by optimizing your app. Maybe its cheaper/easier to just plop it on two vms, but that's a case by case thing, and by going with vms, you are almost certainly picking ip overhead.
Here's one good reason though, maybe a 3rd party component is poorly threaded. We once had two big 4 proc boxes to run a 3rd party app as part of our config, and only when we went to prod did we discover that the underlying embedded db could only run about 2simultaneous operations without bogging down. We needed lots of 1 CPU boxes, not 2 big ones. Virtualization would have let us get everything out of that hardware. So there's one useful case.
There are more though. How about forensics? See a badly behaving vm? Freeze it and put another up in its place, and your back a k own good config. Then you can analyze the copied, bad vm offline. If its a security issue, you immeditely have a bare metal copy that you can dig through as much as you want.
Patching, mainenance, deployments, rollbacks, etc. all get additional flexibility.
I await anxiously the time when the overhead is low enough to virtualize just about everything if it makes sense.
Cool, the original delivery date for my TX-1 was pushed to June, but they apparently shipped it early, 'cause it arrived yesterday from Amazon.
I'll post a mini-review soon. First impression is, wow, I'm glad I got a bigger hard drive, and face detection is just a little creepy. It's also sort of different enough from older Canon's that I'm going to require a trip to the manual to figure out how to do a few things.
Over on the strobist website and flickr community, the group has had it's growing pains a little bit as all online communities do. It's still a great community, but it did have to grow some rules, which is fine.
What's funny is that our fundamental troll-bait issue is that of pocket wizards vs. e-bay wireless triggers.
Rule: Online communities will always find something to argue about.
I was starting to get a bunch of spam sneaking through Scode and MT-blacklist, so I decided to upgrade to 3.33 (which is back to being free for personal use).
Until I get the templates all pimped out, there may be some weirdness. Apologies.
Okay, I love my Mac, but this is nuts. I can connect to the video test accounts just fine. And my parents can connect to those accounts just fine as well.
but when we try to connect together, nadda. Just "error -22". If I remove my voice over ip modem (nice of someone to mention that they all use the same protocol) from the chain, then we get "error -8". That's getting closer to zero, so maybe that's better?
No idea, totally frustrating. Sorry Apple but you missed the boat by about a million miles on this one. I don't think the average user has a chance in hell of making this work.
7/31/07 Update:
I did finally figure this out, and it wasn't quite as freaky a solution as I had feared. I had thought maybe my VOIP modem was doing something to the SIP initialization from iChat, but removing it didn't make a difference. The problem was actually on my parent's end; they have some weird DSL modem that does NAT as well. Then they have a wireless router immediately behind that, and all the computers hang off the router. It was that double-NAT jump that was breaking things (predictably), at least I think. Plugging their computer directly into the DSL modem takes care of the problem.
I'm trying something new today. The blog format gets a little cramped for longer form articles, so I'm going to try a few standalone articles elsewhere on the site. First up is a piece on teaching people troubleshooting and debugging. Let me know if you find it interesting or useful; you can email me or leave comments here.
Who needs SQL? :-)
In the old-school spirit of giant, strung together command lines for unix, you can use sort and uniq (one of my favorite unix commands) as a sort of "group by" clause in SQL.
For example, say you have a bunch of different types of error messages coming from some process (in my case today, a nant build with tons of unit test failures). How many of each kind are there and how many different kinds are there?
mytask | grep "something present in each error message" | sort | uniq -c
That selects just the error messages, sorts them (because the next step only finds ADJACENT identical lines), then groups them and counts how many of each their are. Useful, useful stuff. If the formats aren't exactly the same or what you want, use sed to select just the part you wish to group by.
Here's one that if you feed it my mailbox, displays the spamicity scores and counts of each piece of mail in the box:
grep spamicity $1 | sed 's/.*spamicity=\(.*\), ver.*/\1/g' | sort | uniq -c
Somewhere I have a useful one-line web log analyzer as well, for counting hits by day by selecting off the date only. Ah, here it is:
sed 's/.*\[\(.*\):[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\} -[0-9]\{4\}\] ".*/\1/g' | uniq -c
(No real pressing need to sort, the logs are about 99% time-ordered already). That regex didn't spring fully formed out of my head, I started with a chain of awk and sed that was much simpler to understand, then gradually built it into a single regex for performance. I still keep the original awk around somewhere as a comment. I think that works against the standard apache "combined" format log, and I'm not guaranteeing its perfect, but you get the idea.
I've been thinking a bit about virtualization recently (as in hardware virtualization through software, ala VMWare). It's great, wonderful, solves all kinds of problems.
But I do hope that any enterprise using a bunch of virtual machines has automatic downloading of patches enabled, and keeps the patches available for a long time. Because each of those snapshots are frozen in time, and when you bring them up, they are x months (or years if they are archives of particular development systems or customer images for troubleshooting).
There's a ton of benefits to developers by having a large library of snapshots of systems in various states. There's just some extra thought required in how to handle all those library images securely, since in one way they are a lot like ticking timebombs.
I love Roy Osherove's thoughts on unit testing, so this is rather off-topic but he got exchange running overnight.
This is the whole microsoft schtick. Easy to install.
It's supposed to be a giant, enterprise mail program. I'm only going to install it once every 5 years hopefully. I don't care that someone who hasn't used it before can set it up in 6 hours.
I'm perfectly happy to have that sort of simple install option, bt from what I've seen it comes at the expense of ease of maintenance and automation, which are what I really want out of enterprise products.
Also it's a several thousand dollar program as I recall, and pretty much requires a dedicated machine. Is installing an expensive, resource intensive, "enterprise class" mail server (that you now have to backup, maintain, and keep updated) reall the easiest way to synchronize mail and contacts?
I'm not sure exactly what the requirements are, but I think I do this with a one-line rsync command...;-)
Prakash Malani has a good comparison of the various options for configuring weblogic in his blog. He lays out what's available pretty well.
The option I've gone with is the last option presented, templating the config.xml. Using velocity templates, I can control the configuration to come out however I want. This was all built before the offline scripting tools were really available, and the idea of actually having to have the admin server running first is problematic for me.
Maybe it's an artifact of the way I think about web apps, but I think of them as shrinkwrap software. Hand it off to ops, tell them to untar it wherever they want, run the start script and it should just work. In general I think just about every app server ignores configuration; as if a JAR or WAR is the whole application and is all that's required.
What weblogic really needs is a way to have multiple config files so the developers and ops/engineering stuff is split out. The developer's file contains the -names- of the connection pools, JMS queues, etc. that the code depends on, and then the operations file includes the IP addresses, tuning parameters, etc. that ops should control outside of a codebase.
Currently the config.xml is everything, even though the requirements are totally different for various peices of information. It should be possible for the developers to easily provide a contract for what items are required, but the contract is them implemented by a bunch of config settings that dev cares nothing about. "Give us a connection pool named foo, etc."
This is not specific to weblogic, it's basically every app server I've seen., and a lot of things that aren't even app servers really.
Check out the way that jetty shows for tutorial configurations. It's all done in Java code. Now that's interesting. It also accepts XML, which is good, but that's what everybody does.
I like exposing the container in this way. Instead of being forced to provide a config.xml or whatever, you can handle configuration however you'd like. So environment specific values, meta-information about the build, etc. can be determined or brought in programmatically. This seems very flexible to me, and could be customized for just about any need. I'll have to look into this some more.
What a nightmare. I just spent basically three days troubleshooting a problem under windows. Basically, we needed to talk to a service from an ASP page. Since the MSXML 4.0 control didn't work as advertised (the asych mode where you can control timeout doesn't work at all; it works with version 3.0, but that older one has a bunch of caveats that make it dicey as well), we built it in C#/.NET, and used COM interop to call it. It worked fine on our notebooks. It worked fine on the CMS content server where login as a real user is required. It even worked fine when you put "Server.CreateObject("interop.progid")" into an an aspx test file (running of course, as the ASPNET worker process).
However, in the one case where we needed it to work (under an ASP classic file), nothing. Total failure. Blew up right on the Server.CreateObject() line with:
Active Server Pages error 'ASP 0241' Create Object Exception The CreateObject of '(null)' caused exception C0000005.
Yes, we had already given the IUSR/IWAM users access to the windows temp directory, since .NET web service stubs use the XML Serializer to create a temporary stub class. Filemon reported the .cs file being generated, and a bunch of reads to the dll file...
Put the IUSR/IWAM users into the admin group, no help. Ran a bazillion filemon traces, no obvious access denied messages. Nothing would make this thing budge. So I tried just about every trick in my book, and made some progress in narrowing it down in fits and starts. Finally, when comparing the filemon runs from where it worked locally to the non-working ones, realized that csc.exe was never running...turned on the "leave behind temporary files" (described in the above article) and the dll was coming out zero length, even though if you ran csc.exe on the cs file with the provided .cmdline file, it built a dll just fine....
Turns out that the IUSR/IWAM (probably just IWAM, but I was too tired to pay much attention) users have to be able to access csc.exe and cvtres.exe to compile that file. I had thought of that, but I had just added an inheriting permissions rule for that to the framework directory. Well, guess what; the handy IIS Lockdown tool adds an explicit DENY to the ACL for the web anonymous users in everything under %windir%. We had a long permissions list so the deny was waaaaaay at the bottom since the group starts with a "W". And of course, denys always win. Which is why putting the IUSR's in the admin group didn't help either.
So there you go; if you have COM objects wrapping .NET code that uses the XML serializer, and you want it to work for a classic ASP page, you have to make sure that the IUSR/IWAM users can run the compiler. Obvious, but not obvious when you see a bunch of reads to csc.exe by DLLHOST.exe, and assume that maybe because of some other config switch the compiler might be running in process.
If only there was "truss" or "strace" for windows. Would have taken 30 seconds, because I would have immediately seen the failed "createProcess" or "fork", or whatever it is that Windows calls it. Lack of basic administration tools continues to hamper Windows, even at this time. Without filemon, which isn't even a Microsoft tool, this would have been impossible. Aside from perhaps installing a full debugger and hooking into the right calls that way. But without knowing where to look by narrowing it down via Filemon, this would have been an ugly process.
Anyway, just another typical windows incident where on the surface things look attractive: "Use this API, call a web service, run this little utility, call it from COM," but the reality as you move to a production environment is another thing entirely. When the abstraction breaks and doesn't "Just work" because of one of the million different things that might go wrong on each system, you're left scratching your head, without the tools to solve it easily.
All they have to do is add an officially supported truss/strace equivalent, and most of this goes away. In this insntance, I'm sure that the .NET code was generating a nice exception somewhere, but it was getting lost across the COM boundary.
This was, incidientally, the 2nd hardest time I've ever had troubleshooting something. If anyone's interested, I can write up the hardest later this week.
This is Rob Meyer's weblog, a weblog focused on software development and system administration based on 10 years of experience. Want to explore further? You can find out more me or see the rest of my website.
Wondering if I've written on something in particular? Try searching:
You might want to take a look at some of the more requested postings (as judged by incoming traffic):
Want more? Subscribe to this site 
or contact me at rob at big dis dot com.
See my writings on: