Failing to learn from history again

Learning from history is often hard - without the perfect vision of hindsight, it's easy to overlook the similarities of your current situation with past experience. This isn't at all comforting when one recognised the similarities and dismissed them with "This time, it will be different".

I often write "throw away" code - usually to isolate some specific code path or test some specific case that's currently interesting from the larger system. As the code's initial life expentancy is quite short, I often don't bother with niceties like proper argument parsing or decent error handling. In theory, the technical debt from this never be a problem, just write a little wrapper program, learn what you need, and throw it away.

The scare quotes and "initial life expectancy" phrasing should immediately identify my problem, though. If I didn't but the code in the VCS, the theory might work, but, once the code is there, it's never going to be thrown away, and, since it exists, in a day or a week or a month or something, I'll need something similar, and it's easy enough to just quickly add an option and make the throw script do a little more, and, since it's still just a throw away script, there's still no need to do proper error handling or argument parsing. And, of course, the moment a couple of the command line options become enshrined in a test script or two, the effort required to actually fix things never seems worthwhile, despite the increasingly convoluted "simple quick code" handling things.

Given that I've trodden this path several times before, this week's effort of going from 100 lines of C requiring a single filename to two 250-line interlocked programs, each with complex combinations of options and error handling from the oblivious school (the actual important code of each program is still less than 100 lines each), via a steady stream of "I'll quickly add this", is one of my more impressive examples of failing to heed lessons from the past. Next week, I'll probably spend some time fixing things, although there's still that ever so seductive "it's just one last option and then you can throw it away" voice to deal with.

How not to setup your firewall

(This is of course an entirely hypothetical set of notes, since it's inconceivable [1] that I would make any of these errors).

For the purposes of this exercise, we assume a gateway machine, running a simple firewall, which we want to use to drop various packets on the floor - such as those from machines particiapting in a distributed brute for ssh attack, for instance. We assume there are several scripts managing the blocked list. For best effect, your gateway box should be mounting a few directories via nfs from another machine.

1. Ensure that packets between the gateway box and the file server need to traverse the blocked list first.
2. Don't lock properly when adding new IPs to the firewall. This ensures that, under load, you can add large numbers of duplicate entries to the list.
   
3. Have every packet traverse the blocked list twice.

Under smallish numbers of blocked IPs, the results aren't impressive, so, for best effect, you need to wait until a there's a reasonably large attack before the effects are really noticeable. If done properly, however, at a few thousand unique entries, the network performance should slow to an absolute crawl.

[1] Which means exactly what I think it means.

An annoying discovery

One of the great annoyances of manufactured items is they tend to be very symmetrical. This makes various approaches to automated registration more complicated, since there are often ambiguous possible solutions, so one often has to shove in more domain specific knowledge to get the correct answer. Domain specific information makes thins less portable, and generally requires annoying maintenance effort later.

This is all; relevant, since I've been trying to avoid shoving such domain specific knowledge into the registration step for the calibration cube for the x-ray system, which, being quite a precisely manufactured cube, is rather symmetric. I had thought I had enough information available from the views to get around this, and solve the problem fairly elegantly, but I spent the later half of last week and most of this morning failing to make the idea work. The answer obtained looked almost reasonable, but had a 5 cm offset, and a rather high error.

I thus resorted to a heavy-handed optimisation approach, using the domain specific information to ensure the minimum was near the correct place.

This did not work. Given the fairly specific constraints I had placed on the problem, not working was a very surprising result.

Much digging later, I discovered a sign error in the input data I had created to describe the cube last week, which meant I was trying to register the cube a collection of fairly widely spaced separated planes, and getting odd results out.

While I now have a correct solution, I will now need to revisit my earlier idea, and see if it works better with the correct data. Since I spend considerable time checking the data last week, and had convinced myself that it was actually correct, I'm currently rather annoyed with the universe [1].

[1] Because it's clearly not my fault, after all.

Code Mainentance

One of the problems with an under-resourced development environment, such as the one I'm involved in at iTL, is the lack of decent code review. Consequently, the code base ends up with dirty little secrets, like the routines to automatically identify the calibration cube, a awkwardly tangled implementation of what still is actually quite a good idea. It's a chunk of code I'm both perversely proud of, and terrified someone else will look at and thus call me out as a total fraud.

Despite this, it actually works pretty well almost all the time, which is unfortunately an "almost" too many, so today was spent debugging this dirty little secret, which is never that much fun. It's easier than it may sound, though. Most programmers will have had the experience of looking at old code and thinking "How could this possibly ever work?", and most programmers will also have experienced looking at bits of old code and realizing "Oh, that actually quite a nifty trick". For me, reading this code tends to be a sequence of a's followed by b's followed by a's and so on. When the sequence doesn't end at b, it's probably a bug. The retroactive ego-boosting and deflating does make it a slightly bizarre experience, though, and not one I'm keen to repeat anytime soon.

Who knew?

Turns out that spending an entire afternoon in a meeting that spun around in circles doesn't leave me in a good state of mind for reading a thesis draft.

Stuff from the Day-job

So, after a couple of weeks of much intermittent panicking, we commissioned the latest version of the patient positioning system today. I feel rather chuffed about the whole thing at the moment, since it's always nice to see one's work used, but I could have done with a less stressful last two weeks.

Since the last version was commissioned (November 2006), several chances have been made. Most significantly, the control and drive systems for the chair were replaced. The improved electronics allows great accuracy, and the updated control system fixes the numerous problems with the old version. The vision system received polishing - various inefficiences were ironed out, and the code is generally much less fragile, and numerous bugs on the front-end were fixed. All in all, the new system represents a significant improvement, and is the result of lots of hard work by the MedRad development team.

As the panicking comment above suggests, the comissioning process did not run that smoothly, though. Things that went wrong varied include the serious - I had to rework and significantly simplify parts the lens distortion code due to issues with the repeatability of the results (trading theorical accuracy for practical reliability, which was the correct choice, but I will need to revisit and fix that code), the daft - a stupid error in the mathematical formulation, that all of us who looked at it missed completely, meant the final collimator rotation was only correct in the range 0 - π (Of course, the standard QA tests we used during the development phase only cover that range, so we didn't pick this up until quite late) and the standard - the spare box, which also needed to be updated booted fine, but completely failed to see it's frame-grabbers, so that had to be hastily rebuilt (leaving us short of additional spares for now), combined with all the usual last minute bug-fixes, but it is complete, and the accuracy figures do show a statistically significant improvement in the accuracy. Considering that the previous system was already accurate to within 0.5 mm, I'm pretty pleased with that.