Hunting the Flush Bug

The Flush Bug, a periodic event that effected one of the machines that was worked on while at Shandon, had an career changing impact on Martin. That impact was not in the way that was expected at the time, though.

Among the machines that Shandon made was a Wax Embedder. This machine embedded tissue samples in wax on special cartridges. Thus treated, the samples could be sliced very finely on something that looked like a glorified Bacon Slicer, but was considerably more sophisticated, and known as a Microtome. The Wax Embedder itself looked like a Washing Machine. Again, it was considerably more sophisticated, and since only a few were ever made, basically hand-built. They were thus expensive and profitable to Shandon. The machines consisted of a centrally heated bath with sensors and ports where reagents and wax could be introduced and drained. The whole process was automated, so things could be left running unattended. Initially this automation was in the form of a cam-timer. A kind of clock-work mechanism that was found in many Washing Machines of the time. However, this wasn't modern enough and an eight-bit microprocessor solution was adopted. The microprocessor was, in turn, controlled by several thousand lines of dense comment-free assembler code that had been written in a caffeine fuelled frenzy by one of the electronic engineers. He appears to have instantly regretted this and had been an enthusiastic supporter of my employment, as a software engineer.

The concrete results of all these efforts

Although a relatively mature product, the Wax Embedder suffered the odd unexplained flushing event were wax or some reagents were removed unexpectedly. The source of this problem was widely held to be this monolithic assembler program. The phenomenon was therefore called "The Flush Bug" and I, to my great chagrin, was roped into help with the debugging effort. I would have been happier staying with designing the do-it-properly software-engineered fancy centrifuge, the thing I'd been hired for.

Extensive desk-checking did not disclose this bug, although it improved my Z80 assembler skills immensely. It also gave me an abiding desire to code everything in C in future. And in this the seeds of a real-time Executive based on a UNIX Process was germinated.

In an effort to get more data it was decided that I and a technician should monitor one of failing machines as it went through a complete cycle. This had to be done overnight which involved both of us taking turns to spend time in front of meters making notes. The person off-duty was fortified by some modestly pornographic magazines that my colleague had thoughtfully brought in, and what technical amusements that UNIX of that age afforded. A long tedious night resulted in nothing. The machine behaved impeccably. No Flush Bug was observed.

Still, my hourly-paid colleague could look forward to a fairly substantial over-time payment. I as a salaried employee could not. I was however invited to go out and buy myself a present, by way of compensation. I went to my local hardware store and bought this work-bench thing. I needed something like that for my new house. I've still got it and it's still invaluable.

So what became of the Flush Bug? After all this effort we were no nearer finding it. The mechanical engineers, disparaging our efforts had suggested that the next iteration of the machine should use cam-timers once again. I was detailed to cost up rewriting the software in C. Some time had past, and my real-time Executive based on UNIX processes was up and running. The some what crude software engineering that this allowed was seen as good and wholesome. Something like that should be done for the Wax Embedder and it would be good and wholesome too.

Unfortunately, everybody was fully engaged with the fancy centrifuge, particularly me. I was therefore asked in an rather casual way was to enquire about getting external help. A contractor, in other words. Insouciant enquires lead to a variety of agencies and the realisation that this was an expensive solution. It was therefore abandoned on those grounds. It was interesting to find out just how relatively high a fee a software contractor could command back then. And all for being both technical and flexible. Later my financial situation got tight, shall we say, and my domestic comforts fled, this seemed like an obvious course of action. So the Flush Bug had an impact on my future career too.

The Flush Bug was still there, though. And then, suddenly it wasn't. The mechanical engineers, who has not been idle on their part, had discovered that if one of the sensors in the bath got blocked, with wax, then false readings would be given to the software and an unexpected flushing event occur. A new cleaning regime was recommend to customers and a redesigned sensor enclosure mooted. Little was heard of the Flush Bug after that.

The Flush Bug influenced so many things, my long-term career choices being the least of them. In the end it turned out to be a mechanical problem. I thought that the mechanical engineers owed us software chaps a Big Night Out. But this never happened, and since we were all based in Runcorn it was probably just as well.

Cockermouth, 7th September, 2010.

~Z~

Navigation

This is some navigation, it should appear at the side if the page is wide, and at the bottom if it is not.

Gormless Anecdotes

Martin is known for his long-winded e-mails and has decided to inflict them on the Internet like this, suitably cleaned up and formatted. There's the starting page with some explanation, and then, so far:

• Hunting the Flush Bug
• Chemi-Luminescent Vim
• Helping With an Enquiry
• Light From Under the Door
• I Write Like David Foster Wallace
• The Origami Artist
• Pattern Matching Randomness
• The Indole Chemistry of Bananas
• Volunteer Projectionist

Also of Interest

Jucy morsels from the rest of the site:

• General papers on On-line Transaction Processing (OLTP). It's still the main topic of the web-site.
• Some site news; it's a sort of infrequently updated journal for Martin too.
• Rather specific Technical Notes, Martin can still bore people in the Third-person past tense.
• Of the little hacks that he's done recently, the Unofficial Jobcentre Plus Mirror is the one thing that seems to have gained traction.
• Other ephemeral, eclectic stuff. Mostly photographs.

About the Author

His name is Martin Sullivan. He's a fifty-something some-time computer programmer living in the quaintly named town of Cockermouth in north-west England.

The Company

ZOIS Limited is basically a vehicle for Martin and his OLTP work, he owns it. You can find out more about it elsewhere on the main site.

Pro-Bono Plug

Martin is currently on a health sabbatical, but is interested in the odd bit of pro-bono work by the way of therapeutic recovery. So, if you've anything that he can tackle on a non-commercial basis from his base in Cockermouth please let him know. Unsurprisingly, he's particularly interest in biology, again, and the application of computing to it.