Thursday, 27 August 2009

1st Principles: Curve Fitting (2)

Loose Ends of Quintic Trends

I was asked to fill in the blanks left at the end of the previous Curve Fitting article, so just quickly, here goes. Last time we assumed a linear mathematical model, y = mx + c. This time we start off with a more general one, y = p(x), where function p is assumed to have one or more adjustable parameters that we can tweak to find the best approximation.

As before, we first plot all n data points (x,y) on a graph, and conceptually draw the curve represented by our function p(x) somewhere through the middle of them. Then the sum of the squares of the error terms is,

∑(∆y)² = ∑(p(x) - y)².

For each parameter t in model p, we partially differentiate this sum and look for turning points:

∂∑/∂t = 2∑(p(x) - y)(∂p/∂t), which equals zero when

∑(∂p/∂t)p(x) = ∑(∂p/∂t)y.

Now let's plug in a concrete example, in fact let's make a beeline for that bloody quintic. This has six independent parameters, which we'll label a through f:

p(x) = ax⁵ + bx⁴ + cx³ + dx² + ex + f

We simply read off the six partial derivatives, and substitute each of these in turn into the above formula.

∂p/∂a = x⁵; ∂p/∂b = x⁴; ∂p/∂c = x³;
∂p/∂d = x²; ∂p/∂e = x; ∂p/∂f = 1.

a∑x¹⁰ + b∑x⁹ + c∑x⁸ + d∑x⁷ + e∑x⁶ + f∑x⁵ = ∑x⁵y
a∑x⁹ + b∑x⁸ + c∑x⁷ + d∑x⁶ + e∑x⁵ + f∑x⁴ = ∑x⁴y
a∑x⁸ + b∑x⁷ + c∑x⁶ + d∑x⁵ + e∑x⁴ + f∑x³ = ∑x³y
a∑x⁷ + b∑x⁶ + c∑x⁵ + d∑x⁴ + e∑x³ + f∑x² = ∑x²y
a∑x⁶ + b∑x⁵ + c∑x⁴ + d∑x³ + e∑x² + f∑x = ∑xy
a∑x⁵ + b∑x⁴ + c∑x³ + d∑x² + e∑x + fn = ∑y

There we have our six explicit simultaneous equations in six parameters; solve 'em! Use any decent math library. Or, if there's a cow peering into your Portakabin™, and all you have is Commodore Pet BASIC, take advantage of that 30KB of RAM to knock up a quick Gaussian elimination.

From memory, of course.

Wednesday, 26 August 2009

So Long PCW, and belatedly, Sub Set

It's The End Of The World

Groundbreaking and trailblazing, Personal Computer World magazine, at age 31 the UK's first and oldest, has finally ceased to be.

The August 2009 issue (published on 8th June) gave no indication that it would be the last. It even contained the usual advertisement for the "next" issue.

Lifelong contributor Guy Kewney has already provided the definitive, if typically idiosyncratic, personal obituary. But I want here to say a few words about one of the magazine's lesser known innovations.

I'm not going to claim that PCW was my lifelong companion. My friends and I were already a few years into our careers in microelectronics and computing, thanks to the National Semiconductor SC/MP prototyping board system (1974-6) among others, well before PCW first appeared on the newsstands in February 1978.

Needless to say, the "Scamp" was uniquely innovative for its time. For example, it was the first system to provide for multiprocessor architecture--up to three of these processors could share the same address and data buses. Not that we first year undergraduates could afford more than one between three of us!

Through magazines like Elektor and Practical Electronics, carefully collected and stored in custom binders, we all learned the black art of machine code programming. And while I still owe PCW's editors my greatest debt in that lowest basement of Babel, actually they gave me my first "break" by publishing articles in various other areas; articles about micros, pocket computers, and higher-level technical subjects. In 1980, I bet a friend one pint of Guinness, that I could get a line drawing of a locomotive engine published in PCW. Maybe you can guess the subsequent shape of the state diagram in my "Complex Number Calculator" for the Sharp PC-1211 pocket computer. [Update, 12 March 2011: see it here]

I Know All About Them

Thus spake Spock, when asked "…You know a great deal about computers, don't you?"

The instruction sets of these early up-to-8-bit processors, while not exactly RISC, were quite manageable. By which I mean, anyone could quickly become a master machine code programmer; able to write on a first pass, a perfectly optimized program for any given task. And like Spock, to do it directly in binary, without the need for an assembler (compilers were things the universities used on their minis and mainframes). The "secret" was simply to take the full set of opcodes for the processor, each corresponding to some ordered pair of hexadecimal digits, and construct an "instruction map" - a 16x16 chart - with the digits 0-9, A-F along each axis. This could then be memorised quite easily.

The Zilog Z80 (Tandy TRS-80, Sinclair ZX-80/81/Spectrum, Amstrad CPC-464/PCW) yielded the largest map; firstly because it used up all 256 available opcode bytes, and secondly because it used several of these as prefixes for 16-bit instructions, which obviously opened up more maps (more pages), extending the instruction set of its progenitor, the Intel 8080, e.g. to include new operations utilising its new index registers. Other favourites were the 6502 (Aim-65, Commodore Pet/Vic-20/64, BBC Micro) and of course the Motorola 6809 (in Mettoy's Dragon 32 home computer), the first to include a Multiply instruction.

PCW Sub Set

Apart from this token attempt by the 6809, none of these 8-bit processors had much out-of-the-box support for adding and subtracting numbers above 127, never mind multiplying and dividing them, or allowing fractions. Even when supplied in the form of a home computer, i.e. with interpreted BASIC in ROM, their arithmetic and other subroutines were often very far from optimal.

Personal Computer World provided a great service to the micro development community via its Sub Set series of articles, curated initially by Alan Tootill, joined later by David Barrow. Here we could all meet to set down the gold standard for 32-bit arithmetic routines, signal processing and interfacing, random number generation, development tooling, and a hundred other hotly contested library projects. Simultaneously, the Datasheet format provided a respectable standard for documenting material of this type.

I became a regular contributor, Alan and David frequent pen pals. When I eventually landed my first full-time job in the industry, it was purely 100% as a result of the work published here, and 0% thanks to my undergraduate degree.

Later I'd be tempted more than once, while en route to maybe York or London, to arrive on David's West Yorkshire doorstep with an oft-promised bottle of Talisker. As yet undelivered, I regret. Well, that would have looked like bribery, wouldn't it?

Sub Set was a viciously competitive environment, a forum for developers throughout the UK and beyond, where regular challenges were posted, and we raced to submit the fastest or shortest solution, or to improve on an earlier one. Having one byte trimmed, one clock cycle shaved, from your code - this could mean utter, instant, humiliation.

Naming Isn't Everything

Once I complained to David about his choice of a variable name. His response was the most beautiful utility routine I have ever seen for any processor. It was for the 6502, a device severely impoverished in terms both of instruction set, and of memory addressing modes. It virtually rewrote the silicon. What it did was to expose full leverage of an obscure debug mode "Break" instruction, transparently to add (1) streamlined interrupt processing, and (2) a brand new, unbelievably useful, relative address mode "Call" pseudo-instruction. Not content to name his routine BIRCH (for Break, Interrupt & Relative Call Handler), David went on to supply five-letter tree names, each somehow meaningful, for every label in that code.

Century Communications eventually published their Best of PCW: Assembler Routines for the 6502, and the companion volume for the Z80, in book form. For a little while, I knew the pleasure of getting regular royalty cheques through the post, thanks to machine code subroutines... that I'd already been paid for!

Again, thank you PCW. I'm still pleasantly surprised even today, when surfing for fruit scone recipes, to chance upon namechecks and acknowledgements in projects like MAMEs and homebrew emulators (although "Contains the John Kerr disassembler" nearly scared the shine off of me, first time I saw it).


Like type-in listings of text-only adventures, Sub Set was discontinued some years ago. Since then, machine code has actually remained surprisingly manageable on a human scale, partly due to the emergence of RISC and the ARM, P-Code, and now MSIL and the Jitter; but the quantity has changed, and all the short essential jobs have been done and documented. Much of what we did back then, is now performed in multicore cathedrals of silicon, and even embedded systems developers have any number of compilers to choose from.

For a lot of hobbyists, the demise of PCW Sub Set was one sign of maturity in the industry, similar to the emergence of the standards-defining IBM PC and its Microsoft operating system, begetter of many clones, that signalled time to move on to something more interesting. Leaving the machines, and their league of magazines, to continue a headlong rush into their demarcated business and gaming arenas. And their ultimate obsolescence.

Friday, 21 August 2009

Security Glossary

Check Back Here!

In any field there will be key terms, words and phrases, with specific technical definitions, often related to but usually distinct from their everyday applications. Here I have bundled together some of those from the security arena, along with miscellaneous other acronyms and terms used in this blog, for ease of reference from other articles.

This list will be updated periodically, so that any time you get referred back here, you can see the latest version.
  • ACL - Access Control List (commonly pronounced "ackle"). A table or data file specifying whether a user or group has access to a given resource (application, file, printer) on a computer or network; and the rights and privileges they have to it (read, write, execute, delete).
  • ASLR - Address Space Layout Randomization. Randomly arranging positions of key data areas (executable, libraries, heap, stack) in process address space. Hinders some attacks by making it more difficult to predict target addresses.
  • Asset - any valuable resource, e.g. database data, file system data, system resource.
  • Attack - any action designed to harm an asset.
  • Black Hat - The Black Hat Conference is a computer security conference, bringing together different people interested in information security.
  • BlueHat - a series of invitation-only security briefing conferences, hosted by Microsoft.
  • CIA - Confidentiality, Integrity, and Availability.
  • Cross-site scripting (XSS) - originally, injection of client-side script into pages viewed by other users; now refers to any method of code injection (ActiveX, Flash, Java, VBScript, or even pure HTML).
  • Cross-domain security - specification (e.g. enumeration, possibly involving wildcards) of the websites allowed access to a given site.
  • Countermeasure - any safeguard which addresses a threat, mitigating its risk.
  • Denial of Service (DoS) - an attack which attempts to overwhelm the target system by exhausting its resources, leaving its intended users without a service.
  • DEP - Data Execution Prevention. Prevent apps/services from executing code in non-executable memory areas (eg certain exploits which use buffer overflows to store code).
  • DSA - The Digital Signature Algorithm, a United States Federal Government standard for digital signatures.
  • ECDSA - The Elliptic Curve Digital Signature Algorithm, a widely used DSA cryptography method.
  • Exploit - see Attack.
  • Firewall - a device or a program which monitors and controls communications between computers.
  • Grant - to allow privileges or permissions to an entity.
  • Injection - the (usually unauthorised) addition of new code to an existing website, database, etc.
  • MAC - Message Authentication Code.
  • Malware - malicious software, created to exploit a vulnerability and compromise a computer, its software or data.
  • Mitigation - any strategy, technique or circumstance that reduces the threat posed by a vulnerability.
  • MSF-A+SDL - MSF is the Microsoft Solutions Framework; a set of principles, models, disciplines, concepts, and guidelines for delivering information technology solutions - including primarily applications, but also deployment, networking, and infrastructure projects. A is for Agile software development methodology. Finally for SDL, see below.
  • Permission - the ability to perform an activity on an asset.
  • Privilege - the ability to interact, at a given level of access, with an asset.
  • SDL - Security Development Lifecycle. Less generally, in this blog SDL usually refers to the Microsoft Security Development Lifecycle.
  • Spam - cooked meat, nice in sandwiches.
  • Spoof - an attack using a falsified source (e.g. an email "From" address, or a webpage) which appears to be a trusted third party.
  • STRIDE - Attack type taxonomy acronym: Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege.
  • Threat - any potential occurrence (malicious or inadvertent) that could harm an asset.
  • Vuln(s) - abbreviation of vulnerability / vulnerabilities (below).
  • Vulnerability - any weakness which makes possible a threat to an asset.
  • WTF - Wednesdays, Thursdays and Fridays.
For general use, more extensive security glossaries can be found elsewhere; the SANS site is particularly good, and of course Microsoft has one too. But for the purposes of this series, I'm intending to keep this list up to date with the subset of security related technical terms that I happen to use. Please let me know if I miss one, mkay?

Latest update: 31
st March 2011.

Thursday, 20 August 2009

What's in a Gristleizer?

In the Potting Shed...

Several weeks ago, when I finally got to hear about 1970s industrial music/visual arts group Throbbing Gristle's upcoming June '09 Glasgow show, their first ever Scottish gig since their inception 35 years ago, it was already too late for me; some other commitments prevented my being there. But then to hear after the event, that there had been some 100+ out of 1,000 tickets unsold - that's not merely injurious, that's insulting!

Not that I'd have believed either of those titbits, even had they featured weeks or months earlier in a specially extended edition of Newsnight with Jeremy Paxman. Throbbing Gristle in Glasgow? Yeah, sure. For one thing, it's well known they disbanded in 1981. Okay, there were rumours of a 2004 reunion, but only in far hidden and inaccessible corners of the Californian outback. That they would ever play Glasgow, well ... that was just mad, crazy talk.

Today I found a very small consolation on the internets. It was a wee bit like being reunited with a childhood friend, when I rediscovered: The Gristleizer!

To the uninitiated: it's a guitar effects pedal. Actually, the picture on the left is a more modern version; the original was neither photogenic, nor in fact a pedal. What I actually discovered was that a previously broken link, to a PDF of 16-year-old hero Roy Gwinn's original article in Practical Electronics, had been repaired. From a related Smashing Guitars story:

"Though the original design is over 30 years old, The Gristleizer had never seen commercial production until the summer of 2009, when Endangered Audio updated and improved the circuit. The pedal version, housed in a high-quality, heavy-duty Hammond 1590DD box, retains all the functionality of the tabletop design with the added convenience of footswitch operation..."

That first bit is not really, what you might term, a million percent true.

For a shortish time in the latter 1970s, my father's garden shed became - to his frequent annoyance - home to many analogue electronics projects. Some of these were connected with the mobile discotheque run by my friends and me, or with various bands I was variously involved in. When we couldn't afford a new sound-to-light show, or a replacement PA cabinet, out would come the hammers, saws, soldering irons.

One of my favourite projects was making highly dangerous, 240V AC to variable 3-50V DC, lab bench power supplies - built into the hollowed out cases of expired batteries. Were you to build one of these today, the HSE would not only imprison you quite promptly, but ban all related documentation.

Sounds of the Seventies

Another favourite was guitar amplification and special effects equipment. Who could resist that Practical Electronics article? "A fast decaying ramp on the VCA produces a sound like a mandolin", it enthused. "The same control into the filter gives a bubbling, which slows down into a repeated Waa-Waa", it continued seductively. "A very slow triangle into the filter can be applied to any playing including fast runs", it taunted excruciatingly. Look!-

A beautifully compact, simple, well designed and modular audio circuit, easy to follow and experiment with, as many of us did. Including myself, and as I later learned, TG's Chris Carter - though it seems he may have used the Phonosonics kit. Hey Chris, you're a lightweight! [Oh calm down. CC has always been upfront about this fact. - Ed]

I went through quite a few redesigns over the years, but the first and most important of those was the conversion to use a 14-pin Quad Op Amp chip, the National Semiconductor LM348, to replace four out of the five "741"s; namely, the LF-VCO (low frequency, voltage controlled oscillator) and the VCA/VCF (voltage controlled amplifier & filter); leaving the original preamp unmodified. This, and the expedient of attaching components to both sides of the single-sided Veroboard, allowed me to shrink the board to about 2¼" x 1". A most significant threshold! This let me build the entire miracle device, complete with its dual PP3 batteries, multiple footswitches (another vital improvement), jack sockets and potentiometers, into an upturned Golden Virginia tobacco tin!

Sorry, but you'll have to use your imagination at this point. Like Chris, Cosey, Peter and Genesis P-Orridge, I long ago suffered the loss of my last original. It would have looked a bit like this, only of course upside down, and with switches and sockets protruding, and curly cables emanating, from it. You get the idea.

Both the Quad Op Amp chip, and the newfangled 2N3819 Field Effect Transistor (which as you can plainly see from the circuit diagram, is doing most of the heavy lifting) suffered from heat problems in that confined space; several further internal revamps were needed. Failure mode invariably comprised a sudden transition from super effects pedal, John, I definitely love you! into agonizing noise generator you basterd, inevitably in the middle of someone's impassioned, alienated, and sweat sodden solo performance of a lifetime. But wrinkles were ironed, and life went on. Actually, it was no less reliable than any other electronic device in its day. Rather, it's today's contrasting super-reliability (excepting a few bad apples) that is truly astonishing in retrospect.

As for the "commercial production" aspect, well I swear I sold one of these feckers to every guitarist I met. For about £12 each, roughly cost price. But for a time, whenever I showed any musician these little upside-down hash kit box wonders, or rather when they heard what it could do, I had a sale.

Whilst I personally, and (AFAIK) every "customer", used the device exclusively as a guitar effect, TG's Chris had clearly taken serious note of Roy Gwinn's originally concluding comment: "The unit can of course be used to treat any instrument..." In fact, Chris confirms that they then used it to process "almost everything: synth, guitar, bass, violin, tapes, rhythms and of course on Genesis (P-Orridge's) voice..."

Their music was one factor, together with their pioneering use onstage of disturbing imagery, pornography, and unbelievably innovative pre-punk soundscapes (and even more pioneering usage of sampling), which led to prominence, notoriety, and in 1976 to Tory MP Nicholas Fairbairn famously branding both TG, and the related performance art group "COUM", as "Wreckers of Civilization". Fortunately for them, the label stuck, presumably contributing immeasurably as such things do, to their popular success.

"Countless acts owe them a huge genetic debt -- everyone from Einsturzende Neubauten to Skinny Puppy to NIN to Aphex Twin to Radiohead..." (Xeni Jardin, BB)

Like much else today, the analogue sound processing business has taken some hard knocks from digital tech. It's easy for any beginner now, using standard and relatively cheap kit, to knock out basically any sound that they can imagine. Want to be a virtuoso? Learn Apple Logic!

So - about Throbbing Gristle?

O well, never mind, these alternative heroes just might come back to Glasgow in another 35 years' time, just as we all brace to hit ninety. Meanwhile, we have until 27th September to experience the afterlight of that collaboration with renowned multimedia artist, film maker and dynamic sculptor (and Derek Jarman collaborator) Cerith Wyn Evans, at Glasgow's Tramway. The exhibition is billed as "A remarkable audio sculpture: 16 suspended and highly polished circular Audio Spotlight panels. TG have composed a special multi-channel soundtrack that plays through the sculpture..."

Is it any good? Well yes, of course! But then again, don't ask me. Once more, due to prior commitments...

Monday, 10 August 2009

In Which I Am Outed As A Fraud

Wag Wag Oops

I was planning to bring you Security 101: Part 3 today, wrapping up our intro to the MS SDL and looking forward to some more contemporary developments, and that's all simmering away quite nicely on that burner at the back; but this rather embarrassing (for me) little gem just cried out to be shared first.

Last week I was having some trouble trying to use Remote Desktop to get some work done from home. It simply didn't work at all. Having just switched from Virgin Media to Sky Broadband, I let my suspicions fall on my new wireless router. A more than cursory search of the forums seemed to bear this out, with many people apparently reporting a sudden cessation of RDP functionality immediately on switching to Sky.

Still, Sky for their part denied blocking anything other than a few server types. Also, more than one colleague reported experiencing no such difficulty with the Sky router.

Then our illustrious IT Department, which I would describe not so much as "continually improving" recently, more "maturing in a pubescent thunderclap", heard of my difficulty, and as we all have come to expect, solved it in a few milliseconds by pointing at the PC and incanting "Klaatu, barada, nikto". This, I am reliably informed, translates as "You have disabled, Remote Desktop, you plonker."

I hadn't, not exactly; but by some update mechanism as yet unidentified, whether Microsoft's or ours, it had indeed recently become disabled on my PC. Once re-enabled it worked perfectly, which was all very fine, and certainly embarrassing enough already, had that been the point of this sorry tale.

Fetch Me My Donkey's Ears

What was much, much worse than this, was what IT discovered when testing my newly re-enabled RDP, which could be loosely phrased as, "Did I say plonker? Not strong enough. You have never had anti-virus on this machine."


At this point IT withdrew, to let my conscience and me reflect on what we had learned here today. Obviously, I could display my loyalty and commitment to the company, by protecting this highly sensitive secret, that the appointed security researcher had been unmasked as a complete fraud. It wasn't too late to
  1. install AV,
  2. put my hands in my pockets,
  3. whistle insousiantly, and
  4. walk away,
as I was confident about not being snitched. On the other hand, the lessons seemed too good to miss out on...

I practice safe computing, for example downloading only legally purchased music tracks (well, whenever possible). I never use a disk of unknown origin, or a pen drive other than my own transfer device; click on a link in an email; give out any personally sensitive data; and so on. In short, when all other factors such as the AV checks applied by our corporate email supplier are taken into account, I like to consider myself a net (pun intended) virus / trojan threat of zero; well informed; careful; maybe even a little intelligent, no?

But just look at the security collander described above. What if someone else was to log on to my PC (perfectly legitimately, using their own network credentials), and perform a slightly risky operation on the web, assuming that the local machine was fully protected? Or, what about the flagrant disregard of my much trumpeted Security Strategy? After all I must have ignored a notification to install the correct AV software, incorrectly and unsafely assuming that was something already done for me by others, in preparing this PC for first use.

Well, at least I put that part right, and pronto.

Q: What more can I say?
A: Weeaboo?

Naming Is Everything!

Lame Name Blame Game

I recently had to revisit code that I'd last worked on several years ago. The app involves the building up and editing of data items, using a pretty graphical user interface. My job was to implement the sorting of these items into ascending or descending order, according to the values of various properties they might have.

It doesn't take much technical expertise to realise that the sorting operation can be achieved by moving items around. In fact, it's hard to think of any other way to do it. And since the app already allowed items to be moved from here to there, all I had to do was work out a sequence of moves sufficient to implement a given sorting. Elementary stuff!

So I duly implemented my custom QuickSort algorithm in its own special class, added as much wiring as needed to provide for the various user-available options, and vigorously refactored everything until no more methods could be shaken out, even using the new improved method shakeout vigour in the stripey packet.

The result was first tested, then repeatedly rewritten using my Patent Genetic Algorithm (change something random; determine whether the result is an improvement; rinse & repeat) until it worked perfectly with any data I could throw at it, including all the corner cases.

Finally, all that remained was to connect to the existing "move" logic in the app. There was just one point of interface, where my class wanted to move an item so that it ended up next door to another item. As mentioned earlier, this required method already existed in the app, so I hooked it up, committed my masterpiece and said "Test That."

You're way ahead of me, aren't you?

It's not exactly that the results were random. No, they were perfectly repeatable for any set of input data, and perfectly nonsensical. Sequentially repeated sort operations would seem to edge the results closer and closer to the correct order, like a shuffling row of housemates trying to rank themselves in order of niceness, but always giving up before ever getting it right. Clearly Halfwit was in the kitchen.

What The Feck?!

Turns out it's all in a name. The existing method which I linked into, the one that moved item so that it ended up next to another item, had a parameter labelled "AsNextSibling". Now, I had no problem with the "Sibling" part of that name; each sortable set of items had a common parent, so the items were all children of that parent, and therefore siblings of each other. No, the problem was with the "Next".

Gentle reader, imagine that I showed you a list of items including one called A, then handed you a new item B, asking you to place it in the list, next to A, "as next sibling". Where exactly would you place it? Your two options are:
  1. add it after item A, as its next sibling;
  2. push item A out to the right, then insert item B into the just vacated slot where A used to be, so that your new item becomes the previous sibling of A.
Well, I chose option 1 without giving it a second thought. Which, to be fair, must have been exactly the way in which the previous developer had opted to implement option 2!

It's easy to say, whenever there's any ambiguity about the name of a class, method, parameter or variable, more thought need to be invested in the readability of your code. As a very last resort, you should at least document the correct usage in such cases, but do so knowing that your comments will inevitably be blissfully ignored. Always try to eliminate rather than just mitigate the issues.

This was vividly borne out to me recently when another colleague asked why his usage of my code appeared to be causing so many problems. My reply involved just pointing at his screen, where once in the long ago, I had been reduced to the desperate measure of providing quite rare (for me), but full and detailed commentary. I guess he mistook my wise and precious counsel for the kind of commented-out, obsoleted material that he encounters every day! Truly, hell is other people's code...

Yet in the case above, I saw no scope for ambiguity, and still have trouble squinting enough to see it. My interpretation was obviously the correct one, dammit.

Nothing deserves more care and attention than assigning the best possible names to each entity in your design. No, not even getting the functionality right; that's far less important. You will always make mistakes anyway. At least if the intention is clear, the next guy - who will often be you, some years later! - should have a good chance of fixing them.

Is it just me, or does anyone else around here think that this particular problem affects English speakers, with their jammed-up adjectival phrases, more than say the French, with their liberally dispensed prepositions? N'est-ce pas?

Thursday, 6 August 2009

Yes: Tales From Topographic Oceans

Masterpiece. Prog fans, give it a break, buy it, enjoy it.

Warning: this article is a review of an early 1970s progressive rock album!

A poetic grand tour of life, its origins and nature, takes Jon from the shastrick scriptures, on a wonderful adventure of the imagination, yet encompassing every science from evolution - "We fled from the sea ... whole", "As we took to the air", "And we danced from the ocean", to cosmology, "Nous sommes du soleil" - as indeed we are!

In serious mood, Jon's "Dawn Of Light..." quietly heralds the masterpiece first movement, while Steve playfully prepares the main theme in the background, tension building and eventually exploding as Alan's drums cascade towards a comic anticlimax "tap" on the hi-hat; then suddenly, everyone is hitting out the signature refrain that you will never forget. As an incidental touch of genius - surely - the third section of the following theme comprises just the eight notes of an ascending major scale!

Jon soon calms down, and the team deliver a relaxed, harmonious first pass at the famous "moments" theme. By the time they arrive at the second chorus ("What happened..."), there has been just enough variation in the approach to assure you that this piece will evolve intelligently.

As indeed it does, breaking up a few seconds later into a surprising, motoring, rockalong riff reminiscent of the great Siberian Khatru, before collapsing once again into refractory scales and that beautiful refrain. When the riff appears again, it is augmented by Rick's honky-tonk treat, and leads into another section of uplifting scales and surprising key changes. Now Steve takes us into a soft section overlaid with recorders, soon accompanied by some of Alan's remarkably gentle percussion, but building with organ and wordless vocals into a climax of eventual dissonance, and Rick's synth solo.

For the last time Jon takes us back into the main body of the piece, but as the harmonies rebuild, their ambience has changed, phrases are held for longer, and the chorus of voices move down a little in the mix. Yes are about to deliver us into the most beautiful vocal wind-down, a gentle landing that's different from but curiously symmetrical with the introduction, and the only possible ending for this sumptuous, magnificent feast.

Surprise! The second movement starts immediately. Hmmph. Well it didn't do that before, in the vinyl days, thirty years ago. I had to flip over the record, and the temptation was always to play side 1 again. What a difference this makes! Now I'm giving "The Remembering" a fair hearing on a level playing field [Ouch! Metaphor pileup. - Ed], and it's just as beautiful as the foregoing. Steve sets up a nursery-rhyme carousel and harmonising voices join in. The architecture of this piece is a much more gradual build, a gentle progression, twice resolving into a mysterious woodland clearing - but the second time we get there, Steve and Alan are waiting with a surprise acoustic, medieval treat. Then - Relayer! This too sounds different with hindsight, almost like a prequel... The piece repeats this large-scale structure with another variation on the medieval, another Relayer, another forest glade, finally resolving itself with yet another Yes climax-to-calm transition (boy could they knock those out).

Third movement. Gong! Cymbals! Xylophone! A very percussive introduction precedes some very off-the-wall guitar experiments. Then after four and a half minutes of this, a start-stop theme lets Jon get a word in. Strings chug along for a bit, then mix it with the start-stop motif until a very catchy, jaunty tune lifts you and pulls you into a more mathematical area (think Gentle Giant). Percussion and guitar explore their abilities, take us into the jungle, and leave us breathless. Finally running out of nervous energy, we are deposited at Steve's acoustic strings, where we are perhaps reminded of the wonderful Mood For A Day. Jon gently talks us down once again - "Along without you..."

"Ritual" is probably almost level pegging with "The Revealing Science of God" when Yes fans choose their favourite Tales. Chris' bass has much to say here, being turned up in the mix, as the piece emphasises percussion and tribal dancing. Yet another beautifully simple theme runs through this final movement, stopping here and there for a good bit of bass dressing. Jon's triumphant "Nous sommes du soleil" had to wait in the wings for five and a half minutes this time, and he soon settles into his song - during which the anthem from movement 1 is reprised. Half way through, and a lesser climax gives way to the dark stalking bass once more. Here as at the beginning of this movement, Chris' too-short bass solo, laid over Rick's melodic rainfall patterns, somehow manages to evoke a panoramic vista as from some old Western movie. But in another one of those comic asides, it is terminated by a loud blast on a pea whistle!

Frenetic and blistering into two pure percussive / coloured noise sections (ok, drum solos) and falling madly into a remarkable resolution by Steve's sweet electric candy, this stunning piece is carried across the finish line by two verses of Jon's softly sung words - "Flying home, going home...", "We love when we play", as Steve brings out the cast of themes for their final bow, and a single note, repeated and repeated for reassurance and closure, finishes the work so very enigmatically.


So let's get a few things straight. Firstly, the piece is not based on Paramhansa Yoganada's book, as is commonly thought - that's simply where Jon discovered, in a footnote, the shastrick scriptures upon which it is built. Secondly, if this work is flawed, then these flaws are tiny, and two in number. The second verse of "The Remembering" starts out sounding much too like the first, while Steve's singing guitar grates drunkenly for a few seconds at the start and end of "The Ancient". Even mentioning these imperfections raises their profile out of all proportion. Give it a break, buy it, and enjoy it.

First published on, 24th September 2000.

Wednesday, 5 August 2009

1st Principles: Curve Fitting

Get The Trend

In this spoon-fed Internutopia, there will always be reasons to prefer working things out for yourself: personal satisfaction, and the confidence and ability to function without an umbilical cord. Even if it's just high school algebra...

A colleague recently asked me what formula is used by Excel to calculate the parameters of a trend line. Without checking the documentation, I answered that Gauss' method of least squares might be an obvious choice. "So, how do I work out the coefficients used?" Well, on this occasion, we can just use a search engine, and harvest the necessary expressions from any one of approximately half a million suitable websites. Wolfram, perhaps. Or Wikipedia. Or

It wasn't always so easy. A mere quarter of a century ago, you might have found yourself alone in a Portakabin™ sitting by the exhaust of a running aero engine - in the middle of a field full of cows! - taking readings from an array of temperature, pressure and flow instruments that you'd chained in series to the user port on a Commodore Pet computer, and trying to get the attached HP pen plotter to show the associated trend curve. Which, the local aero boffins would have assured you, was theoretically likely to take the form of a quintic, or fifth-order polynomial.

These guys were serious about their fluid mechanics. Any of them could stir a cup of hot coffee vigorously with a teaspoon at 600 rpm, without spilling a drop. When they said the data would follow a quintic curve, you could bet the house on it.

But where to start? It seems reasonable to suppose that the straight-line fit can be generalised to this case, so let's look at that first. Plot all of the n data points (x,y) on a graph, and add a straight line approximation, y = mx + c, just wherever it looks good to you. Now for each point, measure how far away it is from your line. To simplify the sums, we usually measure these discrepancies vertically rather than perpendicular to the line, so we arrive at a list of ∆y values.

If you've fitted the line at all well, then some of these deltas will be positive, some negative. We do not want that! Positive and negative errors would cancel each other out! So, we square each delta, forcing all positive; then sum them all to get something we'll call :

∑(∆y)² = ∑(mx + c - y)².

Finally, we do the mathematical equivalent of wiggling the line clockwise and anticlockwise (i.e. varying its gradient, m), while also shifting it up and down (varying its y-intercept, c), until we find the particular line that minimises , the sum of all those squared deltas. To do this, we partially differentiate first with respect to m, then again with respect to c:

∂∑/∂m = 2∑(mx + c - y)x, which equals zero when

(∑x²)m + (∑x)c = (∑xy).

∂∑/∂c = 2∑(mx + c - y), which equals zero when

(∑x)m + (n)c = (∑y).

Notice that the bracketed terms ∑x, ∑y, ∑x², ∑xy, and of course n, are all known or readily calculated from our original data. So we have two simple simultaneous equations in our trend line coefficients m and c, from which we can readily obtain our linear regression, erm, line.

And that's all there is to it. Generalising the method to any alternative mathematical model, other than the first-order linear one, is usually a simple matter of varying this last step to "Adjust all available parameters of the curve simultaneously, to find the optimum fit." Whatever the mathematical model, the chances are excellent that we will be able to differentiate it as much as we need to, obtaining one additional simultaneous equation for each extra (independently adjustable) model parameter.

Polynomials like quadratics, cubics, and quartics, are particularly straightforward. And as I happily discovered one August day in 1984, alone in a Portakabin™ on East Rogerton Farm: it works equally well for quintics!

Tuesday, 4 August 2009

Security 101: Part 2

An Introduction To The Microsoft Security Development Lifecycle (SDL)

Today we start taking a quick, top-level look at the current software development process, both used within, and recommended by, our friends at Microsoft.

In January 2002 Bill Gates, in response to customer feedback, grabbed his dilemma by the horns, and issued a back-me-or-sack-you directive known lovingly as Trustworthy Computing. Company-wide memos like this were rare. This one went into every department, as company-wide memos are wont to do, and demanded sweeping improvements in the "four pillars" on which the customer experience is based: security, privacy, reliability, and business integrity.

Trustworthy Computing encapsulated the overall process in the guiding principles labelled "S3+C", chosen to represent "Security (and privacy) by design, by default, in deployment; and Communications". When you add steps to your software development process model, to accommodate security, privacy, and the communications elements to support these, you get a security software development process model. But if you are Microsoft, then you get the industry-leading software security assurance process known to millions as The Microsoft Security Development Lifecycle, or SDL.

There is a nice colourful diagram of this entity on MSDN. Can't show it to you here, as it's ©2009 Microsoft Corporation. All rights reserved. But you might follow this link. If you did, you might also stumble across an ad for their SDL Process Template for Visual Studio Team System. I'll be looking at that in a later article, but for now, I'll just mention that it almost looks like sufficient justification, on its own, for changing over a mature development process to VSTS.

Returning to the SDL, there are identified five core phases in the lifecycle, bracketed by pre- and post-SDL requirements. The overall process that these phases cover, can be seen as starting with new developers being introduced to new technologies and new user requirements, and finishing with a product in its "maintenance" phase (incidentally, it gives this 50-year-old developer great satisfaction to see the assignations "maturity" and "continuous improvement" interchangeably applied to this phase!). Actually though, it can equally be treated as a standalone security project lifecycle, suitable for retrofit to an existing or ongoing product development.

Today I'll talk about preliminary security-related training, then go on to describe the first two phases of the lifecycle, Requirements and Design.

In these sections you will find that many and various security and privacy matters are merely glossed over, if indeed they are name checked at all. As mentioned in Part 1, Security Is Big! Some of these will be returned to in future articles, others may be less lucky...

Pre-SDL: Training

The constantly evolving nature of the threats means that frequent retraining should be a component of the development security strategy - at least annually. Threat modelling should be as natural and routine an activity as reference to design patterns, unit tests, or documentation, while itself including elements of all of these.

Secure design, coding and testing afford multiple opportunities for training. So you know about SQL Injection attacks, and the perils of weak cryptography. Your good housekeeping means that your code is virtually immune to buffer overrun attacks. But have you checked what happens when that 32-bit ID generator overflows?

Thanks again, XKCD!

Training should also cover risk assessment, as well as the methodology of security testing, and its automation.

Finally, once everything's been covered from the viewpoint of securing data against unauthorised access, it needs to be triplicated! You still have to deal with guaranteeing access to properly authorised users, and ensuring the integrity of what's stored and/or transmitted.

There are several resources, recommendations and guidelines relating to pre-SDL training, in this MSDN article.

Phase 1: Requirements - Introducing Security Guy

At the requirements stage, if the security and privacy concerns of a project warrant the use of the SDL at all, then there should be identified a team member (or members) with overall responsibility for tracking and managing the security and privacy aspects of the piece. In small teams, these responsibilities often fall to the project manager.

Acceptable levels of quality should be determined, and it is a feature of this particular SDL that the security bug bar is never lowered. Not even on release day! Security bugs are categorised and managed like any others, i.e. by severity. They also have their own, unique family of categories, complete with silly acronym: STRIDE! Basically, you should be able to classify any security bug with one or more values from this list:

Information Disclosure
Denial of Service
Elevation of Privilege

Another unique requirement of security bug management, is that information about the bug must itself be treated in a secure way, for reasons that should be obvious! So, this might mean securing your bug reporting tools, e.g. restricting permissions to team members and security advisors only.

All bugs have associated costs; privacy and security bugs can be among the costliest. Risk assessment is an indispensible requirement of security design, and may identify features of the product that need deep security review.

Phase 2: Design

The design phase is the plan building phase. Functional and design specs are now used to determine best practices, while the project's vulnerabilities and threats are identified by performing the risk analyses identified in the threat modelling of Phase 1.

Threat modelling must be completed during this design phase. How could it be otherwise? We can't have a software design, if we don't yet know what we have to defend against, and how we intend to do that!

Two recent additions to this SDL phase merit special mention.
  • The first is UAC compliance, where we ensure that our app is fully useable and runs correctly as a non-administrator.
  • The second introduces a new set of restrictions and prohibitions in connection with firewalls. Basically, don't disable or change their state; don't open a port or enable a rule without admin privileges; and don't do anything else without the user's consent, mkay?
Design change requests raised during this or subsequent phases will now have to be re-assessed from the point of view of security, to determine whether the proposed change would invalidate the current threat model, or require modifications to cover newly exposed vulnerabilities.

Next time: Implementation, Verification, Release and Response.