Wednesday, 31 March 2010

A Backup Triplet

Three Notes in Quick Succession

Did you ever read Warren Ellis's big data loss story?

My backups all got corrupted, and my backup device died. I'll fix that on Sunday, I thought, as I was under deadline pressure. Saturday evening, my main machine died in flames. Sent it off for data recovery. The guy running the data recovery shop took it in and then went off to Europe for an operation. And died on the operating table. Came back to the shop to get my machine, because no-one was answering the phone, to find it boarded up, the (mostly off-the-books, apparently) employees scattered to the four winds, and the shop stripped down to the plaster. Not a computer left in there -- not even mine.

Among other things, Ellis lost about 30,000 words of "Listener", as his new novel is provisionally titled. That's going to sting.

Delegate IT...

Ellis is a self confessed non-nerd:

No Macs, no Linux: I have a lot of Windows-specific software and function that I need to maintain. Don’t even talk to me about partitions and Windows emulators and whatever, I’m a working writer who can’t programme a VCR and I Do Not Have The Time.

So, he's probably a good candidate for cloud based backup. The target for Google Chrome OS (to choose an arbitrary example at random!) is to make Google's system files the only thing you need to have locally. Other providers are available...

On the face of it, the issue is a lot more specific, limited and circumscribed than the "Internet Operating System" that Tim O'Reilly has been describing for some time, and recently pinned down in an extended article. I think the separation is soft; companies with the size, resources and ambition of a Google or a Microsoft are constantly seeking diversification, finding success in new, unexpected areas. And almost any provider of cloud backup is well placed to supply auxiliary services by the plethora.

... or DIY

Cloud storage is normally, although not always, extremely secure and reliable. But for a variety of reasons, mostly security or privacy based - government snooping? no thanks! - many people are reluctant to see their data leave home, whatever level of RAID or secure encryption you try to flog them.

Apple pundit John Gruber, of Daring Fireball fame, recently wrote in praise of backup / recovery tools DiskWarrior, SuperDuper, and Dropbox, after a hardware meltdown left him in the enviable position of losing not one single byte of his data. It's a great article for users on other platforms to read too, given the calibre of advice like this:

Hard drives are fragile. Read as much as you can bear to about how they work, how incredibly precisely they must operate in order to cram so many bits onto such small disks. It’s a miracle to me that they work at all. Every hard drive in the world will eventually fail. Assume that yours are all on the cusp of failure at all times. It’s good to be spooked about how long your hard drives will last.

Backup. It never gets old.
HDD photo:

Monday, 29 March 2010

Sony FB

My Beautiful Yellow Dog Linux 6.1 Desktop

Updated & extended; most, though not all, foul language excised. My correspondence with retailer John Lewis has been added to the Comments.

When I read this post on Slashdot yesterday...

The next system software update for the PlayStation 3 (PS3) system will be released on April 1, 2010 (JST), and will disable the 'Install Other OS' feature that was available on the PS3 systems prior to the current slimmer models, launched in September 2009. This feature enabled users to install an operating system, but due to security concerns, Sony Computer Entertainment will remove the functionality through the 3.21 system software update.

... I just clocked the date and went back to sleep. But not without some stress. I was perfectly aware, after all, of the very real threat posed to Sony's business model, by the recent work of iPhone hacker George "GeoHot" Hotz, in circumventing the Playstation 3's hypervisor, gaining full memory space, direct-to-kernel and device driver ring 0 access, and ultimately promising the RSX GPU.

More than that, I was now looking forward to DMA pipeline programming just the curmudgeonly but venerable 150 GFlops Cell processor myself (today's Core i7 975 offers just 111 GFlops by comparison), even though aware of corporate eyes watching with poisonous contempt, laying their plans to move against their own honest customers.

With all that DRM breaking, should I be worried about this Slashdot post, claiming that on April Fools' day, Sony will remotely shut down my entire home Without my consent? Without more than half a week's notice? Well, I also had in my other pyjama pocket, this recent (month old) personal guarantee from Sony Computer Entertainment America's principal software engineer, Geoffrey Levand:

Please be assured that SCE is committed to continue the support for previously sold models that have the "Install Other OS" feature and that this feature will not be disabled in future firmware releases.

In the event, sure enough, I awoke today to a chorus (thanks, Google) of more than six million voices, all confirming Linux's imminent removal from my PS3; and the furious backpedalling rattle in my head, of Levand's "The text above was provided to me by SCE management"...

The Rig, Posing

Just in case you don't believe that I run my evil multinational business empire on PS3 Linux: here is our entertainment centre, oblique, home office.

The speakers are normally a room apart, but they'd recently been moved in close together for a photo shoot connected with an unrelated article on audio standards. The main thing you will observe, I hope, is my meticulously maintained PS3 Linux desktop, where as you can see I'm presently engaged in patient composition of my measured response to the latest Sony news.

What we are finding more and more, in this era of devices infected with strong DRM and phone-home functionality, is that not only do we no longer control our own devices to any reasonable level, but that we are also subjected to the summary cancellation of sundry features after the sales transaction has been effected.

How many customers of Amazon's Kindle ebook reader were convinced to buy because of the automatic audio reader functionality? In the blind and partially sighted community, I suspect a significant proportion. Yet Amazon allowed itself to be blackmailed by the disingenuous and thoroughly, hatefully stupid Authors' Guild, to redact this feature after the contract of sale.

Caveat emptor. If you deliberately buy DRM, if you invest in devices that are defective by design, then like me, you're just a mark - you deserve all the pain that you get, and more, because your meek acceptance of this crap brings it down, inevitably, on everyone else.

By Grabthar's Hammer, by the Suns of Warvan, I Shall be Avenged!

This time I really mean it. I will never buy another Sony product of any kind. Not even a CD, a game, a movie DVD or a Blu Ray.

Or at least, I'll do my level best not to. But I'll eventually fail. After all, if I could really forswear all future transactions with the Sony Playstation Network, deny myself every future Sony game, boycott all mandatory Blu Ray updates and all BD Live content, and so on... then I'd be able to yank out the devil machine's ethernet wire, dodge the update, and keep using my Linux solution.

However that would reduce my plastic toy to just a Linux box, and a very mediocre, deliberately impaired one at that. Remember, we gave Sony our money, on the written understanding and agreement, that this technological DRMfest would also serve as an up-to-the-minute games console and media centre.

As one commenter put it: "It's like they sold you a table, then cut off two legs." Yes, that's it, exactly. This is evil of the kind for which Sony is justly famous. Once more (sigh), we must get the lawyers involved.

Oh, yes. There will be lawyers.

Wednesday, 24 March 2010

The Conficker Business Model

Dark Cloud

When the Cloud Connect Convention, "the only event which brings together the entire cloud computing ecosystem", was held at California's Santa Clara Centre on March 15-18, exhibitor Neustar's Senior VP & Technologist Rodney Joffe (pic: © 2010 by techweb) caused a stir by claiming that the cloud is "mostly dark"; which is to say, largely controlled by criminals.

The example he used to establish this was the Conficker worm network, comprising worldwide some 6.4 million zombified computers with 18 million CPUs, and a total bandwidth of 28TB/sec. This vast dark net, he claimed in his presentation Cloud Computing for Criminals, satisfies any useful working definition of a cloud service provider, offering a choice of operating system, bandwidth, etc., and providing services such as mass distribution of unsolicited emails, denial-of-service attacks, and data snooping (exfiltration via covert channels), available for rent anywhere in the world. On this last point, Joffe clarifies, it has infected 230 out of the total 260 existing top level domains.

Compared to the relative "startup" offerings from the likes of Amazon, Google, and new kid Microsoft Azure, the larger footprint Conficker net has been running much longer (since 1998), continually commanding unlimited new resources, spreading its worm far and wide, illegally taking over more and more computers. "And there are no costs. And there are no moral, ethical or legal constraints", said Joffe.

# PCs
# CPUs

Obviously that's because the villains steal their computing and communication capacity, as well as their data, from others! So Joffe wasn't advocating this as a business strategy for companies considering how to run their own IT systems, or considering signing up for, or even providing, cloud services. But his presentation did make some good security points. Botnets such as Conficker will repay diligent study. You must assume that some day, you will become a target for them - "they're great learners" says Joffe - and protect your own infrastructure, and your applications, appropriately. And when you do subscribe to a cloud computing service, remember that your provider can be a security resource, monitoring your general patterns of behaviour, and watching for anything abnormal that could indicate you've been compromised, what they call a "black cloud".

Conficker has been comparatively quiet recently, partly perhaps because of the $250,000 reward offered by Microsoft in February 2009, for any information leading to the arrest and conviction of those particular malware goons. However their most recent really big attack, when Conficker was rented by the Waledac worm perps for spamming duties, actually postdated that, in April 2009. And then there was the Greater Manchester Police shutdown event just last month (update: end of January, actually). Conficker is still alive, still an active threat. And of course, there are many others using the same business model.

Saturday, 20 March 2010

Clapper Reel

Glourious Geraldine

Part of Quentin Tarantino's camera and electrical crew since Jackie Brown in 1997, Geraldine Brezca and her clapperboard have now achieved their very own internet stardom, thanks to her unbounded improvisational approach to scene sequence numbering. How on earth does anyone manage to get any serious acting done, when they're corpsing at the start of every scene!

Violet Blue's writeup at Laughing Squid has the background. When the above link dies, "Camera Angel" will still be available as an extra on the Inglourious Basterds Blu Ray release.

Thursday, 18 March 2010

Quantum Regex

The Go To Guy

Hardly a day goes by without someone asking me a question about regular expressions. Scale This! Chris is one such customer, although not one of my regulars, so to speak. Recently, Chris has also started asking me to explain, in layman's terms, such things as special and general relativity, and quantum mechanics. Why me?! I'm certainly no relativity guru, and could just as easily imagine asking him, or almost anyone else for that mass-energy, to explain it just once more to me!

But on the subject of quantum mechanics, I guess the day is approaching, when those of us presently engaged in high level control of electron flow (software writers, in other words) will have to get with the quantum program, or get a new career in retail greeting. With that in mind, I've just finished reading Quantum Computation and Quantum Information by Michael A. Nielsen (University of Queensland) and Isaac L. Chuang (IBM / Stanford University). Since buying this book it's taken me six years to plow through its 675 pages, but I think in this field at least, I might at last be suitably qualified to discuss just the very basics.

Warning! Metaphorical trainwreckage ahead!

This coming apocalypse will be the greatest ever sea change in the history of practical computing. One thing that might help us weather the storm, would be a source of analogies and comparisons to known references. It was a total blast recently to find a beautiful example of just such an analogy, hiding in plain sight, in the formal language subject of regular expression parsing.

All clear!

The Go Guy

Russ Cox, one of the people behind Google's new programming language Go - and apparently played here by the famous mathematical genius Will Hunting - has just this week completed the third article in his (hopefully!) three part series on the history, theory, and programming practice of regular expressions. Parts one and two have been available since Jan 2007 and Dec 2009 respectively; and inevitably, the sudden and unexpected appearance of part three has forced me to revisit those earlier articles, to refresh the old volatile storage units.

So here's the deal. You've probably heard that unlike classical "bits", each of which is at any given time in either its '0' state or its '1' state, the unit of quantum computation, the "qubit", can somehow be in more than one state at a time. It's said to be in a superposition of states, which is a kind of blurry combination of 0 and 1. It's only when we make a measurement, aka an observation, also what we may now begin calling a computation, that the quantum state or wave function "collapses" into either 0 or 1.

How does this help us compute stuff? Well, it all has to do with the preparation of the initial state of the qubit. I'll be going into that in quite a lot of detail in another article soon, but for now, just know that we prepare a superposition state of qubits, then later we read the evolved state.

So, what could all of this possibly have to do with regular expressions? The connecting tissue is of course this idea of state.

Going Back

Regular expression parsers are best modelled by finite automata: machines with states. Sometimes these are deterministic; given the current state and the next input character, we can always determine uniquely the new state of the machine. Sometimes they are not. It's these non-deterministic finite automata, or NFAs, that interest us here.

Russ's first example uses the pattern A(BB)+A, which matches any input string containing an A, followed by one or more pairs of Bs, then finally terminated by another A. This machine is deterministic; as we scan the input string, say ABBBBA, one character at a time, we move from our starting state, into a found A state, then a found AB state (i.e., found an A followed by an odd number of Bs), then found ABB (an even number of Bs), then back to found AB, forward again to found ABB, then at last upon reading the terminating A from the input string, we end up in a final success - match found state.

Whenever we find that the next state is not uniquely determined by the combination of the current state and the next input character, we can place markers in the pattern and the input string, make an arbitrary selection of the new state, and proceed just as before to see if we reach a match. If we fail, then we can use these markers to backtrack to the earlier decision point, make an alternative choice of new state, and retry. Eventually we'll find a match, or run out of input, or else we'll have tried every available route through the labyrinth, without reaching that final success - match found state.

Go Faster!

From a computational viewpoint, it's this backtracking that's the problem. It can lead to your regex matching code taking an exponential amount of time to decide match or no match.

In 1968 Ken Thompson (of "Thompson and Ritchie" Unix fame), in his CTSS text editor QED, introduced programmers to a superior theoretical approach. And when I say superior, well, just look at Russ's graphical comparison of execution time against x, where the test pattern (superscripts representing repetition) is A?xAx, and the input string is Ax. For the absolute avoidance of doubt: when x=3, the pattern is A?A?A?AAA, and the input string is AAA.

The upper, red curve represents the worst case performance of a backtracking algorithm, of the type which is still widely used today, for example in Java, Perl, PHP, Python, and Ruby. The lower, blue curve represents Ken Thompson's Nondeterministic Finite Automata. And why didn't Russ plot them on the same graph? Look at the vertical axes. One is in seconds, the other microseconds.

So that's settled then, we should all be using Thompson NFA in our Regex implementations. But just before we do, exactly what is its secret anyway? How does it avoid the exponential explosion?

Go Quantum

The secret is to do away with backtracking entirely. Whenever we come to a fork in the road, and we can't guess which turn may lead to a successful match, we take all available turns simultaneously. In other words, we choose the new state to be the set of all available, possible new states. Then the simple, plodding algorithm proceeds as before, but now effectively following multiple threads at once. What matters is simply whether any one of these threads reaches the final success - match found state, or whether we reach a total mismatch blockage, or run out of input.

Multiple new states may coalesce back into one, split again, and so on. But at any given point, the set of current states represents everywhere in state machine space that we could have legally reached, given the input so far. Exactly how we got there is unimportant; we only care that there was a way. So we're not actually following a bunch of threads. The threads have dissolved. All we ever have is that set of current states.

And then, if one of these states does eventually transition to the final success - match found state, the bubble bursts; the wave function collapses; an experimental determination is made.

Go On! You're Having A Laugh!

Of course, this is nothing more than an analogy, designed to help us get our heads around one particular aspect of quantum computation. As soon as you begin to push it, it breaks down. More generally, all attempts to simulate the operation of the quantum realm using classical systems are doomed to failure and disappointment, as I'll be recounting soon in the forthcoming review of the forementioned 10 year old book that took me 6 years to read (note: if you change "quantum" to "parallel", then the new analogy holds much more water). But as an adjunct to a layman's introduction to the spooky realm, I hope that like me, you will find it quite apposite.

If you're particularly interested in his (beautiful!) coverage of regular expressions, here are the links to Russ's entire series:
  1. Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...)
  2. Regular Expression Matching: the Virtual Machine Approach
  3. Regular Expression Matching in the Wild
Only the first third of the first article is needed to illustrate the preceding ideas. So like I say, if you think you could benefit from another diagram or two, click through right now.

Wednesday, 17 March 2010

Security Digest #7

Onward March

An interesting and varied time in the security arena. While the recent highlight for me has to be the EoP card game, which I hope will soon be extended to the EoP dungeon and maybe EoP Assassin on the 360, there are undeniably other developments of equal merit afoot...

The State of App Security

Our first report from 2010's American RSA Conference in San Francisco came from Kelly Jackson Higgins, and contained a couple of shockers regarding the current state of application security in our industry.

The big headline: almost 60% of all applications tested during the past year by Veracode, the app security company, failed in their first round of testing to achieve a successful rating. Remember, this is software written by companies with at least enough security focus to submit their output for independent testing.

Also with a bullet: almost 90% of "internally developed apps" contained vulnerabilities in the SANS Top 25 and OWASP Top 10 lists of most common programming errors. XSS and SQL Injection still rule.

On the bright side, open-source is less risky than conventional wisdom suggests, and typically takes a lot less time (~30 days) than commercial or internal code to fix.

Veracode's full report is here.

Fortify Assures

Also at RSA USA 2010, software security assurance consultants Fortify (SDL page) published their new white paper, whose title says it all: Optimizing the Microsoft SDL for Secure Development: Fortify Solutions to strengthen and streamline a Microsoft SDL Implementation.

Many expert players are now taking this approach, enumerating the correspondence between their own inhouse offerings, and the Simplified SDL Implementation. And that's no bad thing; apart from serving the marketing requirements of both Microsoft and their SDL Pro Network Tools members, this focus has also given the security community a new vocabulary, and a readily-grasped framework, in which to express and process its concerns.

In other words, sometimes we do have to read blatant multipage advertisements like these!

Office 2007 and IE8 Tell Their SDL Tales

Microsoft apps take a lot of stick, don't they? You'd hardly know from looking at the popular press, that their output is far and away some of the very best, in terms of quality of security and privacy, available in any market - and not just the commercial, closed-source desktop money spinners. The security experts who shepherded two of these products, Office 2007 and IE8, through SDL-based development and other security processes, recently broke radio silence in order to elucidate the rest of us on their experience of using SDL throughout formative development.

Now the SDL has been in use for six years at Microsoft, improving software security very effectively; these papers will be useful references as we start to consider the implementation of the SDL and Agile-SDL in our own software development lifecycle.

Office 2007 was the first Office release based upon the SDL process. The related paper summarizes how the SDL process, and additional security work, dramatically improved the security of the software. Read "How the Security Development Lifecycle Helped Improve the Security of the 2007 Microsoft Office System" here, and "Internet Explorer 8 and the Security Development Lifecycle" here (both ~400KB docx).

Client and Cloud Security

Lastly this month, a couple of Silverlight embeddedments, for your security edification.

Get Microsoft Silverlight

Steve Lipner, Senior Director of Security Engineering Strategy, Microsoft Trustworthy Computing Group, discusses how the SDL can help improve client and cloud security.

SDL for Agile

Get Microsoft Silverlight

Bryan Sullivan, Senior Security Program Manager, Microsoft SDL Team, talks about the SDL for Agile addendum, asking: does the widely used Agile development methodology produce secure deliverables?

That's all for today. Mine's a Guinness.

Sunday, 14 March 2010

Post Progressive Post

North Atlantic Oscillation

Are two guys from Edinburgh and one from Glasgow. Update (March 24): Grappling Hooks is Zane Lowe's record of the week (BBC Radio One)! Review here.

They supported Porcupine Tree on a few dates last year, and were in turn well received and supported by the PT audience. Two tracks from their new (debut, out March 22) album "Grappling Hooks", and an interview, feature in the first Kscope podcast.

Which also has two tracks and two interviews from PT's Steven Wilson, speaking about the revival, post Radiohead's OK Computer, of "ambition" music (the original meaning of the term "progressive", as coined by the technically literate), and also about the film version of his Insurgentes solo album.

The Pineapple Thief's 10-year retrospective 3000 Days, and Anathema, also feature. In all, 40 minutes of new decade radio that should cure you of that 6 Music sulk.

Wednesday, 10 March 2010

More Big Numbers

That's a Lotta Watts

And speaking of the sun's total energy output, as we were here, I forgot to explain the quiet chuckle which you might have heard just then. If you were in my house, that is, during the first programme of the tremendous new TV series Wonders of the Solar System, presented by ex-D:Ream keyboardist and the BBC's new go-to guy for physics, Professor Brian Cox.

This would be better told as a cartoon, ideally; but without the required skills, all I can offer is the following background.

There's a scene in the great 1999 Star Trek spoof movie Galaxy Quest set at a show convention, where the announcer, introducing Tim Allen's character, the main star of the TV series, builds the tension by simulating an echo using just his voice:

And finally, my fellow Questerians, the brave Commander of the NSEA Protector: Peter, Peter, Peter, Peter, Quincy, Quincy, Quincy, Quincy, Taggart, Taggart, Taggart, Taggart!

Now it turns out that Professor Brian didn't have the benefit of my previous post when making his BBC programme, so when it came time to divulge his big powerful number, the phrase point three hellawatts probably didn't occur to him. Instead, knowing how badly such numbers can confound the layman, he went with:

Three hundred million million million million Watts.

Which you know, kinda made it sound like he was just saying "300MW", but in a really ham dramatic way.

I know, but it sounded a lot funnier in my head...

Sunday, 7 March 2010

Hella Big - Helio Small

Stop Saying That!

There's a proposal, currently before the International Committee for Weights and Measures, to add the prefix "hella" to the internationally recognised system of powers of ten. As is standard issue today, the motion comes fully equipped with its own official petition, in the guise of a Facebook Group, currently standing at 51,598 signatures. Not a hella big number, when you consider they're trying to convince the notoriously conservative Comité international des poids et mesures, to adopt into their sanctum sanctorum, an arbitrary particle of borderline-offensive slang, originating in the Hunters Point neighbourhood of San Francisco.

But the idea's supporters, including University of California physics student Austin Sendek who started the campaign, reckon it would be a good way to honour the state's impressive record of scientific research. It would be the first such change since 1991, when "zetta" and "yotta" were added to represent respectively the 21st and 24th powers of 10 ("hella" is proposed to stand for the 27th).

In our industry of information technology, we have yet to see the introduction of the corresponding powers of 2 in common use. But now that the creaky old engines have finally begun their increasingly steep ascent from our accustomed 32-bit plateau, where we have been swinging happily in hammocks since Intel treated us, in 1985, to the 80386 processor, and the end of the segmented memory management hell we had thought would never end, it won't be so much longer before we are routinely allocating zettabytes and yottabytes in the cloud. Can the hellabyte be hella far behind?

Hello World

One corollary caveat I'd like to raise is the issue of inverse, or negative, powers of 10. These conventionally end in "-o", and with the changes introduced in 1991, have begun to mirror the positive powers; hence, "zepto" (10-21), "yocto" (10-24). Presumably the "hella" proposal, properly formulated (I haven't read it, that would spoil my punchline), should contain a similar provision such as "helio", or maybe "hello", for 10-27?

In which case, if the sun's output is 0.3 hellawatts, then a Watt must be 3.3 heliosuns.

Or: if the world's mass is 6 hellagrams, then six grams make a helloworld.

Thank you, thank you very much. I'm here all week.

Wednesday, 3 March 2010

Elevation of Privilege

Threat Modeling through the Medium of Playing Cards

This one gets 10 for effort! At RSA in San Francisco this week, Microsoft released the new threat modeling card game, Elevation of Privilege.

Designed for 3-6 players by the SDL's Adam Shostack, who writes about it here, the game comprises a deck of playing cards and a score sheet. To play, you'll also need an initial (preferably, data flow centric) diagram of a system that you're trying to implement in a security- and privacy-preserving fashion.


There are 74 cards in the deck, divided into six "suits" - one for each threat classification in STRIDE: Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege (most suits run 2,3,...,J,Q,K,A but there is no 2 of Tampering, and the EoP "trump" suit starts at 5, hence 74 cards rather than 78).

All cards are dealt out, and the player holding the 3 of Tampering starts the game by showing this card, and explaining how the threat on the card ("An attacker can take advantage of your custom key exchange or integrity control which you built instead of using standard crypto") might apply to the developing system. A credible threat (one you'd file a bug for) gets a point, and is recorded on the score sheet.

Play proceeds thus clockwise, in suit where possible, until everyone has played one card, and the highest lead or trump (EoP) card played wins that hand. The winning player starts the next hand, and may do so with any card. When all hands are played, the winner gets to choose who will log all the bugs on the score sheet!


Elevation of Privilege is released under a Creative Commons Attribution license, meaning you can share, adapt and remix it as you like. The full deck of card images, including instructions, strategy cards (yay! flowcharts!) and threat summaries, can be downloaded (6MB PDF) here, and the score sheet (356K PDF) here. Finally there is a video of Adam explaining the game, and also of people playing it, on the launch page.

Apparently Microsoft are actually giving out real, physical card decks right now, or will be when the sun comes back up, at RSA in San Francisco. This raises two questions for me. Will they be doing the same at RSA Europe, in London's Hilton Metropole this October 12-14? And if so, could my security blog qualify me as my employer's card-carrying delegate? ;)

Monday, 1 March 2010

Tweets - February 2010