Simple Regex #5: Named Groups

Regex.Result()

Almost every Regex question landing on my desk has the potential to get its own blog post. This month's candidate is almost straightforward enough to be painless. Almost, because well, there's always some pain with Regex! Colleague 7 wanted to know how to extract coordinate data tidily from a string...

Regex question for you if you don't mind:

@1,X=123@1,Y=456

Looking to pick the x and y coordinates out of that string and return a point. I can obviously get the bits separately. But that would be too easy!

(?<=X=)[0-9]+
and
(?<=Y=)[0-9]+

Should I use named capture groups?

Well, personally I would, because the outcome is (1) marginally more readable than indexed groups, in my opinion; and (2) more robust, when subsequently you have to extend the pattern to incorporate further groups. Relabeling index-based groups is a nightmare!

Notice that the above suggested digit filters are using zero-width positive look-behind assertions, (?<= ), which were covered in the previous article in this series. These are just looking for an X or a Y, followed by an equal sign, and then the string of one or more digits which we wish to extract. Let's make these constants in our code, so it's easier to concentrate on what's around them (I've replaced the numeral set [0-9] with the digit class \d, another personal preference):

const string x = @"(?<=X=)\d+";
const string y = @"(?<=Y=)\d+";

Without named groups, we would simply incorporate x and y into a grouping pattern with an intervening wild string:

const string pattern = "(" + x + ").*(" + y + ")";
var input = "@1,X=123@1,Y=456";
var match = Regex.Match(input, pattern);
if (match.Success)
 return match.Result("($1,$2)"); // Output: (123,456)

Two changes are required to convert to named groups. First the names have to be applied. This involves adding a ? at the start of each group, followed by its name in either angle brackets, or as here, single quotes:

const string pattern = "(?'X'" + x + ").*(?'Y'" + y + ")";

Then after the match succeeds, these names, this time enclosed in curly braces, can be used to extract the relevant matched values:

if (match.Success)
 return match.Result("(${X},${Y})"); // Output: (123,456)

Follow Up Questions

Any reason you didn't use string.Format to define the regex?
This is what I ended up with: 

const string xCoordGroupName = "XCOORD";
const string yCoordGroupName = "YCOORD";
string pattern = string.Format("X=(?<{0}>[0-9]+).*Y=(?<{1}>[0-9]+)", xCoordGroupName, yCoordGroupName);
var match = Regex.Match(coords, pattern);

Readability by any chance?

Try changing that to "const string pattern...".
The + string concatenation operator is not inefficient at compile time.
Also, you often get a lot of {}s in your patterns, and Format doesn't like that.

More Than One Type of Hash

Oh No Not Again

Why does it seem that we still continue to read about millions of passwords / user credentials being stolen on such a regular basis, almost a quarter century after that famous and oft-cited Unix password security study? This past week has seen three separate large scale breach scares at LinkedIn, eHarmony and Last.fm. Users are being told to change their passwords immediately.

But that only protects the service provider against intrusion and accusations of laxity. On its own it does nothing to protect the user, whose new password can presumably be obtained just as readily, and whose old one may well be tied to more sensitive accounts e.g. at their bank or on line shopping stores. And more specifically in cases such as these, why is there so much panic anyway over the theft of password hashes, i.e. the encrypted versions of the users' passwords, rather than the passwords themselves? Aren't these hashes supposed to be designed for safe transmission in the clear?

Monkey Wrench

Yes and no. And mostly, no. The crucial point here is the distinction between a cryptographic hash and a password hash. The former is designed to be applied to content. As such, it carries a core requirement of fast decryption. Lightning fast in fact, as in: every IP packet arriving at a node must be decrypted quickly and on-the-fly, so as to generate no discernible latency to router traffic. Password hashes start from the exactly opposite requirement: maximal difficulty of decryption.

It matters little to a typical website user if their daily log in takes a full tenth of a second to authenticate. It matters a lot more to an attacker, who must try billions or trillions of alternatives to achieve a single successful break in. So it's not just a case of choosing between MD5 (feeble), SHA-1 (better, but still broken) and SHA-512/384 (no known collision attacks), or of adding a little salt. To arrive at a secure password hash, typically you would first select your atomic encryption algorithm, and then apply it a few thousand times.

Must Try Harder

So why do most providers use a cryptographic hash to do the job of a password hash? Most of the time this is allowed to happen because the developers in charge of a site's security are general practitioners, rather than security specialists. To them, any tool with encryption in its name is as good as any other.

This is a general problem of a lack of knowledge in the field. Developers have been successfully dissuaded from building their own cryptographic systems, being taught instead to rely on proven off-the-shelf solutions. And that's undeniably a great deal of progress, given the potential scope for devastating errors. Now, this community needs also to learn that different kinds of cryptographic solutions are available, for different jobs.

For a more extensive treatment of this subject, see Brian Krebs's recent interview with Matasano Security researcher Thomas H. Ptacek: http://krebsonsecurity.com/2012/06/how-companies-can-beef-up-password-security/.

Implied Consent (Cookies)

What Cookies are set by this Site?

This blog is a web site, within the meaning of the EU cookie law. It uses cookies. As such, it is governed by regulations set forth by the Information Commissioner's Office. What cookies are used here, and why?

Google Analytics

...is a web service provided by Google, Inc. Google Analytics sets a cookie in order to evaluate how visitors find and use the web site. This feature does not collect any personal information about you. I occasionally browse the reports generated by Google Analytics in order to to evaluate just how devastatingly unpopular my blog is.

Cookies from the Blog

...which resides on the Google-owned blogger.com platform. Blogger.com sets various cookies related to that service. I don't control the dissemination of the cookies set by the blogger.com platform, except for the Google Analytics cookies explained above.

Dog Biscuit's Trading Pages

If you visit my archived Yes & Led Zeppelin bootleg trading site, some of your search preferences will be saved by cookie for your next visit. The information saved in this cookie never leaves your browser. In fact that entire site comprises nothing but static HTML pages and a bunch of exclusively client-side JavaScript; there is no server component (yes, the entire database is in that script).

I am certainly not using cookies to gather information about your browsing habits or to push advertisements at you. If you have reason to think you picked up a tracking cookie here, it might mean I've been hacked; please contact me, for example by leaving a comment on a post - anonymously if you prefer.

This has been a public service announcement made necessary by a directive of the European Commission, confusing a worthy goal (securing web users' privacy) with a particular technology (cookies).

Caring and Sharing

Who knows what possessed my friend Scale This! to seize on this particular tweet of mine, immortalising it through the medium of ceramic mug?

And What Is Truth?

In the perceived software development wars between academic ideals and commercial pressures, the first casualty, according to the ingenious definition supplied by Alfred Tarski, is unlimited satisfaction.

My first CS tutor at university, in a 1976 introductory lecture, first stressed the need to make every program solve "in some sense" the maximum possible set of problems, before going on to say Hello World in FORTRAN. Contrast that with the philosophy behind some of today's popular industrial methodologies - Agile, Test and Behaviour Driven Development - where the stated goal is to write just the minimum amount of code satisfying strictly delineated requirements.

The trouble with the lean and sparse approach is just that, assuming we know what we're doing, we do tend to design complicated systems in a modular way, ideally using plug-in replaceable components. These components need to have readily discoverable characteristics, specifications, and interfaces; they should not be capable of surprising us. In an audit scheduling application, a control for selecting a date should never be constrained to use only future dates, on the justification that audits can't be scheduled for the past (and support for past dates would require extra coding effort in the control implementation).

Evolution

Finding the happy medium can involve a certain degree of oscillation. In the early days of our company, I and others championed the idea of a Code Gallery, holding chunks of reusable Delphi; for example, utilities for date, string and other data types, or controls such as hierarchical combo boxes. For various reasons - including a span of years spent developing just a single app! - there was never a great deal in that gallery. Most of what there was, was never in fact shared with a second project.

Source control arrived, and one fine day when my back was turned, the Gallery disappeared! Absorbed whole into the repository of our flagship product. It was impossible to protest, as by then the Gallery had come to be seen as merely a mechanism allowing code sharing across projects, and this clearly wasn't happening. Personally I thought the reduced visibility of Gallery code, which was nonetheless applied across multiple disparate modules of that single app, to be an unfortunate mistake, effectively precluding any notion of testing other than the black-box kind. A more prominent Gallery, I felt, might have encouraged more unit testing, more component-level integration testing, and eventually perhaps (step 2: "?"), better modularity and re-usability (Profit!).

Departments, portfolios and personnel all change. Today we have an internal review group tasked to investigate matters like these. One new developer in particular has done much, using not only his ridiculously popular and now-famous dynamic slide shows, but more importantly his own experience of design methods, code review, sharing and reuse, to begin once again promoting awareness of our roles as both producers and consumers of each other's output. This is a good time for our department!

Meeting Music Heroes

One Up

Pen pal Murray Easton recently posted his recollection of buying Gary Manny "Mani" Mounfield (Stone Roses, Primal Scream) a lunchtime pint in 1999. It's a good read, definitely getting across that joy of meeting one of your true life heroes - who turns out not only to be a good guy, but in Murray's words, to exceed your expectations.

Now, I'm never going to try to one-up Murray; he does run his own music management company, after all! But that read prompted me to copy into this blog, from various other places, descriptions of a few times when I've managed to meet up with my own personal musical heroes, and more particularly, when I've bought them a drink...

Bitter Medicine




Name: John Fiddler
Band: Medicine Head
Venue: Strathclyde University Students' Union, Glasgow, 1976
Drink: Pint of Bitter

Before the gig I found a well-moustachioed John sitting entirely unmolested in the Beer Bar, chatting with his bandmate Peter Hope-Evans (Medicine Head were a duo at this time), and nursing a pint of bitter. Uncharacteristically un-starstruck, and yes I may have had some Dutch or other national courage, I casually asked if they'd like another. John accepted, Peter declined. Assuming they were settling pre-performance nerves, I naturally and respectfully left them alone just as quickly as I'd arrived.

This was an odd gig. Despite the fact Medicine Head were universally known for the quite slow, blues-influenced gentle rock and sway and Jew's harp of their singles, John kept insisting on telling everyone to get up and "dance like an idiot, go crazy!" Two girls at the front and centre of the performance area (there being no stage) duly obliged all night, but it seemed like everybody else just wanted to continue sitting on the floor, drinking their beer, smiling broadly, and smoking something with quite a sweet perfume. "Medicine Head" indeed.

Orange L




Name: Steve Hillage
Band: (ex-Gong)
Venue: Strathclyde University Students' Union, Glasgow, 1977
Drink: Fresh Orange Juice

Another somewhat unremarkable encounter, though at the time quite thrilling indeed. Though the air was every bit as fragrant, I remember the crowd being quite a lot more active than the Medicine Head massive, and doing a lot of their dancing horizontally, under the discarded coats at the back of the room.

Hurdy Gurdy Glissando was a wee bit spectacular in the small room setting. Some of us got the chance to tell Steve so afterwards, over some fruit juices.

Yes - Sorry About the Shirt

Alan White!

Name: Steve Howe, Chris Squire, Rick Wakeman, Alan White
Band: Yes
Venue: Playhouse Theatre, Edinburgh, 2003
Drink: Brandy (for Chris, served in a pint of beer)

Rick & me!

A great concert, better than YesSymphonic according to my wife because Rick was there! Followed by a remarkable after-show gathering at the Edinburgh Sheraton Hotel, organised by Brian Neeson for the Scottish Yes Network. Jon appeared briefly at the entrance to the function room, but didn't come in; I think he may still have been suffering some back pain after the notorious incident of The Ladder And The Fairy Lights. Could only happen to Jon.

Our favourite bass guitarist was first at the bar, followed soon by Alan, Rick, and Steve. All were more than happy to talk to the fans, pose for photos, autograph programs and so on.  Rick was his usual convivial self. Alan is the nicest chap on the planet. Steve joined in just as much as anyone, though for a somewhat shorter time. And when we offered Chris a drink, he studied the gantry for a minute, selected a good brandy, then promptly poured it into his pint! Ah, the hectic lifestyles of those crazy rock'n'roll kids...

(From http://jmkerr.com/db/Yes_2003.htm)

The Two That Got Away

Also in 2003 - I've waited almost a decade to publish this, out of respect for my heroes' privacy - Linda and I took her mum with us on a caravan holiday in Embo. And just like this April, we all took a day trip from the east coast to the west, and Ullapool, augmented as then by the obligatory Achiltibuie scenic route. Having spent a terrific afternoon on the sunny beach there, we were driving along the single track road to the village when I got distracted by a little green car, a left hand driven Citroën (or was it a right hand Morris Minor?) speeding towards us. As we stopped to let it race past, I suffered the illusion of its prescription windscreen magnifying the golden tresses of Robert Plant, surrounding his wide grin, beaming and waving to us from the passenger seat.

We drove on to the Post Office, where I sat dazed and... speechless. That was Robert Plant, I eventually managed to squeeze out. Rubbish! explained Linda. Who? added her mum from the back.

Later, driving back toward the main road, I continued belabouring the assertion we'd just passed the original Golden Rock God™ on a Highland dirt track. Then rounding a corner where the road widens temporarily, we passed a black limousine, with smoked windows and the registration ZL7, heading the other way, followed by an entourage of assorted vehicles. I swear I heard music pounding, and peering briefly through the windscreen, discerned the unmistakable outline of Jimmy Page in the middle of the back seat.

Reflexively I spun the car around, racing them back to the little deserted beach where we'd just spent our afternoon. We arrived at dusk. Planet Zeppelin converged in that place that day, celebrating the release of their new DVD. There was music and merriment and naked dancing and whisky and every flavour of sex and drugs and did I mention sex. I'd brought my guitar with me, and later that evening, gave Jimmy some advice about that tricky Stairway opening...

Actually the only U-turn I made was into fantasy fiction at the previous paragraph. That year I'd decided to bring my mother-in-law instead of the guitar; we were in no shape to gatecrash an all-night wild beach party. The rest of the journey back to our Embo caravan was... I'd say, inconsolably quiet.

My Work in Print: 1987

Previously: 1985

DISZ80 - a Z80 Disassembler

Delighted today, to receive a comment from Geoff Wearmouth on that previous Personal Computer World: Sub Set article. Geoff it was, who first typed my 1094-byte Z80 disassembler DISZ80 into a Sinclair Spectrum some quarter century ago. His site http://www.wearmouth.demon.co.uk/ gathers Sinclair ZX Spectrum ROMs, as well as emulation tools for the related ZX80, ZX81, and the eternally awe-inducing Jupiter Cantab Ace.

Geoff's own Spectrum Globeplotter, featured in The Register Hardware's Basic Instinct last week, and itself more than a quarter century old, proves us to be PCW stablemates. Whilst presented as a BASIC listing, this spinning-sphere-transpoising, sinusoidal-Mercator-projecting screen plotter actually employs untold amounts of custom crafted machine code.

Some ten years ago, Geoff made appeals in comp.sys.sinclair for any further information about PCW back issues. And just today, Stephen Parry-Thomas of http://jupiter-ace.co.uk/ has replied, with full scans of Sub Set from February-May 1987. At last, the extensive line comments can be restored to my crusty old utility!

February 1987

Click to big. In "Z80 Radix Conversion" David Barrow starts off with CONVHL, which is used at various points within the main disassembler to convert 16-bit integers to various bases (binary, octal, decimal, hex) for readable output. He remarks upon the "quite ingenious" optimised method used to convert raw digits to ASCII.

I remember being quite happy with the way David had split out the various modular components of the program. Publishing these in multiple datasheets over several months resulted in a steady stream of payments, giving a pleasant illusion of gainfully paid employment!

March 1987

March saw the publication of routine XTRACT, designed to read and print strings from a table. As David correctly notes, this too is pathologically storage space optimised, for both code and data table size.

April 1987

The I/O fest continued in April with a set of five connected utilities dealing with the output of hex data and addresses. This is the point at which David, having correctly diagnosed my space optimisation compulsion, shows where I could have saved one program byte, two stack bytes and 14 clock cycles. Though 25 years later I've still to validate the assertion, respect for David's manifest expertise compels me to believe it, albeit truculently.

May 1987

The May edition was given over to just the two remaining big routines, spread across five new datasheets / appendices, and completing the 1094-byte Z80 disassembler.

So, once again: hooge thanks to the guys, Geoff Wearmouth and Stephen Parry-Thomas, who got these page scans to me - I do hope you'll both understand and forgive the indecent haste with which I've posted them here.

Try It Out!

Finally, Geoff's just been in touch again to tell me that DISZ80 is actually live online, and has been for years, embedded in his ZX Spectrum Emulator. How cool is that? Click here to run it. I just tried it out using Internet Explorer 8, 64-bit (!) version. Click the "screen" then press Enter for instructions. The Disassembler itself can be disassembled (!!) by typing the address $5100 then pressing Enter. The code you see starting at that address is actually the DISZ80 entry point, as illustrated in the first of the May 1987 pages above. Mind = Blown!

April in Embo

Photo Essay: Easter Break Snaps

Look mum! The sand's back.

Eek! A periscope.

The Russians have been watching this place for years.

Home from home: the new Linnhe caravan (6-8), introduced just this year. With sun deck! (Sun not included)

Look at those bananas. Just look at them.

The neighbours.

This is the life.

Some nice weather.

Garden fence conversation.

Handy for the beach.

Moody, broody, old North Sea.

The old pier, no longer accessible - officially. Thanks, HSE!

The view towards Portmahomack. Wonder if we'll ever find that lighthouse?

More neighbours: the Loch Fleet seals basking at low tide.

Dornoch International Airport.

Windsock action closeup.

Dunrobin Castle, Golspie.

Grounds.

More grounds.

Still more grounds.

Clynelish Distillery.

Clynelish Distillate!

Day trip to Ullapool.

Encore en route.

High art concept shot!

Are you lookin' at me?

Are you lookin' at me?

Arrival at sunny Ullapool.

The obligatory scenic diversion to Achiltibuie.

More scenic ridgery.

We found it! It's in Tarbat Ness (we love lighthouses).

Photos copyright © 2012 by Linda Kerr, except those containing Linda, which are copyright © 2012 by me.