Sunday, 29 September 2013

Natural Sort Order

Ten Before Two But 10 After 2

A recent update to our flagship product takes advantage of the server-side paging capabilities of the DevExpress Grid control. Naturally, this change has involved the migration of much client-side C# code into server-side table changes, triggers, and stored procedures, all written in SQL. One of the casualties was a particularly hirsute C# method which used to sort table contents into what's sometimes called "Natural Order", so that e.g. DC10 would come after DC9, rather than the naive collation or ASCII order, which would have them reversed.

For reasons unknown it fell to me to implement our Natural Sort. Easy I thought, I'll ask Google for the answer. That's when I discovered there's really no such thing as The Natutral Sort Order. It depends on your particular data, and your specific requirements, to a frankly surprising degree. Of course, there is plenty of material to choose from on the web. Jeff Atwood's Coding Horror on the subject is quite a good central station for your exploration of the matter. But this plenitude is also a bit of a problem. After an hour or two of research, I'd decided my best plan was to design and implement my own, newly reinvented wheel.


The basic approach is to partition or "stripe" the input (unsorted) data into a sequence of fields, alternating alphabetical content with numerical, and then treat each field as its own Sort By column - sorting the alpha fields as normal, and the numeric fields numerically. In the above DC9 / DC10 example, this results in an initial alpha field containing "DC" in both cases, followed by a numeric field containing the integers 9 and 10, which are then subjected to a numerical sort.

Some of the examples I'd read performed this latter sort by actually converting the input data field into an integer, then using the language's numeric comparison capabilities. I didn't want to use that approach, because a 32-bit signed integer can only be used for field sizes up to 9, a 64-bit one 18, and so on. I had no specification to assure me customer data wouldn't exceed such an arbitrary bound, so I fell back on the old workaround of keeping the numeric fields in string form, but left-padding them with zeros until all the rows were the same length, allowing an alpha sort to do the job of a numeric one. This is essentially part of what we do when we adopt the ANSI date format, e.g. 2013-09-29, to get correctly ranked dates.


Notice that it doesn't matter which padding character you use to right-align the numeric fields, just as long as it doesn't come after 0 (zero) in the collation order. This is important later.

One fun fact I found while researching collations was that aa comes after ab in the Danish / Norwegian collation order. For historical-typographical reasons, aa is treated (while sorting) as identical to the letter å, which just happens to be the last letter of their shared alphabet. Never mind; we'll just have to assume anyone using a particular collation order knows what they're doing, and won't be surprised by results which after all should in all cases appear perfectly non-anomalous to them.

Field Sizes

Okay, so now we have this requirement to right-justify our numeric fields by left-padding them with e.g. zeros. What fixed field size should we pad them out to? Well our input data, being stored in a traditional, relational database, has some particular maximum string length. In my case that length was 200. There's nothing to stop our customers filling every single character space with a decimal digit. So we could be looking at a numeric field of width 200.

What about the alpha fields? These don't require padding, since standard left-to-right string comparisons work fine for them. But note that every alpha character "uses up" one position in the input data, rendering that location unavailable for storing a subsequent digit. Long story short, we can stuff any alpha prefix into the left edge of our 200-character field, and still have enough room to fit and pad out the remaining, right-justified, numeric content.

For performance reasons, the ultimate destination for this calculation was a "live" Sort Order column in the relevant database table, rather than a UDF call at SQL SELECT time. That's why my buffer size had to be allocated statically, rather than optimised with reference to the worst case requirements of the data actually in the table; we didn't want a new row of data invalidating the precomputed sort orders of the old data.

Islands and Seas

You might worry about non-digit characters encroaching on the padding space of our numeric fields, and you'd be right to. Actually everything works okay as long as we stick to letters and digits. Anomalies can start to appear when we introduce punctuation characters, especially when using ASCII collation. The numeric digits and the upper- and lower-case letters can be seen to form three disconnected "islands" of codes, surrounded by four seas of punctuation characters.

In practice, these anomalies are mitigated by our customers' sparse use of such punctuation, and tendency to apply it consistently whenever it is used at all. As a further mitigation, I changed the padding character from a zero digit to a space, ensuring that padded-out numeric fields are essentially guaranteed to sort lower than any alpha or other character found in the same region.


The following, correctly sorted data can be seen to illustrate these adjustments. Notice, in the Natural Sort column, the use of the space character as filler, and the fact that the 'h' of 'Coach' occupies the same character column position as the '1' of 'Van 1234' without causing any problem:

Simple Sort   Natural Sort

 Coach 12      Coach  2
 Coach 2       Coach 12
 Van 1234      Van  234
 Van 234       Van 1234

Field Count

Obviously the input data might contain further alternating fields of non-numeric and numeric data. What should we do with subsequent fields? Well, we just package them up in pairs using exactly the same algorithm as the first, and when we have done this enough times to consume all of the input, return the concatenation of all these field blocks as the sortable key.

There is another minor optimisation available here. Obviously the first field pair must have consumed at least one character from the input. This means that the field "buffer" for the second pair can be made one character shorter than the first - say, 199 characters instead of 200. Likewise, if input characters remain after the second or subsequent field pair have been extracted, then that pair must have consumed at least two characters, so the next buffer size can be 197, or 195, or 193, or...

Yes, quite. The law of diminishing returns cuts in quite promptly here, especially since we decided that a total of 3 field pairs would be more than adequate for our customers' requirements (actually there is an auto-generation element in our product, designed to nudge them into using just a single field pair: an alpha prefix, followed by a numerical index). So I just left all my buffers at width 200. You should obviously make use of this optimisation if your input size limit is much lower than 200, or if you decide to use a lot more significant field pairs.

Coding and Testing

This works adequately and was ultimately accepted for production, but I must acknowledge here the excellent work done by our Test Department, both in testing my new UDF out-of-sequence when I asked - quite unreasonably - for an early comparison with the old client-side C# function (come to think of it, how the hell did they even do that?), and also in uncovering pretty quickly all of the corner cases - punctuation, padding characters - mentioned above.

Oh, the code? Yeah sure, here ya go. Should be quite easy to hunt down the few rogue "200"s to adapt it for your use. You might also want to limit the number of iterations to prevent DOS attacks (we use only 3). As I said at the outset, it's still surprisingly specific and might not work for you. For example, it doesn't recognise decimal points / numeric separators. The truth is, there simply does not exist one Natural Sort Algorithm suitable for all data.

CREATE FUNCTION [dbo].[NaturalSort](@input NVARCHAR(200))
    @count INT = LEN(@input),
    @result NVARCHAR(MAX) = '',
    @p INT = 1, @q INT, @x INT, @y INT
  WHILE @p <= @count
    SELECT @x = PATINDEX('%[0-9]%', SUBSTRING(@input, @p, @count) + '0') - 1
    SELECT @q = @p + @x
    SELECT @y = PATINDEX('%[^0-9]%', SUBSTRING(@input, @q, @count) + '!') - 1
    SELECT @result = @result + SUBSTRING(@input, @p, @x) +
      REPLICATE(' ', 200 - @x - @y) + SUBSTRING(@input, @q, @y)
    SELECT @p = @q + @y
  RETURN @result

Three Onions Holding Up A Melon

Who Ordered That?

A favourite Twitter follow is @maanow, with their daily MAA Minute Math problems. These are usually simple little scenarios that take, as implied by the name, a minute or so to solve. But one of the best so far, I don't mind you knowing, took me a little longer than that. It involved three touching 1" radius spheres supporting another sphere of radius 2", and asked: what's the height of this arrangement?
What surprises me about the solution to this one is the appearance of the square root of 69 in the answer. Doesn't that strike you as more than a little bit literally odd?

The solution is arrived at by first noticing that a line connecting the centre of the large sphere to the centre of one of the smaller, touching ones, has length 3 (all dimensions may as well be in inches; it's immaterial to the result). Now, this line is the hypotenuse of a right triangle, whose third vertex is the midpoint of an equilateral joining the centres of the three smaller spheres. This second triangle has side length 2, so the distance from its midpoint to its centre is 2/√3. Squaring both this length, and that of the earlier hypotenuse, en route to the ultimate solution, we discover that curious √69 emerging from the fact that 3³ - 2² = 23 = 69/3.

That's all, just wanted to draw your attention to this little oddity today.

Thursday, 21 February 2013

The Raven that Refused to Sing

and other stories

Prolificacy's name is Steven Wilson, as you might agree when you've seen the man's 500-page discog here.

Each new Steven Wilson album release is a bigger, fuller, more satisfying art-event than its predecessor. I'm talking of course about the limited deluxe editions. Nothing else is worth having, if you can have instead one of these wondrous objects, while nothing else - no mere collection of tunes and songs - is in our time, worth our time reviewing. Obviously then, it's especially edifying when - reportedly to beat an internet leak - your preorder arrives, entirely unexpectedly, four days early.

Rivaling the dimensions and heft of some paving slabs, well the 10" ones anyway, these special editions seem to have evolved by a process of continuous improvement into a format which by now should make the fan relax immediately into the familiar luxurious form, simultaneously with the unfailingly new and truly multimedia content. It's like a lottery win of the senses. For consider:

First, it looks and feels great - a solidly substantial objet d'art, with its great 5mm thick hardbound book cover enveloping 128 heavy and glossy illustrated pages full of impressions, stories, lyrics, sleeve notes, mysteries. Next, you get the smell of it, obviously the high aromas of a newly printed, full gloss inky tome. Only at the end of these sensory introductions, and as a final formality, will you eventually get to hear it. Before then, this beautiful thing has already carved its little square snug into your world, and as you begin your first consumption of these sounds (2 CDs) and visions (a DVD and a Blu-Ray), you reflect that Steven could easily have achieved a full sweep of all five senses, by the simple expedient of including a stick of chewing gum. Maybe next time.

The Musicians

Every work of this standard of quality, and each major aspect of such a work, is essentially in itself a collaboration. The original and most vital aspect here being of course the music, that collaboration comprises primarily Steven himself, singing and playing various keyboards (including King Crimson's original mellotron, borrowed from his friend Robert Fripp) and guitars, as befits a solo album. Thence in the astonishing ensemble of legends that is his touring band, alphabetically we have Nick Beggs on bass, Chapman Stick and vocals; Guthrie Govan on lead guitar, and Adam Holzman on keyboards; while Marco Minnemann* hits stuff, and Theo Travis blows into things.

Oh, and Steven's choice of associate producer / recording engineer? Only bloody Alan Parsons.

The Music

The album comprises six pieces, with three of these - satisfyingly enough, for those with an adult's attention span - each being over 10 minutes long. It opens with a song previously released on Steven's earlier live document, the quite recent Get All You Deserve 2CD / DVD / Blu-Ray.

1. Luminol: an uptempo percussive 4-to-the-floor with driving bass, signals the initial intent. Transitioned by Theo's flute and harmony vox / mellotron, it falls and fragments into a typically prog clipped and snipped timesig, before slowing and softening into the story of the busker (one of Steven's ghost stories), with yet more lush harmonies. There are hints of and nods to a shared progressive jazz heritage throughout, culminating in a typically strong KC and mellotron restart and return to quick tempo for the finish. From Wikipedia:
"Luminol", which was first performed by Steven Wilson and his band on the second half of his Grace for Drowning tour, takes its inspiration from a busker, who, according to Wilson, is "there every single day. It doesn’t matter what the weather is like; he’s always there, playing his acoustic guitar and singing these songs. Snow, rain, gale force wind – nothing will stop him from being in his spot. ... He’s the kind of guy who is so set in his routine that even death wouldn’t stop him." Wilson considers the notion "that somebody could be a ghost in life, as well as a ghost in death, somebody who’s completely ignored even in their lifetime – it hardly makes a difference; and death doesn’t make a difference, either; it doesn’t break the routine." [1]
2. Drive Home is soft and gently acoustic, showcasing Steven's famous talent for hook laden tune smithery. You need to clear away all the jetsam in your brain and face the truth. Even this gentility itself takes a percussion break, a breather wherein for example a delicate guitar part ambles alone for a few bars. Builds to a mixed (both climactic and anticlimactic) finish.
"Drive Home" is based on a suggestion from illustrator Hajo Mueller. It is "about a couple driving along in a car at night, very much in love; the guy is driving, and his partner – his wife or girlfriend or whoever she is – is in the passenger seat, and the next minute she’s gone." The ghost of the man's partner eventually returns, "saying, ‘I’m going to remind you now what happened that night.’ There was a terrible car accident, and she died, etcetera, etcetera – again, the idea of trauma leading to a missing part of this guy’s life. He can’t deal with the reality of what happened, so he blocks it out – like taking a piece of tape and editing a big chunk out of it." [2]
3. The Holy Drinker indulges a sequence of guitar / keyboard / clarinet solos as well as some welcome overdrumming in its satisfyingly extended (2½ minute) 4/4 introduction, prior to leading into another SW ballad, whose verses are similarly separated by inventive instrumental fills. The middle duet between Fender Rhodes and flute, leading into some big Keith Emerson chops, is particularly worth the ticket price. When the story continues in a lente pianissimo section, its dramatic and mathematical conclusion can be sensed from afar.
According to Wilson, "The Holy Drinker" concerns "a guy who’s very pious, very religious, preachy and self-righteous. I’m thinking of TV evangelist-types – guys who are prepared to tell people that they’re living their lives wrong and that they’re missing something because they don’t believe in God or whatever it is." The man, who, despite criticising other people's lifestyles, is himself an alcoholic, unwittingly challenges the Devil to a drinking competition, with disastrous consequences: "Of course, you can’t beat the Devil at a drinking competition – you can’t beat the Devil at anything – and so he loses. ... He gets dragged to Hell." [3]
4. Vocally, The Pin Drop starts unpromisingly. But instrumentally, this is a consistently strong and driving track, with particularly tortured sax.
"The Pin Drop" addresses "the concept that you can be with someone because it’s comfortable and convenient, not because there’s any love or empathy." Wilson explains that "The song is basically sung by the wife. She’s dead, she’s been thrown in the river by the husband, and she’s floating down in the river while singing this song – from beyond death, beyond the grave, as it were." The song considers "The idea... that sometimes in a relationship there can be so much tension, so much unspoken resentment and hatred, that the tiniest thing can set off a violent episode, and in this case, one that ends in tragedy. The sound of a pin dropping on a floor can be the thing that instigates the fury." [4]
5. Back now in time and context to Foxtrot era Genesis, for the introduction to the little proggy masterpiece that is The Watchmaker. Not a love song, but a tale of touching companionship and failure, human existence, inevitability and frailty. Its second act warms and beats to the musical imagery of clockwork, accelerating subjectively through a spectacular clarinet and guitar duet, before a braking return to the piano-backed, heart-rendingly honest narrative. The endgame, by turns instrumental and vocal, is a particularly pleasing and quirky workout.
The fifth track on the album explores "the story of the watchmaker, the guy who is meticulous about his craft, but he never has any kind of emotional outburst, nor does he express violence or any extreme emotions whatsoever." It concerns "a couple who have been together for 50 years or more, purely because it was convenient and comfortable." Wilson explains that "The watchmaker ends up killing his wife and burying her under the floorboards of his workshop. But, of course, she comes back, because she’s been with him for 50 years; she’s not going to leave him now." The song concludes when "the wife comes back to take him with her, which", Wilson suggests, "is another classic ghost story, in a way." [5]
6. As for the closing, title track: The Raven that Refused to Sing is perhaps the most chillingly haunting ballad on this quite haunty album.
The title track explores the story of "an old man at the end of his life who is waiting to die. He thinks back to a time in his childhood when he was incredibly close to his older sister. She was everything to him, and he was everything to her. Unfortunately, she died when they were both very young." The man becomes convinced that a raven, who visits the man's garden, is something of "a symbol or a manifestation of his sister. The thing is, his sister would sing to him whenever he was afraid or insecure, and it was a calming influence on him. In his ignorance, he decides that if he can get the raven to sing to him, it will be the final proof that this is, in fact, his sister who has come back to take him with her to the next life." [6]
See and hear for yourself:

The six indented comments above, quoted from Wikipedia, are in fact primarily sourced from a particularly illuminating interview with Steven (appearing two weeks ago in musicradar) in which he talks about the initial development of the project, and discusses the background to each song in turn and in generous detail.

The Bonuses

The second CD contains different versions of all of the main songs, curiously labelled "(demo)" - who on earth would demand a demo from an effective supergroup** like this? - together with one "unused idea", Clock Song. Were the pendulum percussion and the musical chimes regarded as too obvious, too Mellotron Scratch? Maybe it just couldn't be bent to inhabit the overarching and otherwise cohesive supernatural / ghost story / fear of mortality / end-of-life regret concept circle of the piece. Whatever the reason, it is indeed little more than a not-fully-developed idea, inessential, though not unpleasant.

The documentary DVD and Blu-Ray (see below) also contain between them:
  • a 96/24 Stereo LPCM of the album;
  • one bonus track - Drive Home - lounge version, also 96/24 Stereo LPCM;
  • instrumental versions of all the album tracks (96/24 Stereo LPCM);
and as you would expect from the progressive world's most prolific 5.1 remastereer, the following 5.1 mixes:
  • DTS 96/24 5.1 surround
  • Dolby AC3 5.1 surround
  • 96/24 5.1 LPCM
  • DTS-HD Master Audio 5.1

One reviewer, while describing this new album as painful, moving, desperate, melancholic and superbly beautiful and exhorting us to Buy it, also advises, It’s pretty much a progressive affair. As simple as that. There is nothing here that manages to stretch itself out of the canons of such a well-defined genre. And he may be right; but many long-time fans of Steven Wilson (and of Porcupine Tree in particular) are impressed by the jazz sensibility and free-form styling that have infected his recent work - perhaps inevitably, given the creative cadre of influences here, and his own experiences in remastering those King Crimson, Jethro Tull et al masterpieces for surround audio. To those fans, this kind of prog is new and fresh today, forever heading off in unexpected directions.

The Stories and their Art - Hajo Mueller

While Steven himself provides the quite literally haunting stories for Luminol and The Birthday Party, collaborator Hajo Mueller supplies the concept and illustrations throughout the book, as well as the original idea for the title story which he then reworked with Steven. Both the DVD and the Blu-Ray include his art gallery.

Photography & Documentary - Lasse Hoile

It wouldn't be a new Steven Wilson opus without that final contribution from his long and fruitful collaboration with Lasse Hoile, who here provides for both the DVD and the Blu-Ray, the photographs used in the photo gallery, as well as filming and editing the indispensable studio documentary.

The Gig

Fellow Weegies, let's all catch the touring band at Glasgow's ABC this March 2ndUpdate: Oh wait...
* Zappa alumnus supreme Chad Wackerman (Best Drummer Name Ever!) stands in for Marco Minnemann on the North and South American legs of the current tour.
** Actually, each member leads his own band. Hypergroup, then?

Thursday, 14 February 2013

Claims Based Access Control

The previous article on Role Based Access Control described the IIdentity and IPrincipal implementations available with Microsoft .NET 1.0 through 4.0. Designed in 2002, these soon began to creak under the stress of increasingly distributed systems of rising complexity. With WCF in 2006, Microsoft provided another security model incompatible with the old one, and in 2009, succeeded in combining the best parts of these two implementations into WIF, and ultimately in 2012, into .NET 4.5.

IIdentity and IPrincipal Revisited

The new hotness was Claims Based Authorization. A System.Security.Claim is a statement about an entity, made by another entity known as the Issuer of the claim. The statement takes the form of a key/value pair, which is obviously much more powerful than the true/false settings used in Role Based. For example, Active Directory might issue claims that the current user is named "alice", is an administrator, has email address "", and so on. We would treat these claims, coming from AD, with a high level of confidence.
public class Claim
  public virtual string Type { get; }
  public virtual string Value { get; }
  public virtual string Issuer { get; }
  // ...
We could set up claims using syntax like this:
var claim = new Claim("name", "alice");
However, recall that previously we found reasons (of localization and breaking admin changes) to avoid using literal labels to identify security resources such as groups. It turns out that we might also want to use the predefined members - essentially namespace URIs - of the ClaimTypes class, to ensure that our claims can be serialized, shared and understood across a federated identity space:
var claims = new List<Claim>
  new Claim(ClaimTypes.Name, "alice"),
  new Claim(ClaimTypes.Email, ""),
  new Claim(ClaimTypes.Role, "Sales"),
  new Claim(ClaimTypes.Role, "Marketing")
ClaimsIdentity and ClaimsPrincipal

Obviously, claims can carry a lot more detailed information about an entity than roles. For excellent reasons of backward compatibility, claims were integrated into the existing IIdentity / IPrincipal framework through the creation of two new container classes:
public class ClaimsIdentity : IIdentity
  public virtual IEnumerable<Claim> Claims { get; }
  // ...

public class ClaimsPrincipal : IPrincipal
  public virtual IEnumerable<ClaimsIdentity> Identities { get; }
  // ...
Now we can set up Alice's principal like this:
var id = new ClaimsIdentity(claims, "Console App");
var user = new ClaimsPrincipal(id);
Thread.CurrentPrincipal = user;
The second parameter to the ClaimsIdentity constructor, authenticationType, is needed because, unlike the case with Role Based Identity, Claims Based Identity allows claims to be attached to e.g. anonymous principals. By default then, if authenticationType is not provided, IsAuthenticated will return false.

Backward Compatibility

Examining the framework source, it's interesting to note that every (non static) public in these new classes is virtual. That's because they have been "interposed" in .NET 4.5, so that all the implementation classes we looked at last time (WindowsIdentity / WindowsPrincipal, and GenericIdentity / GenericPrincipal) as well as those we didn't (e.g. FormsIdentityRolePrincipal) are now derived from these common base classes. This design gives all consumer code access to the claims collection if desired, while still preserving the old interfaces (Name, IsInRole, etc).

The base classes implement the legacy security model with the help of the claims collection. For example, accessing the ClaimsIdentity.Name property causes this collection to be traversed, searching for an entry with type ClaimTypes.Name, while checking for a role is done by searching for ClaimTypes.Role entries.

Incidentally, both of these associations can be overridden using the supplied, overloaded, ClaimsIdentity constructors. So you can use these base classes to benefit from the power of the claims approach in your own custom implementation, while still retaining the flexibility of selecting which ClaimsType values you want the legacy support to treat as names and roles:
public class MyClaimsIdentity : ClaimsIdentity
  public MyClaimsIdentity(IEnumerable<Claim> claims, string authenticationType) :
    base(claims, authenticationType, ClaimTypes.Email, ClaimTypes.GroupSid)
In the above example, accessing the Name property will search for a claim of type Email (often used as the unique user identifier), while a call to IsInRole will search for a match within the groups to which the user belongs.

Consuming the Principal

To get access to the claims from runtime code, we simply retrieve the IPrincipal from the thread as before, and cast it to a ClaimsPrincipal. Alternatively, and equivalently, we can obtain it in a single step via the handy, static ClaimsPrincipal.Current property:
var user = ClaimsPrincipal.Current;
With this we can now run arbitrary LINQ queries on the Claims collection property, or take advantage of the special FindFirst, FindAll and HasClaim methods, all of which accept either string parameters or lambdas.
var email = user.FindFirst(ClaimTypes.Email).Value;
var adClaims = user.FindAll(claim => claim.Issuer == "AD AUTHORITY");
Credential Types and Transformations

.NET 4.5 boasts out-of-the-box support, via descendants of the SecurityTokenHandler class, for a wide range of industry standard authentication protocols, which it translates automatically to ClaimsPrincipal format. These include:
  • Windows (SPNEGO, Kerberos, NTLMSSP)
  • Forms authentication
  • basic HTTP authentication
  • SSL client certificates
  • WS-Security tokens
  • SAML
Incoming credentials arrive, are processed and transformed, and then delivered to your application in the "final" ClaimsPrincipal. This processing begins with an appropriate SecurityTokenHandler deserializing and validating the incoming token, from which it constructs the "initial" ClaimsPrincipal. This can then be further processed or transformed, for example to validate the incoming claims or translate security-domain groups into application-domain user permissions, via your custom ClaimsAuthenticationManager (to run the following sample code, add references to System.IdentityModel and System.Security):
public class MyClaimsTransformer : ClaimsAuthenticationManager
  public override ClaimsPrincipal Authenticate(
    string resourceName, ClaimsPrincipal oldPrincipal)
    // Validate the supplied claims
    var name = oldPrincipal.Identity.Name;
    if (string.IsNullOrWhiteSpace(name))
      throw new SecurityException("User name is missing!");
    // Process claims
    var creditLimit = LookupCreditLimit(name);
    var newClaims = new List<Claim>
      new Claim(ClaimTypes.Name, name),
      new Claim("http://myclaims/creditlimit", creditLimit.ToString())
    // Create & return the new ClaimsPrincipal
    var newId = new ClaimsIdentity(newClaims, "Local");
    return new ClaimsPrincipal(newId);
Obviously we have to find somewhere to tell the system to use our new custom ClaimsAuthenticationManager class, and the equally obvious place to do that is in the configuration file. There's a new "system.identityModel" section:
      type="System.IdentityModel.Configuration.SystemIdentityModelSection" />
        type="MyNamespace.MyClaimsTransformer, MyAssemblyName" />
Finally, at the point in our application code where previously we would perform our ad hoc claims processing, polluting our beautiful domain code with extraneous user security permissions logic, we now have essentially this fixed one-liner (no, don't laugh):
Thread.CurrentPrincipal =
    "none", // Can pass eg a URL resource for context
    new WindowsPrincipal(WindowsIdentity.GetCurrent()));
Next time: session management.

Tuesday, 12 February 2013

Role Based Access Control

This introduction to aspects of Identity and Access Control in Microsoft .NET 1.0 through 4.0 serves as groundwork for examination, in a later article, of the same issues in .NET 4.5. We start with two short interface declarations:

IIdentity and IPrincipal

These two interfaces first appeared in 2002, with the initial release of .NET 1.0 (and ASP.NET 1.0):
interface IIdentity
  bool IsAuthenticated { get; }
  string AuthenticationType { get; }
  string Name { get; }

interface IPrincipal
  IIdentity Identity { get; }
  bool IsInRole(string roleName);
For developers these interfaces abstract, and their implementing classes encapsulate, the most important features of identity-related concerns. IIdentity deals only with authenticationName obviously identifies the user; AuthenticationType describes how that user became authenticated (e.g. via Windows Authentication or Kerberos, or some other custom mechanism); and IsAuthenticated indicates whether the user is actually authenticated at all (client-server apps sometimes allow anonymous users). IPrincipal extends this provision into the adjacent realm of authorization, by providing the single method IsInRole allowing the aggregated IIdentity to be interrogated at the first level of role-based access control granularity.


Not to be confused with the Win32 "thread token", this is a static property of the Thread class, giving access to the currently executing thread's IPrincipal implementation. So not strictly static then, as every thread in your system can have a different value of IPrincipal. More sort of thread-static. It is set up when the security context for a client is established, either by the startup code of a desktop app, or by some runtime in the case of an ASP.NET or WCF server app.

WindowsIdentity and WindowsPrincipal

When the AuthenticationType indicates Windows Authentication, these are the concrete System.Security.Principal classes used to wrap the Windows security token, encapsulating the Windows account of the current desktop app process (or of the client, in a client-server system):
// Get Windows account user name
var id = WindowsIdentity.GetCurrent();
var userName = id.Name;
// Check that user belongs to Builtin\Users
var principal = new WindowsPrincipal(id);
var isUser = principal.IsInRole(@"Builtin\Users");
Note that while this code works, it is not advisable to refer to user groups using strings. For one thing, the built-in group names are localized. For another, an administrator might change the name of a group, obviously breaking any hard-coded string based checks. For reasons like these, Windows internally uses not strings but immutable Security IDs (SIDs) to reference both users and groups:
var account = new NTAccount(userName);
Console.WriteLine(account); // MY-DOMAIN\johnk

var sid = account.Translate(typeof(SecurityIdentifier));
Console.WriteLine(sid); // S-1-5-21-29562463-4174942564-1615450050-1377

foreach (var group in id.Groups)
This last line prints the list of groups as human-readable strings, e.g.,
MY-DOMAIN\Domain Users
NT AUTHORITY\Authenticated Users
NT AUTHORITY\This Organization
MY-DOMAIN\Local Group
MY-DOMAIN\Employment House
Remove the Translate call to see the raw security IDs underlying your group names.

Returning to the earlier question about the interrogation of roles, we can now use the idea of a security identifier, together with the provided enumeration WellKnownSidType, to determine for example whether the current user is a local administrator:
var localAdmins = new SecurityIdentifier(
  WellKnownSidType.BuiltinAdministratorsSid, null);
The second parameter to the above SecurityIdentifier constructor is itself a SID. As suggested by its parameter name domainSid, domains also have security IDs, and can be passed around as such when required. For example, to determine whether the user is a domain administrator, you have to supply the SID of the relevant domain, which in this case is the user's account domain:
var domainAdmins = new SecurityIdentifier(
  WellKnownSidType.BuiltinAdministratorsSid, id.User.AccountDomainSid);
Lastly, to answer the original question about whether the user is a member of the built-in users group,
var users = new SecurityIdentifier(
  WellKnownSidType.BuiltinUsersSid, null);
UAC Reminder

Ah, yes. Do remember when interpreting these results client side, that the returned values respect User Account Control (UAC), enabled by default since Windows Vista. UAC prefers to run client processes without administrative privileges, unless instructed otherwise (e.g. by Run as Administrator).

GenericIdentity and GenericPrincipal

So much for the WindowsIdentity and WindowsPrincipal classes. These are ideal when your user list is stored in Windows itself, for example when you use Active Directory. But what if your user list is in some other repository, such as a SQL database?

Using the GenericIdentity and GenericPrincipal classes, you can manage user names and roles for yourself, persisting them in your own custom credential store, while still taking advantage of those very useful IIdentity and IPrincipal properties - and specifically, the IsInRole method. Using System.Threading, we have (assuming that Alice has previously supplied a correct password, or otherwise authenticated):
var id = new GenericIdentity("alice");
var roles = new[] { "Sales", "Marketing" }
var user = new GenericPrincipal(id, roles);
Thread.CurrentPrincipal = user;
This sets up the necessary principal for subsequent use. Later - say, during the execution of some other method on this thread - we might want to find out the current user name, and whether she is in Sales:
var user = Thread.CurrentPrincipal;
Console.WriteLine(user.Identity.Name); // alice
Console.WriteLine(user.IsInRole("Sales")); // True
Authorization and Roles

Whichever IIdentity and IPrincipal implementations we use, the mechanisms of role-based authorisation are the same. Here are four alternatives, from the imperative to the declarative, all of which resolve directly or eventually into IsInRole calls:
  1. make direct calls to Thread.CurrentPrincipal.IsInRole();
  2. assert System.Security.Permissions.PrincipalPermission demands;
  3. apply [PrincipalPermission] attributes to methods or classes;
  4. add <authorization /> elements to web.config.
Example 1 - coded tests:
if (user.IsInRole("Marketing"))
Example 2 - security assertions in code:
// Throws a SecurityException if the assertion fails
new PrincipalPermission(null, "Marketing").Demand();
Example 3 - security assertions in attributes:
[PrincipalPermission(SecurityAction.Demand, Role = "Sales")]
[PrincipalPermission(SecurityAction.Demand, Role = "Marketing")]
private static void DoSomeSalesOrMarketing()
  // ...
Example 4 - <authorization /> elements in web.config:
         <allow roles="Sales,Marketing"/>
         <deny users="*"/>
Notice that options 1-3 above all have the disadvantage that the security model is hard coded throughout the source. Obviously, this can be very hard to maintain. Also in option 3, there is no simple, convenient and tidy way to hide these security assertions from the CLR, for example to run unit tests. These objections and many others have been addressed by the claims-based security authorization model introduced in .NET 4.5, which is the subject of the next article in this series.

Sunday, 3 February 2013

m b v

Plainly, everyone who is anyone is either talking or joking, or both, about the biggest music news of the year to date, not directly involving David Bowie.
Namely, the new

My Bloody Valentine

album, m b v, whose quiet release (escape?) softly echoes that of Sir David's latest song.
Not too many reviews yet though.
Let's see, I'm on to my fifth listen now, should be able to start collecting impressions.
Why not read along while you listen to it on their official YouTube channel:
(Oh incidentally, Robert's joke is that MBV are always being confused with Bridgend's finest metallurgists, Bullet For My Valentine.)

she found now

Solid Loveless territory.
I'm instantly reminded to turn up to 11 and blow my ears clear with a tin of Allsops Airduster for the full effect.
Ah yes, there ya go, hello old friends.

only tomorrow

Unremittingly shoegazing, with beautiful vocals, angelic in places.
Well, in one place really: the end of each verse.
And sporadically between verses.
Two places, then.
Nicely melodic guitar fuzzes in the breaks.

who sees you

A dissonance repeatedly reaching out, almost too slowly, too falteringly, and then only - just! - acquiring its goal on every turn.
We used to have to drill extra eccentric holes in our 45s to generate this kind of entertainment.
Like when you love chillies, and someone gives you picked jalapeños.
Not a home run, nor likely to grow on me.

is this and yes

Very pretty.
No surprises.

if i am

Good rhythm, shape and melody.
Getting the warp thing (pitch dip) better this time.
Some fine percussive effects shared across instruments.
Silly ending - well, anything's better than a fade.

new you

Possibly the most un-MBV thing they've done, erm, "recently".
Clear tracked volume-envelope rhythm guitar, keys, bass, drums, wtf?
Only thing making this cut recognizably theirs, is Belinda's singing.

in another way

Wow, some real originality going on in here.
Great guitar sounds, twisted drum loops.
Those strident slashing stabby guitar breaks have an occasionally Scottish flavour.

nothing is

Look out, here comes T-Bone Burnett on caffeine!
In an infinite ascending, crescendoing, percussive loop.

wonder 2

A mad drum loop phased into sounding like traversing high speed trains.
Two-way multiply-echo-tracked vocals.
Well that was different, maybe one of their best.

That's Yer Nine

What did you think of it?
Here's what I thought.
A worthy, if imperfect and untimely, sequel to its two decades senior Loveless sibling.
This is the band's Majora's Mask to their Ocarina Of Time.
Bottom line: it has earned its place in the top 60GB that resides permanently in my phone.

Verdict: 7.5 out of 10.

Come to Glasgow and catch them at Barrowlands on March 9th!

Thursday, 24 January 2013

Pioneer DEH-1400UB

CD RDS Tuner with WMA/MP3 Playback and Front USB

Update: Halfords upgraded my dodgy Pioneer to a superior - and 40% more expensive - Sony, free of charge! Nice customer service.

To: The Manager

Dear Sir or Madam,

Last November my wife purchased a Pioneer car stereo, model number DEH-1400UB, from Halfords as a birthday present for me. It was later fitted, by Halfords personnel.

Fitting took a full two hours instead of one. One reason given for this was that the mechanic did not have the necessary tools to hand. When the job was eventually completed, my new car stereo appeared to work, electrically speaking; however, it seemed mechanically to be quite loosely fitted, as the right side of the unit did not appear to be fixed to the dashboard, but just flexed in and out quite freely.

The earliest opportunity we had to get this apparent fault inspected and fixed was during the Christmas holiday break, when another mechanic showed us that one of the two fixing bolts had apparently been cross-threaded and stripped - in fact he found the missing bolt within the dashboard cavity. He told us that we would have to go to our Honda dealer and order a replacement part for this. My new car stereo unit, its mounting hole having apparently been similarly stripped during the original fitting attempt, would also have to be replaced.

The replacement part arrived and we collected it from Hamilton. Still in December (I think) we returned to Halfords, brand new Honda mounting bolt in hand, to have the replacement stereo unit fitted. This time, after the requisite hour, the unit seemed to be firmly and correctly mounted.


We immediately noticed that the unit was prone to skipping and jumping when playing CDs. It would also cut out entirely while the car was not even moving, e.g., while parked or stopped at traffic lights. Then in January, while driving smoothly along a main road at about 10mph, i.e. no bumps, quite suddenly three of the speakers went silent, while the fourth (offside rear) jumped up in volume and became distorted. Also, the unit’s firmware appeared to have “crashed”, as none of the controls - volume up/down, band/channel change, signal source - had any effect. Fortunately the OFF function still worked.

Upon telephoning Halfords that afternoon, we were instructed to bring the car back on Monday at 2pm, to have the unit looked at.

Meanwhile, we continued to run the unit as a radio tuner with its volume turned down to zero, so that we could at least get the functionality of a dashboard clock out of it. Then on the morning of Sunday 20th I decided, while parked, to very cautiously turn up the volume. I discovered that all four speakers were working once again. However after only a few minutes of operation at this low volume, three of them packed in again, leaving only the offside rear working, with distortion, as before. This time however the unit’s firmware did not crash, and volume up/down etc. appeared still to work.

When we telephoned again at 1pm on the Monday to confirm our upcoming visit at 2pm, we were told that the shop was “fully booked” and that we would have to pick another day. The next day (Tuesday) unfortunately being unsuitable to us, we were forced to agree on 3pm, Thursday - one full week after reporting these latest faults.


On Tuesday, while parked and listening to the FM radio, I noticed the station signal fade then disappear, leaving only a little faint radio noise. I nudged the frequency up then back down exactly one single notch, using the </> buttons, which caused the unit to retune correctly. This happened several times.

On Wednesday, again while parked and listening to music from my phone using the AUX input, I noticed the volume in the right side fall to zero, then return after a minute or so. This also happened at least three times.

This purchase is now two months old, and still we have been unable to get a single day of fault-free operation out of it. In fact while driving we have been forced to continue to keep its volume at zero, in case it decides spontaneously to jump up and distort again, a terrifying prospect which could easily cause a serious accident.

Having encountered such a bewildering array of faults, I am almost at a total loss to explain them - and I have a university degree in electronics! Perhaps the unit has a secret remote control, and someone is hiding in our rear seat pressing buttons at random? And I mean every single button. That might explain it...

I say “almost” because I do have a serious theory, namely: sloppy firmware programming, combined with poor power line noise suppression. But my theories are neither here nor there, and I guess you do have many other customers happily using this same model.

Just to Recap

Recent faults have included:
  • CD jumping (brand new shrinkwrapped CD, vehicle stationary)
  • Spontaneous power down (while playing brand new CD)
  • Triple speaker failure
  • Single speaker distortion
  • Complete loss of control of all functions (except power OFF)
  • Tuner jumping to a random frequency (vehicle stationary)
  • Right speakers fading (vehicle stationary)
We have completely lost all confidence in this particular manufacturer and model of car stereo.