Oh, That Would Be A Fine Trick

After a quiet month or two, I've started hitting my stride again. I've reached my quota of four technical articles for this month, two of those security related (however vaguely). This means I can relax, and fill up the rest of June with popular culture references. Let's enjoy a couple of my favourite tricks from last year's Penn and Teller: Fool Us series. This first is by the Canadian World Champion of Magic, Shawn Farquhar:



This next shows the maestros themselves doing what they excel at. They let the audience in on both the joke and the magic, while still overfilling with enough additional indirection to ensure our ultimate bewilderment:

Third Party Creep

Just Out Of Interest...

The somewhat laborious DevExpress build processes described in the previous article did eventually bear fruit, letting me pare down the problem to its bare essentials. Now I need to send this sample code off to DevExpress, together with as many caveats as I can muster - otherwise, they're sure just to write back with some entirely unworkable suggestions, like refactoring our entire business object hierarchy (test impact = humongous).

// Run this project using DevExpress version 9.2.6, and inspect the Field List tab.
// "Contacts" will contain two properties, "Name" and "Telephone".
// Run it again using DevExpress version 12.1, and inspect the Field List tab again.
// "Contacts" will contain only the "Name" property - "Telephone" has disappeared!

using System;
using System.Collections;
using System.ComponentModel;
using System.Windows.Forms;
using DevExpress.XtraReports.UI;

namespace XtraReportsTest
{
 public partial class Form1 : Form
 {
  public Form1()
  {
   InitializeComponent();
  }

  private void Form1_Load(object sender, EventArgs e)
  {
   new XtraReport {DataSource = new ContactTypedList()}.ShowDesignerDialog();
   Close();
  }
 }

 public class ContactTypedList : ArrayList, ITypedList
 {
  public PropertyDescriptorCollection GetItemProperties(PropertyDescriptor[] listAccessors)
  {
   return TypeDescriptor.GetProperties(typeof (Contact));
  }

  public string GetListName(PropertyDescriptor[] listAccessors)
  {
   return "Contacts";
  }
 }

 public class Contact
 {
  public string Name { get; set; }
  [Browsable(false)]
  public string Telephone { get; set; }
 }
}

Suitably squashed to highlight the relevant areas, here are the results of the test runs with DX version 9.2.6 (left) and 12.1 (right):



This calls for a course on defensive programming!

Building DevExpress Sources

How to Build DevExpress DLLs for Debugging

This article is your 10-step guide explaining how to build a debug version of the DevExpress component assemblies from their supplied source code. My recent experiences force me to say, wonderful component suite, but their instructions on this particular activity fall a little short of being self-explanatory!

1. Get the Zip

First go here:

http://www.devexpress.com/Support/Center/p/A609.aspx

Pick up the Attachment version relevant to the target DevExpress DLLs version. If it's not listed, request it, go home, and come back next day.

Notice the file PublicKeyToken.txt, which we'll be using later. Actually that's the only, paper-thin justification for this article's Security tag...

2. Check DEFINES.BAT

Follow the first couple of instructions in the Attachment's README.TXT file: copying all the files from its folder or Zip archive into your DevExpress Components Sources folder, and then checking the content of DEFINES.BAT. Note from the location of this file, that should you have to edit it, you will probably have to do so using e.g. Notepad Run as Administrator. Verify in particular the specific version of Studio you are using (vsver), the GAC path(s), and the locations of the correct sn, gacutil and MSBuild executables.

3. Edit the Supplied Batch Files

You're going to want to Run as Administrator the DevExpress source building scripts, CLEAR.BAT and BUILDALL.CMD, following the directions in the associated README.TXT file. But before you do, note that since the files CLEAR.BAT and BUILDALL.CMD will probably have to be Run as Administrator, their default current directory will be something like C:\Windows\system32. We need them to be running instead in the DevExpress source folder, and one way to achieve that is to edit these files so they look a bit like this:

REM @echo off
C:
CD \Program Files (x86)\DevExpress 2009.2\Components\Sources
call defines.bat

and so on. However, note also from the location of these files that you will probably have to edit them using e.g. Notepad Run as Administrator.

4. Build the DevExpress DLLs from Source

Okay, so now Run (as Administrator) your edited CLEAR.BAT and BUILDALL.CMD. Obviously you can safely ignore any generated error messages that relate to software components you haven't installed (CompactFramework, LightSwitch, SharePoint, SilverLight, etc).

5. Create Toolbox

If your target version of DevExpress supports it, now is the time to run the ToolboxCreator utility. How will you know if it's required? Well, if it's in the Start menu for your version of DevExpress, e.g.

Start / All Programs / Developer Express v2009 vol 2 / Components / Tools / ToolboxCreator

then it's required, and you should run it. Whereas if it's not, then it's not, and you shouldn't.

6. Copy New DLLs to Local Folder

Before proceeding to run ProjectConverter, if your third party component binaries are under version control as an External, you'll probably want to copy the newly compiled DevExpress DLLs from whichever output folder they've just been built into, e.g.

C:\Program Files (x86)\DevExpress 2009.2\Components\Sources\DevExpress.DLL
- or -
C:\Program Files (x86)\DXperience 12.1\Bin\Framework

to their local destination, e.g. MyProject\ThirdPartyComponents\Developer Express\DLL, or whatever.

7. Convert Projects

The devExpress README.TXT gives a command line for running the Project Converter utility. I prefer to omit the parameters and use the interactive graphical UI version. However that's not always possible, as earlier versions of the converter don't have the Advanced options UI letting you specify the new Public Key (PK). Be aware that the command line version, if provided with a Project Path, will start conversion immediately, without giving you a chance to adjust any of the other parameters. Either way, be sure to make all of the following settings correctly:

• Your Project Path(s);
• HintPath behaviour (I normally use Update rather than Remove);
• The new PK value (this should be available in the aforementioned file PublicKeyToken.txt).

Inspect any errors in the log output by the conversion process and decide whether or not you care enough about them to start again from Step 1 and be more careful!

8. Check Post Build Events

One subject easily forgotten during this process is the area of post build events in your own project code. These are often used to consolidate the outputs of multi-project or multi-solution systems into a final staging or deployment folder, and in particular, to ensure that all third party assemblies are copied efficiently, i.e., once only. For this reason, the events you are likely to have to care about are those associated with your main exe or web app projects: clients, services, data portals, auxiliary apps and so on.

The aim of this step is to ensure that all DevExpress references end up pointing to the same third party component assemblies folder.

9. Clean Project Output Folders

Again, the aim of this step is to ensure there are no remnant, pre-source-build versions of the DevExpress libraries visible to the compiler. Prior to compiling your code, be sure to delete all bin, obj, Output and/or Run folders generated during your build process, to ensure a clean result.

10. Build Your Solution(s)

And we're done! Get back into Visual Studio and check that you can step through the DevExpress source code, see it in the Call Stack, and enjoy all of its sourcey goodness. Or failing that, consult the troubleshooting section of README.TXT, or the DevExpress Support Centre, Knowledge Base and Forum.

11. Reversion

As described in the Attachment's README.TXT file, you can revert to the installed DevExpress binaries by running the installer in Repair mode. This is essentially the undo of Step 4 above, so to complete the process you will possibly want to follow this with steps 5-10. Alternatively, you might just restore your entire solution image(s) from version control.

Best Rock Gag Ever

The surprise return of Dexys after mumble mumble years, with this charming bit of theatre...



...somehow reminded me of the best ever lyrical gag in a rock song...



After listening to Meat's and Ellen Foley's performance of that Jim Steinman epic (though that's actually Karla DeVito in the video), I think you'll agree the best bit starts at 07:00.

Simple Regex #5: Named Groups

Regex.Result()

Almost every Regex question landing on my desk has the potential to get its own blog post. This month's candidate is almost straightforward enough to be painless. Almost, because well, there's always some pain with Regex! Colleague 7 wanted to know how to extract coordinate data tidily from a string...

Regex question for you if you don't mind:

@1,X=123@1,Y=456

Looking to pick the x and y coordinates out of that string and return a point. I can obviously get the bits separately. But that would be too easy!

(?<=X=)[0-9]+
and
(?<=Y=)[0-9]+

Should I use named capture groups?

Well, personally I would, because the outcome is (1) marginally more readable than indexed groups, in my opinion; and (2) more robust, when subsequently you have to extend the pattern to incorporate further groups. Relabeling index-based groups is a nightmare!

Notice that the above suggested digit filters are using zero-width positive look-behind assertions, (?<= ), which were covered in the previous article in this series. These are just looking for an X or a Y, followed by an equal sign, and then the string of one or more digits which we wish to extract. Let's make these constants in our code, so it's easier to concentrate on what's around them (I've replaced the numeral set [0-9] with the digit class \d, another personal preference):

const string x = @"(?<=X=)\d+";
const string y = @"(?<=Y=)\d+";

Without named groups, we would simply incorporate x and y into a grouping pattern with an intervening wild string:

const string pattern = "(" + x + ").*(" + y + ")";
var input = "@1,X=123@1,Y=456";
var match = Regex.Match(input, pattern);
if (match.Success)
 return match.Result("($1,$2)"); // Output: (123,456)

Two changes are required to convert to named groups. First the names have to be applied. This involves adding a ? at the start of each group, followed by its name in either angle brackets, or as here, single quotes:

const string pattern = "(?'X'" + x + ").*(?'Y'" + y + ")";

Then after the match succeeds, these names, this time enclosed in curly braces, can be used to extract the relevant matched values:

if (match.Success)
 return match.Result("(${X},${Y})"); // Output: (123,456)

Follow Up Questions

Any reason you didn't use string.Format to define the regex?
This is what I ended up with: 

const string xCoordGroupName = "XCOORD";
const string yCoordGroupName = "YCOORD";
string pattern = string.Format("X=(?<{0}>[0-9]+).*Y=(?<{1}>[0-9]+)", xCoordGroupName, yCoordGroupName);
var match = Regex.Match(coords, pattern);

Readability by any chance?

Try changing that to "const string pattern...".
The + string concatenation operator is not inefficient at compile time.
Also, you often get a lot of {}s in your patterns, and Format doesn't like that.

More Than One Type of Hash

Oh No Not Again

Why does it seem that we still continue to read about millions of passwords / user credentials being stolen on such a regular basis, almost a quarter century after that famous and oft-cited Unix password security study? This past week has seen three separate large scale breach scares at LinkedIn, eHarmony and Last.fm. Users are being told to change their passwords immediately.

But that only protects the service provider against intrusion and accusations of laxity. On its own it does nothing to protect the user, whose new password can presumably be obtained just as readily, and whose old one may well be tied to more sensitive accounts e.g. at their bank or on line shopping stores. And more specifically in cases such as these, why is there so much panic anyway over the theft of password hashes, i.e. the encrypted versions of the users' passwords, rather than the passwords themselves? Aren't these hashes supposed to be designed for safe transmission in the clear?

Monkey Wrench

Yes and no. And mostly, no. The crucial point here is the distinction between a cryptographic hash and a password hash. The former is designed to be applied to content. As such, it carries a core requirement of fast decryption. Lightning fast in fact, as in: every IP packet arriving at a node must be decrypted quickly and on-the-fly, so as to generate no discernible latency to router traffic. Password hashes start from the exactly opposite requirement: maximal difficulty of decryption.

It matters little to a typical website user if their daily log in takes a full tenth of a second to authenticate. It matters a lot more to an attacker, who must try billions or trillions of alternatives to achieve a single successful break in. So it's not just a case of choosing between MD5 (feeble), SHA-1 (better, but still broken) and SHA-512/384 (no known collision attacks), or of adding a little salt. To arrive at a secure password hash, typically you would first select your atomic encryption algorithm, and then apply it a few thousand times.

Must Try Harder

So why do most providers use a cryptographic hash to do the job of a password hash? Most of the time this is allowed to happen because the developers in charge of a site's security are general practitioners, rather than security specialists. To them, any tool with encryption in its name is as good as any other.

This is a general problem of a lack of knowledge in the field. Developers have been successfully dissuaded from building their own cryptographic systems, being taught instead to rely on proven off-the-shelf solutions. And that's undeniably a great deal of progress, given the potential scope for devastating errors. Now, this community needs also to learn that different kinds of cryptographic solutions are available, for different jobs.

For a more extensive treatment of this subject, see Brian Krebs's recent interview with Matasano Security researcher Thomas H. Ptacek: http://krebsonsecurity.com/2012/06/how-companies-can-beef-up-password-security/.