PhilipMat

Replicating macOS's say command in Windows

Is say, macOS has a wonderful command line utility which I found to be useful to use in conjuction with long-running processes or even debugging to help draw attention, more so that typical beeping would do.

In short, say speaks text - is a Text-to-Speech (TTS) program.

say "Hello, there"

I wanted something similar on Windows and while there’s no direct equivalent, luckily .NET provides an entire host of utilities through the System.Speech.Synthesis namespace.

The say command has a number of parameter, mostly dealing with technical attributes such as voice selection (the speaker), output of spoken text, quality, e.t.c.

For this example, we’ll stick with the default voice of the speech synthesizer. As such, the solution is really simple using a Powershell script:

[cmdletbinding()]
param(
    [Parameter(Position = 1, Mandatory = $true)]
    [String]
    $message
)
Add-Type -AssemblyName System.Speech
$synth = New-Object -TypeName System.Speech.Synthesis.SpeechSynthesizer
$synth.Speak($message)

The Position-al parameter binding allow us to either call it directly:

say.ps1 'Hello there'

Or pass is with a switch argument:

say.ps1 -message 'Hello there'

I wish Visual Studio still had the ability to call macros on breakpoint because the code could translate into a one-liner in C#:

new System.Speech.Synthesis.SpeechSynthesizer()
  .Speak("Breakpoint hit");

As such, one would have to wrap it first into a method that can then get called when a breakpoint is hit.

Debug action with speech synthesis

It would be interesting to replicate the rest of the commands, in particular the voices since that would also allow for proper I18N speech synthesis.

Enumerating Directly to a FileResult

ASP.NET Core controllers have a series of file method that deal with returning files to the browser.

The methods can be grouped in three categories, all returning a FileResult:

  • returning a file specified by path: File(String, ...) and friends;
  • returning an array of bytes making up a file contents: File(byte[], ...), etc;
  • returning a file whose content is read from a stream: File(Stream, ...), etc.

As of v2.1 there’s over 20 File(...) methods supporting the various scenarios of these 3 categories.

What is missing, likely because it doesn’t fit with the simplicity of types used in the other signatures, is the ability to write an enumeration directly to the output stream and do so with minimum memory usage.

An good example would be to serve the results of a query as CSV.

There are certain ways to work with the existing methods, for example we could write the enumeration to a file and then use on of the File(string, ...) methods to serve the file.
Another approach would be to write it into a stream, in memory (MemoryStream), re-setting the stream position to 0 and using the File(Stream, ...) signatures.

Both of those approaches are inefficient in that they either perform unneeded I/O or use unnecessary memory.

In either of these cases, the more efficient approach would be enumerating/iterating through the records rather than materializing the entire dataset, and writing them out to the response stream one by one.

To do so we’ll construct our own implementation of the abstract FileResult class. The source of inspiration is the fact that each of the File methods mentioned above returns their own type of FileResult: FileContentResult, FileStreamResult, etc.

Our implementation, EnumerableFileResult will accept an IEnumerable and will write its elements one by one to the Response.Body stream (System.IO.Stream).

However EnumerableFileResult would not know how to write the elements to the stream, so it will delegate that resposibility to an adapter class, one implementing an proposed IStreamWritingAdapter interface.

To make the this whole implementation even more flexible, we’ll consider allowing the adapter to write a header, for example the column names, and since we’re there allow it to write a footer too (maybe the total record count?).

The IStreamWritingAdapter looks like the following:

public interface IStreamWritingAdapter<T>
{
    string ContentType { get; }

    Task WriteHeaderAsync(Stream stream);

    Task WriteAsync(T item, Stream stream);

    Task WriteFooterAsync(Stream stream, int recordCount);
}

The ContentType ties the adapter to the file type, after all they’re in synca, and allows the adapter to inform the FileResult parent of the MIME content-type of the content dispatched to the caller.

To recap:

  • EnumerableFileResult<T> inherits from FileResult;
    • Accepts an IEnumerable<T>;
    • Uses an IStreamWritingAdapter<T> to write each element of the enumeration to a Stream.
class EnumerableFileResult<T> : FileResult
{
    private readonly IEnumerable<T> _enumeration;
    private readonly IStreamWritingAdapter<T> _writer;

    public EnumerableFileResult(
        IEnumerable<T> enumeration,
        IStreamWritingAdapter<T> writer)
        : base(writer.ContentType)
    {
        _enumeration = enumeration ?? throw new ArgumentNullException(nameof(enumeration));
        _writer = writer ?? throw new ArgumentNullException(nameof(writer));
    }

    public override async Task ExecuteResultAsync(ActionContext context)
    {
        SetContentType(context);
        SetContentDispositionHeader(context);

        await WriteContentAsync(context).ConfigureAwait(false);
    }

    private async Task WriteContentAsync(ActionContext context)
    {
        var body = context.HttpContext.Response.Body;
        await _writer.WriteHeaderAsync(body).ConfigureAwait(false);
        int recordCount = 0;
        foreach (var item in _enumeration)
        {
            await _writer.WriteAsync(item, body).ConfigureAwait(false);
            recordCount++;
        }

        await _writer.WriteFooterAsync(body, recordCount);

        await base.ExecuteResultAsync(context).ConfigureAwait(false);
    }
    ...
}

The usage is fairly straighforward; within a controller or a page handler, we return an instance of the EnumerableFileResult initialized with the enumeration and the writer:

public IActionResult OnDownload() {
    IEnumerable<Person> people = GetPeople();
    return new EnumerableFileResult<Person>(
        people,
        new PeopleToCsvWriter()) {
            FileDownloadName = "People.csv"
    };
}

I’ve provided on GitHub a fully functioning example.

Simply clone and run the application and notice that the memory usage of the application while generating 100k or even 1M records is fairly small and constant past the initial load.

Note: a nicer implementation of this would make use of a Visitor Pattern, in which a CsvVisitor would visit any T implementation that accepts such a visitor allowing even further decoupling between the objects being enumerated and the class doing the writing.

Two Approaches to Searching Users in Active Directory

I’m sure there are more than two ways to perform searches against Active Directory, however I wanted to highlight two approaches: DirectorySearcher and PrincipalSearcher.

The former, DirectorySearcher comes from System.DirectoryServices and it’s the more “bare-metal” version of the two.

PrincipalSearcher, of System.DirectoryServices.AccountManagement provenance, is more of a query by example pattern and I’d say a higher level abstraction of directory searching.

To use DirectorySearcher, namely through it’s Filter property, one requires a bit more advance knowledge (or Googling skills) in order to decipher and employ the LDAP format filter string.

The payoff of using DirectorySearcher is the ability to construct complex query, including compound expressions across various objects: "(&(objectCategory=person)(objectClass=contact)(|(sn=Smith)(sn=Johnson)))" would find all contacts with a surname of Smith or Johnson.

However, for simple queries, the simplicity of PrincipalSearcher makes for easier to read code.

Consider the example of searching for all domain IDs (SAM account name) that begin with “john”:

var domain = "CORP";
var container = "DC=ad,DC=example,DC=com";

using(var context = new PrincipalContext(ContextType.Domain, domain, container)) {
    var principal = new UserPrincipal(context) {
        SamAccountName = "john*"
    };
    using(var searcher = new PrincipalSearcher(principal)) {
        PrincipalSearchResult<Principal> result = searcher.FindAll();
        result.Dump();
    }
}

Contrast with the same code using DirectorySearcher:

var ldapPath = "DC=corp,DC=ad,DC=example,DC=com";

using (var entry = new DirectoryEntry($"LDAP://{ldapPath}"))  {
    using(var searcher = new DirectorySearcher(entry)) {
        searcher.Filter = "(&(objectClass=user)(sAMAccountName=john*))";
        SearchResultCollection result = searcher.FindAll();
        result.Dump();
    }
}

Should we want to find a user with the last name being “Smith”, in the PrincipalSearcher case is as easy as setting the UserPrincipal’s Surname property - easily discoverable, whereas for the DirectorySearcher one would have to research and find out that the property is called, a bit more cryptical, sn.

What was also interesting to me is that perhaps owing to PrincipalSearcher formulating better criteria that I could, DirectorySearcher seems to be about 1.5-2x slower that the Principal version: whereas the former returns, in my attempts, in about 500ms, the directory searcher version takes 800-1,100ms for the same operation.

The type returned by the two methods is also another factor worth considering.

The SearchResult returned by the directory searcher method is sparse and all interaction is to be done through its Properties property, which is an implementation of System.Collections.DictionaryBase.
These properties are really LDAP properties and to get information out of a search result one needs to know what these properties represent – for example, knowing that “c” represent “country”, or “sn” is “surname”, or “cn” is “common name”.

In contrast, the UserPrincipal class offered by the PrincipalSearchResult<T> has more straighforward properties: Surname, GivenName, etc, although it might not have some of the properties stored in LDAP, for example the afore mentioned c = countryName.

Due to its more straightforward nature, I will be personally employing PrincipalSearcher for simple search queries and hope that I would never have to land in a case where I require the full power of the DirectorySearcher.

However, if I do - I now know what to search for.

Loading Claims when Using Windows Authentication in ASP.NET Core 2.x

Much like almost everything else in ASP.NET Core, enabling Windows Authentication in ASP.NET Core is well documented and has supperb step-by-step examples.

The Claims-based authorization system is documented just as well and the examples are well chosen.

Where I thought the documentation fell short was the marrying of the two concepts; there is little explanation given to how the claims are actually made available to be check and asserted on.

If we were to inspect the Identity of a User, we would notice that it already has a substantial Claims collection. These claims are all seemingly associate with specific Windows user properties, and to me have largely legible names yet indecipherable values, save perhaps for the .../name claim:

Type Value
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name HOME\philip
http://schemas.microsoft.com/ws/2008/06/identity/claims/primarysid S-1-5-21-616010284-1202357983-1921873989-1000
http://schemas.microsoft.com/ws/2008/06/identity/claims/groupsid S-1-1-0
http://schemas.microsoft.com/ws/2008/06/identity/claims/groupsid S-1-5-4
etc etc

In contrast, the Claims examples make use of such nicely named claims like "EmployeeNumber" or ClaimTypes.DateOfBirth, none of which can be found in the claims collection of our Windows user.

To load claim in ASP.NET Core 2.x we make use of one or more claims tranformations, classes implementing IClaimsTransformation (used to be called IClaimsTransformer in earlier versions), which get access to the ClaimsPrincipal and can construct new ones or add claims to the loaded one.

In the following example we’ll look at adding our own claims to the collection. To make it a bit more interesting, let’s assume we have a table in the database that stores the ids of the users who are administrators of our own application and we would like to add a flag in claims if a user logging in is part of this table.

Assuming we use these in combination with Authorize attribute, likely to check for an "IsAdmin" claim: [Authorize(Policy = "IsAdmin")], we will be making the following changes to our application:

Packages required

If running against .NET Core 2.x, the Microsoft.AspNetCore.App meta-package is sufficient.

If running against .NET Framework 4.6+, we need to add:

  • Microsoft.AspNetCore.Authentication - provides a large host of authorization classes, policies, and convenience extension methods;
  • Microsoft.AspNetCore.Server.IISIntegration - adds support for IIS (and IIS Express) in further support of the authentication process.

Code changes

launchSettings.json

Enable Windows authentication for IIS. Also enable anonymous access if usage of [AllowAnonymous] attribute is needed:

{
  "iisSettings": {
    "windowsAuthentication": true,
    "anonymousAuthentication": true,
...

Startup.cs

Enable authentication by adding the following to the Configure(IApplicationBuilder app, ...) method:

app.UseAuthentication();

Add IIS authentication scheme in ConfigureServices:

services.AddAuthentication(IISDefaults.AuthenticationScheme);

We’ll be back here in a bit to register our claims loader

ClaimsLoader.cs

Before we implement IClaimsTransformation a couple notes about it.

First, they run on each AuthenticateAsync call, which means for IIS Authentication they run only once and whatever claims we add to the collection are cached for as long as the user is logged in.
If we remove a logged in user from the list of administrators, they will continue to behave as such until they log in again.

Second, they run on each AuthenticateAsync call, so we will heed this warning from the documentation of TransformAsync:

Note: this will be run on each AuthenticateAsync call, so its safer to return a new ClaimsPrincipal if your transformation is not idempotent.

This is because if any call (tests?) causes AuthenticateAsync to be called twice, the same claim is added twice to the collection as pointed out in this article by Brock Allen.

using System.Security.Claims; // for ClaimsPrincipal
using Microsoft.AspNetCore.Authentication; // for IClaimsTransformation

public class ClaimsLoader : IClaimsTransformation
{
    public const string IsAdminKey = "IsAdmin";
    private readonly UserContext _userContext;

    public MigrationsUserClaimsLoader(UserContext userContext)
    {
        _userContext = userContext;
    }

    public async Task<ClaimsPrincipal> TransformAsync(ClaimsPrincipal principal)
    {
        var identity = (ClaimsIdentity)principal.Identity;

        // create a new ClaimsIdentity copying the existing one
        var claimsIdentity = new ClaimsIdentity(
            identity.Claims,
            identity.AuthenticationType,
            identity.NameClaimType,
            identity.RoleClaimType);

        // check if our user is in the admin table
        // identity.Name is the domain-prefixed id, eg HOME\philip
        if (await _userContext.IsAdminAsync(identity.Name))
        {
            claimsIdentity.AddClaim(
                new Claim(IsAdminKey, "So say we all"));
        }

        // create a new ClaimsPrincipal in observation
        // of the documentation note
        return new ClaimsPrincipal(claimsIdentity);
    }
}

Startup.cs - adding policy

Now that we created our claims loader, let’s register it with the service collection and add a policy for it too:

services.AddTransient<IClaimsTransformation, ClaimsLoader>();

services.AddAuthorization(options =>
{
    options.AddPolicy(
        "IsAdmin",
        policy => policy.RequireClaim(ClaimsLoader.IsAdminKey));
});

At this point we can decorate our controllers or controller actions and employ the policy we just added:

[Authorize(Policy = "IsAdmin")]
public Task<IActionResult> AddUser() {
    ...
}

Variation

The example adds the "IsAdmin" claim only if the user is an admin.

If we wanted to add the claim anyway and rely on the value of the claim, the code changes as following:

ClaimsLoader.cs - variation

bool isAdmin = await _userContext.IsAdminAsync(identity.Name));
claimsIdentity.AddClaim(new Claim(IsAdminKey, isAdmin ? "yes" : "no"));

Startup.cs - variation

services.AddAuthorization(options =>
{
    options.AddPolicy(
        "IsAdmin",
        policy => policy.RequireClaim(ClaimsLoader.IsAdminKey, "yes"));
});

or to add a JavaScript flavor to it ;)

services.AddAuthorization(options =>
{
    options.AddPolicy(
        "IsAdmin",
        policy => policy.RequireClaim(
            ClaimsLoader.IsAdminKey,
            "yes", "Yes", "true", "True", "1")); // ugh
});

Thirty_minutes_before_coding

By noticing them missing in a good deal of projects and the friction and occasional frustration that came with it, I have came up with a list of things I would like to have in place before I start writing even a single line of code.

This list attempts to address three questions:

  • What is this project and/or what purpose does it solve?
  • How does one run it or use it?
  • How can one contribute?

Those are answered, I believe, by having a few essential things in place:

  • A well written README file - this is the first introduction to my project, so I will put extra care to make sure it’s clear, concise, semantically, and syntactically correct.
  • A script (Makefile, npm scripts) that helps run or use this project; if it’s a library, have documentation, with copious examples of usage.
    It can be part of the README - if not, the README should include a prominent link to this documentation.
  • Build script and Continuous Integration - the latter is in particular so easy to set up nowadays there’s no good excuse not to.
    Include linters and style guide (e.g. pep8).
  • Guidelines on how to contribute:
    • call out expectations and requirements for pull requests (e.g. documentation, tests, other artifacts);
    • set up templates for submitting issues and request improvements;
    • Set up a code of conduct, even if I’m the only contributor. It’s something to read when I get upset or frustrated (we’re all humans);
  • Publish scripts: nuget, npm, pypi all have their own preferred formats (optional but highly recommended if it’s a library);

With that in mind, here how the first 30 minutes before I write any code look like:

T-30: Head over to GitHub, Bitbucket, or GitLab and create a new repo with a README.
Select the .gitignore appropriate to my project as well as the license.

T-29: Clone the repo on my computer. Open the README.

T-29: Take 5 minutes to document what the project does and what problem it solves. Add links to any relevant resources (e.g. blog posts, Stack Overflow).

T-24: Set up the directory structure.

T-23: Create a minimal build script. I use:

  • Cake for .Net projects;
  • scripts node in package.json for JavaScript project;
  • For Python projects I like using the requests Makefile and Kenneth Reitz’s setup.py.

Add nodes/entries for linting (pep8, flake8, eslint) and unit testing (even if I don’t have any yet).

T-18: Go back to the README and add a section about how to run the script.

T-15: Add CI integration: Travis CI for Linux and AppVeyor for Windows projects; or both.
The build script should come in handy for this step.

T-10: Back to the README, add the AppVeyor and Travis badges so visitors know the current status of my build.

T-9: Add support for Snyk to help with vulnerability monitoring.

T-7: If the provider support it, I spend a few minutes creating issue templates to help with reporting defects and suggest improvement; if not, document in the README the type of information needed for defects.

T-4: Consider adding contributing guidelines and adopting a Code of Conduct.

T-1: Add an ## Examples or ## Usage node to the README as a reminder to add more documentation once the code or interfaces get fleshed-out.

T-0: Happy coding.

TL;DR / Checklist

  1. Repo with .gitignore, license, and README;
  2. Immediately document what the project does and what problem it solves;
  3. Set up folder structure;
  4. Create minimal build script;
  5. Add to README instructions on how to run the build script or the project via the build script;
  6. Add CI integration; add badges for CI status;
  7. Add configuration for Snyk to help detect vulnerabilities;
  8. Write issue templates or document how to report issues;
  9. Add contributing guide lines and Code of Conduct;
  10. Add Examples and Usage entries in README to fill in later.